The 5 Rights Aren't Wrong,... But...

"In any moment of decision, the best thing you can do is the right thing, the next best thing is the wrong thing, & the worst thing you can do is nothing."

Theodore Rooseveldt 

In November of 2014, The ONC (Office of the National Coordinator, I liked it better when it was ONCHIT) posted guidance regarding clinical decision support (CDS). This guidance proposed “5 rights” that applications & implementations that provided CDS would have to support. The five rights[1] were defined as:

         ·      Right information (evidence-based guidance, response to clinical need)

         ·      Right people (entire care team – including the patient)

         ·      Right channels (e.g. EHR, mobile device, patient portal)

         ·      Right intervention formats (e.g. order sets, flow-sheets, dashboards, patient lists)

         ·      Right points in the workflow (for decision making action)

The ONC also stated that CDS is more than just the delivery of alerts. The 5 rights provide a good context for thinking about CDS & how it can/should be used. In particular the CDS/PI (Clinical Decision Support Collaborative/Performance Improvement) detailed “steps” table[2] seems quite useful, at least as a source of information on CDS.

A number of older, but still relevant (IMHO) academic studies provide more insight into the most effective capabilities & features of CDS systems. Specifically, Kawamoto, et al.[3] did a meta-analysis of the effectiveness of CDS systems in improving outcomes & found that four features made such systems most effective (in descending order):

       1.     Automatic provision of CDS as part of clinical workflow

       2.     Provision of recommendations rather than just assessment

       3.     Provision of CDS at the time & location of decision making, &

       4.     Provision of fully automated CDS rather than partial reliance on the provider to look up information

Other features that were significant but not at the level of these four were:

       ·      No need for additional provider data entry

       ·      Requesting explanation (from provider) of why recommendation(s) were not followed (requires    more analysis of decision by provider & may change decision)

       ·      Provision of CDS results to patient (acknowledgement of usefulness of system & transparency), & finally

       ·      Provider education (often not by vendor) for use & interpretation of CDS

Other studies e.g. Garg, et al.[4] also a meta-analysis, provide highly detailed analysis of effective features for different types of targeted CDS (diagnosis, preventive care, diabetes treatment, etc.).

My semi-recent experience on a federal grant has given me some perspective on this. I was Principal Investigator on a Department of Commerce (NIST) grant to develop a secure network with identity verification at all endpoints for clinical eReferral. This work was done with UC San Diego Medical Center & the San Diego VA. Direct network connections were set up for UCSD & its medical affiliates including a set of CHCs in the Columbia River Gorge area (OR). Part of the project was that eReferrals done through the Direct connections would include the possibility of using the ActiveHealth Management (now owned by Aetna) application for CDS. What we found was that providers, on both sides of the connection were loath to use the CDS as it required: additional &/or redundant information to be entered (usually by the provider) & going out of the Direct connect/EHR context. In addition, providers did not like that the application made recommendations & provided copious amounts of background for the recommendations, but allowed for only minimal provider input & feedback. (FYI – ActiveHealth Management never followed through on its commitments to the grant, so this capability was only tested in prototype.) The result was clear, however. Providers were not interested in CDS/recommendations if they were not fully integrated into their EHR & workflow. Even though ActiveHealth Management was considered “best-of-class” CDS at the time, it was not enough to tempt providers to use it consistently.

Other types of patient-centered recommendation systems may also be interesting & relevant here, specifically the point-of-care recommendation systems under development (& in some cases being used by) by Kaiser, Partners, Geisinger, Mayo Clinic, Cleveland Clinic etc. These systems do operate for the provider at the point-of-care & meet most of the criteria described by Kawamoto, that is:

         ·      Are provided automatically at the time & place of decision making

         ·      Make recommendations, not just summarizations/assessments, &

         ·      Do not require any additional data entry.

Of course, even though these systems make recommendations based on the analysis of many thousands - & in some cases millions – of patient records, they are only as good as the quality of the data that they analyze. Based on our experience with data quality in the Path-2-Analytics[5] project, we can make some assumptions regarding the quality of a data set with millions of records, the compilation of which started 12-15 years ago. Kaiser, for instance, has 9M+ patient records dating from 2003. These types of systems will become extremely important over the next 5-8 years, but today they are more conceptually important than functionally important.

OK – so what’s the bottom line here? The ONC, CMS etc. will continue to mandate that providers begin using clinical decision systems in their clinical practice, but what leverage do these systems actually provide? & how can providers & healthcare organizations select CDS systems that will improve their practice? The ONC defines CDS as: “Clinical decision support (CDS) provides clinicians, staff, patients or other individuals with knowledge and person-specific information, intelligently filtered or presented at appropriate times, to enhance health and health care. CDS encompasses a variety of tools to enhance decision-making in the clinical workflow. These tools include computerized alerts and reminders to care providers and patients; clinical guidelines; condition-specific order sets; focused patient data reports and summaries; documentation templates; diagnostic support, and contextually relevant reference information, among other tools.”[6]  

The Stage 2 Meaningful Use measure for CDS (2016-17) is:[7]

“Measure1:Implement five clinical decision support interventions related to four or more clinical quality measures at a relevant point in patient care for the entire EHR reporting period. Absent four clinical quality measures related to an EP’s scope of practice or patient population, the clinical decision support interventions must be related to high priority health conditions.

Measure 2: The EP has enabled and implemented the functionality for drug ‐ drug and drug allergy interaction checks for the entire EHR reporting period.”

As ONC points out, this set of capabilities requires the ability to access & analyze patient-specific clinical data, some type of reasoning mechanism (rules, big data analysis, human intervention etc.) to interpret the data & construct recommendations, create care plans, assess current care etc. & the ability to present information & inferences in an understandable & useful way.

It should also be pointed out that the Stage 2 criteria are also linked to interventions related to to the CMS clinical quality measures, that is some action must be taken based on input from a CDS (alert, etc.) that is related to a quality measure as described by CMS[8]. There are currently 64 of these measures ranging from HbA1c measures to colorectal cancer screening to pediatric asthma screening

There are many standalone CDS products, most of which are focused on a specific clinical area. Examples are: interpretation of tumor imaging for specific types of tumors, interpretation of retinal imaging for treatment of macular degeneration, tracking of quality measures for individual patients & alerting provider when measures fall outside of guidelines & many, many others. There are very few general CDS systems, the best-known being ActiveHealth Management, a system that relied on a set of clinicians & human experts that continuously reviewed medical journals & other material to extract the latest information on treatment patterns & outcomes. This information was then translated into a rule set that was used to interact with provider input (or automated input through CDS) regarding specific patients. Treatment recommendations & treatment plans were then suggested to the provider. Very few CDS except for ActiveHealth & the point-of-care recommendation systems using big data analytics work like this. Most are much more targeted. Most CDS function provided by EHRs is in the form of reminders & alerts, although this is starting to change.

So how does a healthcare organization select a CDS that will work to improve outcomes & not just meet meaningful use requirements? Here are some steps:

          1.     Determine what data you have available to serve as the base for decision support. This will include your EHR & PM systems as well as other internal data. It may also include external data as relevant. External data may include clinical data from other sources (HIE, eReferral etc.) or public &/or private clinical datasets.

         2.     Determine what type of decision support you require & who the recipient of this support will be. Are you looking for general point-of-care diagnostic & treatment planning support (order sets, procedure lists etc.)? or are you looking for specific support in particular types of image interpretation or other highly specialized support?

         3.     Develop a set of expectations, goals, requirements, users & use cases for your CDS system.

         4.     Evaluate what CDS capabilities you already have in your EHR or other HIT systems. Identify the gaps in the capabilities you have versus your requirements.

         5.     Research & identify CDS systems that will fill these gaps. Make sure you determine compatibility of any systems you are researching with your existing HIT systems. Also make sure you understand the skills required to use these systems & what training is required (including but not limited to vendor supplied training as your staff may need more general training on decision making & decision support).

        6.     Evaluate & test the system(s) you have identified using the real use cases you described in Step 3. Use data that is as realistic as possible, that is as close to your clinical data as possible (in some cases, it may be possible to actually use your own data).

       7.     Work with the vendor to make sure that the system you will purchase is customizable &/or configurable the way you need & that it is fully compatible with your other software.

       8.     Arrange for a test period & do a series of tests that exercise your use cases. Work with the vendor to remediate any issues you find.

        9.     Arrange for any training that the vendor &/or external organizations can provide.

At this point you are ready to purchase your CDS system.

OK, you say,… that seems like a lot to go through, especially since my EHR vendor assures me that their product provides clinical decision support that is compliant with meaningful use, PCMH, MIPS & any other regulation or guideline you have asked about. My experience, as I’ve already stated, is that the CDS associated with EHR products is mostly limited to alerts that are programmed to fire when specific clinical measures are above guidelines or when a series of values trend above guidelines. More recently, they can also make recommendations for & enable ePrescription of specific drugs or even propose order sets (for CPOE). This is all good, in fact much better than not providing such capabilities at all, but if you have determined a broader set of requirements, then your EHR may not provide everything you need. The bottom line is CDS that actually meets the ONC’s definition is not provided by any current EHR or in fact even by any combination of EHR & dedicated CDS application. It is most closely approached by the point-of-care recommendation systems that are under development & in preliminary use by organizations such as Kaiser Permanente, Geisinger, Partners etc. These systems combine ultra-large data sets with advanced deep learning & pattern recognition capabilities that will be in general use in the next 3-5 years, but today are the exception.

I believe that the best we can do today is a combination of CDS provided by your EHR along with a more specialized system that addresses some of the additional requirements you have identified. EHR decision support will get better only if it is required to, but Stage 2+ meaningful use requirements as well as the proposed Stage 3 (maintaining the two requirements for Stage 2 with additional emphasis on appropriate position of CDS in the clinical workflow[9]) do not push for CDS as described by the ONC except to say that CMS wants to encourage innovative development & use of decision support beyond alerts & notifications. We have a long way to go before CDS is a really effective addition to the care of patients… No number of definitions, usage descriptions of reports by Federal agencies will make it happen. Vendors will develop effective CDS only if it is required, so it is up to providers, policy experts & healthcare organizations to both push for this development & to produce their own definitions, descriptions, use cases & even prototypes as appropriate & possible. Now is the time to start…

---

[1] https://www.cms.gov/Regulations-and Guidance/Legislation/EHRIncentivePrograms/Downloads/ClinicalDecisionSupport_Tipsheet-.pdf

[2] https://sites.google.com/site/cdsforpiimperativespublic/CDSQI-stepbystep

[3] K. Kawamoto, C. Houlihan, E.A. Balas, D.F. Lobach. Improving clinical practice using clinical decision support systems: A systematic review of trial to identify features critical to success. BMJ, doi:10.1136/bmj.38398.500764.8F (published 14 March 2005)

[4] A.X. Garg, et al., Effects of computerized decision support systems on practitioner performance & patient outcomes. JAMA. 293(10). P. 1223-1238.

[5] Path-2-Analytics Project: Process & Results Review, Association of Clinicians for the Underserved Annual Meeting. Washington, D.C, June 2015.

[6] https://www.healthit.gov/policy-researchers-implementers/clinical-decision-support-cds

[7] https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/EP_ObjectiveMeasuresTable-.pdf

[8] https://www.cms.gov/regulations-and-guidance/legislation/ehrincentiveprograms/ecqm_library.html

[9] https://www.healthit.gov/providers-professionals/how-attain-meaningful-use

      

Social Media & Clinical Workflows 2

I have previously written about the integration of public social media into clinical workflows in order to improve the engagement of patients in collaborating with their providers to make healthcare decisions. In that post (http://posttechnical.blogspot.com/2013/12/clinical-workflow-other-arcane-rituals.html), I said that the inclusion of information from public social media such as Facebook, YouTube, G+, Twitter etc. into clinical workflows might provide important information not otherwise available to the clinician & also might serve to more deeply engage the patient in clinical decision making as information that they independently provided would be part of the decision process. I wrote in general about how this might work – here’s a more detailed (& nuanced) view.

Providers are very protective of their workflows, especially workflows that involve patient interaction. In the past several years, these workflows have had to change substantially if a provider is going to qualify for meaningful use, that is specific use of electronic health records in order to qualify for higher reimbursements from CMS (the Centers for Medicare & Medicaid Services, HHS). Most providers I have talked to recently are not interested in more workflow changes, especially if they are not going to be paid for them. There is, however, one motivating factor that can change this - improving patient outcomes. If providers could be convinced that including information from external sources, such as public social media, in their patient interaction & diagnosis workflows, then it might be possible make this integration work productively, but several principles would have to be followed:

•        Change the current EHR-based workflow as little as possible
•      Maintain the information from the external source separately from the EHR so that the EHR workflow can still be executed as currently
•      Provide information that is either unique & relevant to the diagnosis & treatment, or enhances information already available through the EHR.

I’ll describe a possible workflow scenario that follows these principles in a little, but first…

I also described the criteria for collaboration in that previous post. If the patient is going to be re-engaged in making medical decisions with their provider, there has to be a model for how to do this. As I stated previously, that model is not collaboration-based because the criteria for collaboration: shared goal structure, similar reward structure & symmetry in knowledge or resources, are not met by the patient-provider interaction. The model for this interaction is shared decision making, so an additional principle for this workflow change is to provide the context for shared decision making by the patient & provider.

The principles of shared decision making in the medical context are well documented[1] & include at least:    

• relationship building between patient & provider,
•        introducing choice with respect to treatment
•     offering options
•     deferring decision
•      discussing & deciding on a treatment plan.
•    Risk-reward trade-offs
•      Use of decision aids (paper-based, electronic etc.) for education
•       Elicit preferences
•      Discuss preferences
•      Reach consensus

Shared decision making is most appropriate in situations where there are several possible treatments with somewhat similar side effects, costs & outcomes (if there is one most effective treatment, the need for shared decision making, but not for transparency, is moot). This process only works if the decision-making effort is mutual, that is both the patient & provider contribute to the trade-off, preferences & decision discussions.

So where are we? I’m maintaining that the integration of information from public social media can be an important contribution to clinical workflows for two reasons: it may provide unique & important information to the diagnosis & treatment process that has not been previously available & it may serve to more deeply engage patients in their healthcare decisions because information provided by them is being used by their provider(s). I’ve also stated that shared decision making is the most effective model for structuring the provider-patient interaction. How might this actually work?

The most common provider workflow for patient interaction today is provider by the electronic health record system in use. There is no current provision for integrating external data into this workflow (other than lab, pharmacy & other provider related data), & there is little motivation by either providers or vendors to provide such integration. There is, however as previously noted, a lever to motivate providers – better outcomes. The provider (or one of their staff) is previewing the patient’s records prior to a scheduled encounter. The system informs the reviewer that additional information is available from an identified source. This source could be a PHR or a social media stream that the patient has posted personal health information to. The information may be a post from the previous week where the patient described feeling sick or even an image of a skin rash or other physical symptom. As already stated, this information is in a separate stream from the EHR data & is not moved from its source unless the reviewer requests it. The reviewer can look at the information, decide if it is relevant, & choose to include it in the overall data package to be available during the encounter. If the reviewer chooses not to include it, a notification is included that external information from specified sources was reviewed but not included & the provider has the option to look at it during the encounter. If the information is included, it is presented as a separate stream at the beginning of the workflow so as not to alter the rest of the encounter. It is also stored separately & tagged as part of a specific encounter, so that it can be recovered as part of the encounter, but is not managed as part of the EHR data. This could change over time as such information becomes more accepted & important to diagnosis & treatment.

There are many other ways that information from these external sources could be integrated into provider workflows. They could be part of the practice management workflow when the appointment is set up, but someone on the clinical staff would have to review them & the above model might have to be followed at the time of the encounter so that the most current information is available. Other models would also be developed through experience as such information is used.

Certain types of information related to specific practices may be more relevant initially than others. These information types might include:

•         Social media dialogs that reveal a patient’s expectations, attitudes & personal limitations with respect to their healthcare
•     Social media dialogs relevant behavioral healthcare
•     Dermatological or trauma images taken at the time of injury or presentation.

The types of information seen as relevant & effective will expand as this data is used & accepted more.

Of course, there are issues & constraints related to this model as well. Right up front is the status of using this type of data for diagnosis & treatment. If a provider were simply reading a patient’s social media stream & attempting to treat them on that basis, it appears that they would be in violation of the HIPAA statutes. I believe that if a provider had consent from the patient (as with any other personal health information) to use such information, the HIPAA guidelines would be met (but I could be wrong). Then there is the question of what constitutes such information. If the relevant or interesting content is part of a stream that other people are participating in, then the consent is only good for the patient & not the other participants, however the most interesting content might be in the interaction that is lost if only a single person’s contributions are captured. What about individual postings that have comments from other people, like an image that a patient wants feedback on? There are many such issues.

Finally, there is the issue that no systems capable of providing this type of function are available currently in healthcare. All of the capabilities are present in other types of systems, but someone would have to build a prototype & pilot this type of use. Who would do this? Health & Human Services (ONC)? Commerce? the DoD? the VA? Kaiser Permanente? Partners? Hopefully someone would. I’m trying to start a pilot (through the RCHN Community Health Foundation) to at least determine people’s attitudes about the use of social media content in clinical workflows. Stay tuned.

Coming next:

•  A deeper look at analytics in healthcare
• Some opinions on current trends
• &, the Future might even make another guest appearance.

Please also see my writing for the RCHN Community Health Foundation at: http://www.rchnfoundation.org/?page_id=484

---

[1] c.f - Makoul G, Clayman ML. 2006. An Integrative Model of Shared Decision Making in Medical Encounters. Patient Educ Couns.60:301–12. or Elwyn, G. et al. 2012. Shared Decision Making: A Model for Clinical Practice. J.Gen’l. Int. Med. 27(10) 1361-67,

Clinical Workflow & other Arcane Rituals...

In my last two posts, I’ve been writing about the problem of re-engaging patients in collaborating with their providers in order to make their healthcare decisions. Patients do not seem very motivated to use current tools (PHRs, patient portals, private social media) to affect this collaboration, & they are already overwhelmingly using public social media (Facebook, YouTube, Twitter, etc.) to share very intimate details of their personal health information. I proposed that the integration of public social information into the workflows of providers & healthcare organizations might be an effective way of encouraging or eliciting this collaboration. What does this mean? & is it feasible?

First several short, but relatively interesting (I hope) digressions. What is it that we mean by workflow? This is one of those things that everyone understands but that no one can define, design or optimize effectively. Wikipedia defines workflow as “…a sequence of connected steps where each step follows without delay or gap and ends just before the subsequent step may begin.[1]” Workflows can be abstract (models) or concrete (task or process steps). In the case of a clinical or other medical workflow, it is the sequence of tasks that a provider or other medical professional carries out in order to provide care for a patient. These steps may include information gathering &/or treatment tasks. Until recently, most clinical workflows had been developed & modified historically by provider actions & by the efforts of medical professional associations. More recently, workflows have been provided (at least in part) by the use of EHR systems that have implied workflows associated with their use. Many providers in the U.S. have gone through major changes in clinical workflows as they have adapted to the use of EHRs in order to qualify for meaningful use incentives offered by the Centers for Medicare & Medicaid (CMS). This adaptation may make it harder to get providers to accommodate to additional workflow changes.

Second, what is collaboration? Again Wikipedia defines collaboration as: “…working with each other to do a task and to achieve shared goals. It is a recursive process where two or more people or organizations work together to realize shared goals.[2]” The criteria for achieving collaboration have been deeply researched. Eisenhardt[3] has described the criteria necessary for actual collaboration as:   

•     Having a shared goal structure, or explicitly agreeing to disagree on goals,
•         Having a similar reward structure so that one party to the collaboration does not benefit more than the other, &
•     No substantial asymmetry in knowledge or information between parties.

Given these criteria, it is clear that the interaction, even decision making interactions, between provider & patient &/or caregiver is not & cannot be a collaboration. Even if they share the goal of a positive clinical outcome, they have very different contexts for that goal; the reward structure is similarly skewed, as is the symmetry of knowledge & information. What can happen is shared decision making[4] that is the patient & provider making decisions together with the resources & understanding that they have each developed. There has been much research on shared decision- making (SDM)[5] that has focused on building a consensus between provider & patient on a preferred treatment plan & its implementation. Notice that this is different than collaboration. What is needed, then, is a workflow that facilitates shared decision-making between providers & patients & their caregivers.

OK – how can this workflow be developed so that it engages both the provider & the patient. As I have already written, the integration of public social media for information sharing, community building & communications will be important as will gamification of the provider/patient relationship & the use of healthcare apps on smartphones & other devices. The easiest, & perhaps most painless way to do this for providers is to let the EHR vendors do it. This, of course begs several questions: what does this provide for the patient? How can the quality & efficacy of the vendors’ efforts be ensured? Even, how can the vendors be motivated to do this? My initial answers to these questions are: nothing, with difficulty & with difficulty – sorry about that. Many EHR vendors are moving in these directions, but their motivation to provide patient functionality via their products, except for that required by meaningful use criteria will be low. Many healthcare organizations are providing PHRs for patients to contribute information to their personal healthcare information, but providers have low motivation to use this “data”. There are companies now providing private social media capabilities for healthcare organizations, but it is just that – private. These private networks have trouble gaining traction with patients with the possible exception of directly messaging their provider (although many of the private networks &/or healthcare organizations using them do not permit this).

The real solution is to re-engineer clinical workflows so that there are (at least) alternatives task paths that include import or evaluation of data from public social networks, including chat & tweet streams, images, direct messages, video etc. Such streams could be displayed as is or extracted so that data might be available for import into other applications. Re-engineer you say… Yes, that re-engineering. About five years ago I did a set of work on comparisons of productivity measurements in ambulatory healthcare & two benchmark industries: auto & information. Ambulatory healthcare had relative good productivity based on measurements[6] such as value added to GDP from overall revenue & value added to GDP through Full-Timed Employee (FTE) wages. Ambulatory healthcare measurements were generally than Auto (1998-2005) & a bit lower than Information Industries[7]. Other productivity measurements are more focused on how organizations are structured & managed (Total Factor Productivity, TFP) or on the effect of multiple factors (Multi-Factor Productivity, MFP)[8] such as research & development investment, economies of scale, managerial effectiveness, etc. Ambulatory Healthcare had negative trends in MFP & TFP during the 1998-2005 timeframe while both Auto & Information had highly positive trends. It is thought that these results reflect the very large investment made in the benchmark industries in process optimization & workflow re-engineering. These efforts, with the exception of adaption to EHR adoption, have yet to be made in healthcare, & it could be argued that optimizing clinical & other healthcare workflows could result in much larger gains in productivity subsequently resulting in positive trends in outcomes.

A provider working through the EHR workflow could get to the history page (clinical history including diagnosis from past encounters, clinical data for selected measures etc.) or other relevant page & be presented with an alternative page that incorporated data from the patient’s PHR, selected (relevant) data from tweet & post streams, patient supplied images & video as well as potential symptom descriptions from social media streams. This additional data could provide valuable information not elicited by the normal EHR workflow. A good deal of work would have to be done to determine the most effective & productive way to provide this data as part of the workflow, so projects to explore this should be started now.

Patient portals could be redesigned with what we have learned about effective shared decision- making. This would make these portals much more interesting to patients (as well as to providers). Such portals could still provide access to patient data, provider messaging, but could also serve as workspaces for shared decision making with their own workflows, information sharing, game elements etc. The combination of information from public social media available in the clinical workflow (engaging providers with data not usually available) & a patient portal that was a shared decision-making workspace for the patient & provider could be a breakthrough combination.

Stay tuned for:

•         a continuation with some thoughts on shared decision making workplaces, &
•     I still haven’t posted the talk I had with the Future

---

[1] http://en.wikipedia.org/wiki/Workflow

[2] http://en.wikipedia.org/wiki/Collaboration

[3] Eisenhardt, K.M. 1989. Agency theory: An assessment and review. Academy of Management Review. 14(1):57-74. January, 1989.

[4] Shared Decision Making (SDM) is an approach where clinicians and patients communicate together using the best available evidence when faced with the task of making decisions, where patients are supported to deliberate about the possible attributes and consequences of options, to arrive at informed preferences in making a determination about the best action and which respects patient autonomy, where this is desired, ethical and legal. http://en.wikipedia.org/wiki/Shared_Decision_Making

[5] c.f. Elwyn G, Tsulukidze M, Edwards A, Légaré F, Newcombe R (2013). "Using a 'talk' model of shared decision making to propose an observation-based measure: Observer option^{5 Item}". Patient Educ Couns. doi:10.1016/j.pec.2013.08.005.

[6] Harper, M.J. et al. 2008. Integrated GDP-Productivity Accounts. American Economiics Association Annual Meeting. San Francisco, CA. 1/2009

[7] Hartzband, D.J. 2008. GDP-Based Productivity of Ambulatory Healthcare: A Comparison with Other Industry Segments. ESD-WP-2008-11. Engineering Systems Division Working Papers Series. Massachusetts Institute of Technology. February 2008.

[8] Bureau of Labor Statistics. Multifactor Productivity Homepage. http://www.bls.gov/mfp/