This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.
Physician office practices are increasingly adopting electronic medical records (EMRs). Therefore, the impact of such systems needs to be evaluated to ensure they are helping practices to realize expected benefits. In addition to experimental and observational studies examining objective impacts, the user’s subjective view needs to be understood, since ultimate acceptance and use of the system depends on them. Surveys are commonly used to elicit these views.
To determine which areas of EMR implementation in office practices have been addressed in survey-based research studies, to compare the perceived impacts between users and nonusers for the most-addressed areas, and to contribute to the knowledge regarding survey-based research for assessing the impact of health information systems (HIS).
We searched databases and systematic review citations for papers published between 2000 and 2012 (May) that evaluated the perceived impact of using an EMR system in an office-based practice, were based on original data, had providers as the primary end user, and reported outcome measures related to the system’s positive or negative impact. We identified all the reported metrics related to EMR use and mapped them to the Clinical Adoption Framework to analyze the gap. We then subjected the impact-specific areas with the most reported results to a meta-analysis, which examined overall positive and negative perceived impacts for users and nonusers.
We selected 19 papers for the review. We found that most impact-specific areas corresponded to the micro level of the framework and that appropriateness or effectiveness and efficiency were well addressed through surveys. However, other areas such as access, which includes patient and caregiver participation and their ability to access services, had very few metrics. We selected 7 impact-specific areas for meta-analysis: security and privacy; quality of patient care or clinical outcomes; patient–physician relationship and communication; communication with other providers; accessibility of records and information; business or practice efficiency; and costs or savings. All the results for accessibility of records and information and for communication with providers indicated a positive view. The area with the most mixed results was security and privacy.
Users sometimes were likelier than nonusers to have a positive view of the selected areas. However, when looking at the two groups separately, we often found more positive views for most of the examined areas regardless of use status. Despite limitations of a small number of papers and their heterogeneity, the results of this review are promising in terms of finding positive perceptions of EMR adoption for users and nonusers. In addition, we identified issues related to survey-based research for HIS evaluation, particularly regarding constructs for evaluation and quality of study design and reporting.
The importance of office-based electronic medical records (EMRs) and related systems is being recognized internationally. For example, Canada Health Infoway [
Given what appears to be a slow but increasing trend of EMR adoption, the next area that needs attention is the impact of such systems to both ensure that they are adopted and that they are helping practices to realize the expected benefits. Talmon et al [
Taking a closer look at impact, we see that it can be evaluated objectively (for example, by using proxy measures such as reduction in medication errors), but there is also a subjective component for individuals involved with EMR adoption. EMRs are expected to have positive impacts in many areas, but do providers believe this? Ultimate acceptance and use of the system is up to the provider, so there is a need to understand their point of view. Based on the trends presented above, there are two general views to consider: nonuser/preimplementation and user/postimplementation. Those who already have an EMR are able to share their perceived experienced impact of use, whereas those who don’t will have perceived expected impacts (ie, perceived benefits or concerns) that may hinder or drive adoption. One way to collect the views of users and nonusers is through the use of surveys. Surveys are commonly used in information systems evaluation [
In this review, we specifically address survey-based research studies. Surveys, or questionnaires, refer to the actual instruments used to gather data within a survey-based research study [
We briefly summarize the search strategy here, with selection flow details available in
In terms of design quality, we considered methods and reporting quality as well as the constructs for evaluation. For study methods design quality, we used the set of 9 survey methodological attributes developed by Grover et al [
For the first step in the data extraction process, we identified all survey items and questions from each paper, which we termed
Example mapping of metrics to the Clinical Adoption Framework. EHR = electronic health record, EMR = electronic medical record, HIT = health information technology.
The goal of the meta-analysis was to identify the most commonly addressed areas and combine the reported results for these areas to determine users’ and nonusers’ overall views toward EMRs. We determined that the raw data presented in some papers needed to be transformed to make them comparable. The first step was to consider whether the metric was posed as negative or positive so that the reported results could be interpreted as either negative or positive.
The surveys collected two types of data: dichotomous (ie, proportions or percentages for agreement with statements) and categorical (eg, Likert-type scale scores), and they were not reported in the same manner in all papers. For the dichotomous data, if the result was not already expressed as a proportion, we calculated a proportion estimate based on the sample size reported in the paper. As well, some papers divided results into further groupings within the nonuser and user categories, so we pooled these where possible using 95% confidence intervals to confirm an overlap for pooling. We created a series of 2 × 2 tables to organize the reported results for each metric with respect to positive and negative views for users and nonusers. Using the tables, we calculated the estimated odds of a perceived positive view for users and nonusers and then, where possible, an estimated odds ratio for a positive view for users to nonusers. For the categorical data, we redefined the scales used in the papers where needed to make mean values comparable. Most papers used a 5-point scale, but it was sometimes reversed or used different values. We transformed each scale so that it ranged from 1 (strongly negative) to 5 (strongly positive). Mean scores were recorded for nonusers and users where possible. The resulting odds calculations and mean scores were interpreted and compared with reported findings in the papers to determine overall perceived views for each selected area. Positive views leaned toward more perceived benefits of the potential use of systems, whereas negative views represented more perceived concerns or barriers that could possibly hinder use.
We selected 19 survey-based papers for inclusion in the review [
Papers reporting results for the two categories of use status.
Author (year) | Preimplementation/ |
Postimplementation/ |
Not |
Chiang et al (2008) [ |
X | X | |
DesRoches et al (2008) [ |
X | X | |
Devine et al (2010) [ |
X | ||
El-Kareh et al (2009) [ |
X | ||
Gans et al (2005) [ |
X | X | |
Johnston et al (2002) [ |
X | X | |
Kemper et al (2006) [ |
X | X | |
Leung et al (2003) [ |
X | ||
Loomis et al (2002) [ |
X | X | |
MacGregor et al (2006) [ |
X | ||
Mackenzie (2006) [ |
X | ||
Magnus et al (2002) [ |
X | ||
Menachemi et al (2007) [ |
X | X | |
Russell and Spooner (2004) [ |
X | X | |
Simon et al (2007) [ |
X | X | |
Simon et al (2008) [ |
X | ||
Simon et al (2008) [ |
X | X | |
Singh et al (2012) [ |
X | X | |
Terry (2005) [ |
X |
Many papers aimed to determine the perceived impact of adoption [
General paper characteristics.
Author |
Country | Survey/study objective(s) | Respondents | Clinical context |
Survey method | Total |
Response |
Chiang et al (2008) [ |
United States | Assess the state of EHRa use by ophthalmologists, including adoption rate and user satisfaction | Ophthalmologists | Medical practices | Web-based survey (with 2 email reminders) and telephone survey | 3796 | 15.6% (592) |
DesRoches et al (2008) [ |
United States | Assess (1) physicians’ adoption of outpatient EHRs, (2) satisfaction with such systems, (3) perceived effect of the systems on the quality of care, (4) perceived barriers to adoption | Physicians | Physicians providing direct ambulatory patient care | Mailed questionnaire (2 reminders by mail and phone); cash incentive | 5000 (4484 eligible) | 62% (2758) |
Devine et al (2010) [ |
United States | Identify prescriber and staff (end user) characteristics that would predict attitudes and behaviors toward e-prescribing adoption in the context of a preexisting EHR | Prescribers (physicians, physician assistants, nurse practitioners) and staff (nurses and medical assistants) | 3 primary care sites | Administered at the sites with 2 reminders sent via email | Total of 188 opportunities | Overall: 62% (117); prescribers: 82%; staff: 50% |
El-Kareh et al (2009) [ |
United States | Measure changes in primary care clinician attitudes toward an EMRb during the first year following implementation | Physicians, nurse practitioners, physician assistants | Ambulatory health centers | Mailed questionnaire at 1, 3, 6, and 12 months postimplementation (2 mailings and reminder emails) | 73 physicians; 10 nurse practitioners; 3 physician assistants | Month 1: 92% (79); month 2: 95% (81); month 3: 90% (76); month 12: 82% (69)c |
Gans et al (2005) [ |
United States | Assess the rate and process of adoption of information technology and EHRs by medical group practices | Group practices | Group practices with 3 or more physicians practicing together with a common billing and medical record system | Web-based and mailed survey; a subset of nonresponders were surveyed by phone | 17,195 | 21.1% (3628) |
Johnston et al (2002) [ |
China | Identify prevailing attitudes among physicians to use of computers in the clinical setting and specifically those attitudes that may be associated with the adoption of computers in practice | Physicians | Individual practices | Mailed questionnaire | 4850 | 18.5% (897) |
Kemper et al (2006) [ |
United States | (1) Measure penetration and functionality of EMRs in primary care pediatric practice, (2) identify plans for adoption of EMRs, (3) understand common barriers to adoption, (4) evaluate attitudes toward EMRs among those with and without one | Pediatricians | Office-based practice | Separate mailed questionnaires to those with and without an EMR (3 mailings); cash incentive | 1000 (901 eligible) | 58% (526) |
Leung et al (2003) [ |
China | Understand the contributory barriers and potential incentives associated with information technology implementation | Physicians | General physician population (individual and corporate settings) | Mailed survey (3 mailings and maximum of 7 phone calls) | 949 | 77% (731) |
Loomis et al (2002) [ |
United States | Investigate possible differences in attitudes and beliefs about EMRs between EMR users (early market) and nonusers (mainstream market) | Family physicians | Active members in the Indiana Academy of Family Physicians | Mailed survey (2 mailings) | 1398 | 51.7% (618 usable) |
MacGregor et al (2006) [ |
Australia | (1) Examine perception of benefits derived from information technology adoption, (2) determine whether practice size, number of patients treated, gender of practitioner, or level of computer skills of the practitioner are associated with the perception of benefits | General practitioners | General practice | Mailed questionnaire | 690 | 17.7% (122) |
Mackenzie (2006) [ |
New Zealand | Nurses’ and doctors’ perceptions of the introduction and subsequent use of the Medtech 32 clinical module | Nurses, doctors | Family planning clinics | Paper questionnaire | 132 | 57% (47 nurses and 28 doctors) |
Magnus et al (2002) [ |
England | (1) Assess general practitioners’ views on the relevance of information provided by computerized drug interaction alert systems, (2) determine the proportion of general practitioners who admit to frequently overriding alerts without properly checking them, (3) explore factors that might be associated with a tendency to override alerts | General practitioners | Primary care trust areas | Mailed questionnaire (2 mailings) | 336 | 70% (236) |
Menachemi et al (2007) [ |
United States | 1. Examine rural–urban differences in the use of various information technology applications by physicians in the ambulatory setting | Physicians (family medicine, internal medicine, pediatrics, obstetrics and gynecology) | Ambulatory settings | Mailed questionnaire (2 mailings) | 14,921 | 28.2% (4203) |
Russell and Spooner (2004) [ |
United States | (1) Determine the use of EMRs in area practices, (2) identify physicians’ attitudes adopting EMRs, particularly differences in attitudes between users and nonusers and between internal medicine and pediatric clinicians | Physicians (internal medicine and pediatrics) | Medical outpatient practices of internal medicine and pediatrics | Faxed and mailed survey (3 faxes and mailing); cash incentive | 51 internal medicine, 24 pediatrics | Internal medicine: 51% (26); pediatrics: 63% (15) |
Simon et al (2007) [ |
United States | (1) Determine the degree to which physicians used the various functions available in their EHR systems, (2) identify factors that correlate with use | Physicians | Office-based practice | Mailed survey (3 mailings with phone calls in between); cash incentive | 1921 (1884 eligible) | 71.4% (1345) |
Simon et al (2008) [ |
United States | (1) Assess the degree to which the MAeHCd practices are representative of physician’ practices statewide, (2) assess practice characteristics related to EHR adoption, prevailing office culture related to quality and safety, attitudes toward HITe, and perceptions of medical practice | Physicians | Physician office practices | Mailed survey with multiple reminders | MAeHC: 464; statewide: 1884 | MAeHC: 77% (355); statewide: 71.4% (1345)f |
Simon et al (2008) [ |
United States | (1) Determine the state of EHR adoption and the degree to which doctors with EHRs are using the functionalities of those systems, (2) assess whether practices that had not yet adopted EHRs planned to adopt such systems and when, and what barriers impeded their progress | Office practice managers | Active medical and surgical practices (hospital and non-hospital based) | Mailed questionnaire (2 mailings and 2–6 phone calls) | 1829 | 46% (847) |
Singh et al (2012) [ |
United States | (1) Examine HIT and EMR adoption and use among primary care offices across the rural–urban spectrum, (2) assess perceived benefits and perceived barriers and facilitators to adoption | Offices (targeted office medical directors or owners) | Primary care offices | Mailed survey (reminder and second mailing); cash incentive | 4669 | 21.4% (1001) |
Terry (2005) [ |
United States | Determine EHR penetration, satisfaction, and use | Medical doctors and doctors of osteopathic medicine (including family practitioners, general practitioners, internists, obstetricians and gynecologists) | Office-based practice | Mailed survey | 10,000 | Not reported |
a Electronic health record (term used in the paper).
b Electronic medical record.
c Only included month 12 data in analysis.
d Massachusetts eHealth Collaborative.
e Health information technology.
f Only included Massachusetts eHealth Collaborative data in analysis, as statewide data are reported in Simon et al [
Using the survey methodological attributes, we deemed more than half of the papers (12) to be of adequate quality (see
Quality assessment using the survey methodological attributes.
Author (year) | Criteria itemsa | Total |
||||||||
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ||
Leung et al (2003) [ |
0.5 | 1 | 1 | 1 | 1 | 0.5 | 1 | 1 | 1 | 8 |
Chiang et al (2008) [ |
0.5 | 1 | 1 | 1 | 1 | 0.5 | 1 | 1 | 0 | 7 |
Singh et al (2012) [ |
1 | 1 | 0.75 | 0 | 1 | 0 | 1 | 1 | 1 | 6.75 |
DesRoches et al (2008) [ |
0.5 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 6.5 |
Gans et al (2005) [ |
1 | 1 | 0.5 | 1 | 1 | 0 | 0 | 1 | 1 | 6.5 |
Magnus et al (2002) [ |
1 | 1 | 1 | 0 | 0.5 | 0 | 1 | 1 | 1 | 6.5 |
Devine et al (2010) [ |
0.5 | 1 | 1 | 0.25 | 1 | 1 | 0.5 | 1 | 0 | 6.25 |
Loomis et al (2002) [ |
1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 6 |
Menachemi et al (2007) [ |
0.5 | 1 | 1 | 0 | 0 | 0.5 | 0.5 | 1 | 1 | 5.5 |
Simon et al (2008) [ |
1 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 5 |
MacGregor et al (2006) [ |
1 | 0.5 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 4.5 |
Simon et al (2007) [ |
0.5 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 4.5 |
El-Kareh et al (2009) [ |
1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 4 |
Kemper et al (2006) [ |
0.5 | 1 | 0.5 | 0 | 0 | 0 | 1 | 1 | 0 | 4 |
Russell and Spooner (2004) [ |
1 | 0.5 | 0.5 | 0 | 1 | 0 | 0 | 1 | 0 | 4 |
Simon et al (2008) [ |
1 | 1 | 0.5 | 0 | 0 | 0 | 0 | 1 | 0 | 3.5 |
Johnston et al (2002) [ |
0.5 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 3.5 |
Mackenzie (2006) [ |
0.5 | 1 | 1 | 0 | 0 | 0 | 0 | 0.5 | 0 | 3 |
Terry (2005) [ |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.5 | 0 | 0.5 |
a 1 = sample selection approach, 2 = profile of sample frame, 3 = respondent characteristics, 4 = data collection methods, 5 = sample of questionnaire, 6 = validation of instrument, 7 = instrument pretest, 8 = response rate, 9 = test for nonrespondent.
b Out of a possible maximum score of 9.
During the data extraction phase of the review, we pulled reported metrics from the papers and grouped them into more general metric areas under the categories of the Clinical Adoption Framework. Only those metrics related to EMRs were extracted, which excluded general information technology. However, metrics related to use of other clinical information technology were extracted into the category of information and infrastructure. For example, Gans et al [
Several Clinical Adoption Framework categories did not have any metric areas identified: individual and groups, roles and responsibilities, added values, legislative acts, political trends, economic trends, use behavior pattern, intention to use, and participant and caregiver participation.
Most background areas corresponded to categories under the meso level, since surveys often had items pertaining to the background of respondents and practices such as practice size (number of staff), system use status, gender, future intention to use a system, and specialty. These areas and their metrics were often used not only to describe the sample, but also to determine whether there were any correlations within the reported findings. For instance, several papers found that system adoption and use was greater in larger practices [
We also extracted all other areas addressed that related to EMR adoption. These were not determined to be specifically impact related but were other associated aspects of EMR implementation that have been addressed through surveys.
The third group of addressed areas were the ones of interest for this review. These areas specifically addressed the perceived potential or actual impacts of implementing and using an EMR. As shown in
Mapping of metric areas to clinical adoption framework.
Level | Dimension | Category | Metric area | Typea | Papers |
Total number |
Meso | People | Individuals and groups | (Determined by type of respondent survey is administered to) | All | 0 | |
Personal characteristics | Age | B | 23, 26, 28, 29, 31, 32, 33, 36, 39 | 9 | ||
Gender | B | 22, 23, 24, 26, 28, 29, 30, 32, 33, 35, 36 | 11 | |||
Race and ethnic background | B | 22 | 1 | |||
Income | B | 28 | 1 | |||
Active in general practice and status | B | 35 | 2 | |||
Graduation year and years of practice | B | 22, 24, 26, 34, 35, 36 | 6 | |||
Specialty | B | 22, 23, 26, 28, 33, 34, 35, 36, 37, 38, 39 | 11 | |||
Computer skills and literacy | B | 23, 26, 30, 31, 34, 36 | 9 | |||
First to have new tests or treatments (general practice) | O | 36 | 1 | |||
Personal expectations | Comparison between paper based and electronic | I | 27 | 1 | ||
Feelings toward practice in general | O | 35, 36 | 8 | |||
Protecting physicians from personal liability for record tampering by external parties | I | 22 | 1 | |||
Roles and responsibilities | 0 | |||||
Organization | Strategy | Actively improving quality (general practice) | O | 36 | 1 | |
Local physician champion | O | 38 | 1 | |||
Physician recruitment | I | 25 | 1 | |||
Culture | Bad previous experience with an electronic record system | O | 27 | 1 | ||
Attitude toward the electronic record system | I | 22, 24, 25, 26, 27, 29 | 4 | |||
Physician and staff resistance | O | 36, 37 | 9 | |||
Isolation from colleagues (general practice) | O | 35, 36 | 2 | |||
Innovative staff (general practice) | O | 36 | 2 | |||
Information and infrastructure | Ability to interface and integrate with existing practice systems | O | 21, 25, 27, 39 | 6 | ||
Technical limits | O | 36 | 1 | |||
Use of other clinical information technology | O | 25, 37, 38 | 4 | |||
Structure and processes | Practice size (number of staff) | B | 21, 22, 25, 26, 27, 28, 29, 30, 33, 34, 35, 36, 37, 38, 39 | 18 | ||
Practice size (number of patients) | B | 24, 29, 30, 35, 36 | 5 | |||
Practice size (number of offices) | B | 38 | 1 | |||
Time spent caring for patients (hours) | B | 24, 26, 28 | 3 | |||
Practice type (eg, group) | B | 26, 28, 33 | 3 | |||
Remuneration patterns | B | 26, 28 | 2 | |||
Practice setting (eg, hospital or medical center) | B | 22, 37 | 2 | |||
Type of office | B | 23, 38 | 4 | |||
Patient population | B | 38 | 2 | |||
Practice location | B | 22, 29, 33, 36, 37, 38 | 7 | |||
Communication with general practice business suppliers | O | 30 | 1 | |||
Return on value | Business expansion | I | 30 | 1 | ||
Expense of implementation | O | 21, 22, 25, 26, 27, 28, 29, 33, 36, 37, 38 | 13 | |||
Maintenance costs | O | 21, 27, 26, 29, 33, 36 | 7 | |||
Expected return on investment | I | 22, 25, 27, 33, 34, 38, 39 | 7 | |||
Implementation | Stage | Use status | B | 21, 22, 25, 26, 27, 29, 32, 33, 34, 35, 36, 37, 38, 39 | 16 | |
Future intention to use | B | 21, 22, 23, 27, 33, 34, 37, 38, 39 | 12 | |||
Project | System development or selection | O | 21, 22, 25, 27 | 5 | ||
Time costs associated with computerization | I | 21, 25, 26, 28, 33, 36 | 7 | |||
Loss of productivity during transition | I | 22, 33, 36, 38 | 5 | |||
Entering historical data | O | 25 | 1 | |||
HISb–practice fit | Staff requirements for implementation and maintenance | O | 26, 27 | 2 | ||
Meeting needs and requirements | O | 22, 25, 27, 33, 37 | 5 | |||
Capital available for practice expansion | O | 36 | 1 | |||
Macro | Health care standards | HIS standards | Standardized medical terminology | O | 21 | 1 |
Transience of vendors | O | 27 | 1 | |||
Uniform data standards within the industry | O | 25, 33, 36 | 3 | |||
Performance standards | Evaluation of changes to improve quality (general practice) | O | 36 | 1 | ||
Quality problems (general practice) | O | 36 | 1 | |||
Procedures and systems to prevent errors (general practice) | O | 36 | 1 | |||
Practice standards | Adding to the skills of the practice | O | 30 | 1 | ||
Standardized questions to ask vendors | O | 21, 25 | 2 | |||
Model requests for proposal for contracts | O | 21, 25 | 2 | |||
Funding and incentive | Remunerations | Payment for having or using system | O | 22, 36 | 3 | |
Payment for patient survey results or clinical quality | O | 36 | 2 | |||
Direct financial assistance | O | 25, 38 | 2 | |||
Added values | 0 | |||||
Incentive programs | Financial incentives for purchase and implementation | O | 21, 22, 25, 28, 35, 38 | 6 | ||
Clarity of benefits | O | 28 | 1 | |||
Legislation, policy and governance | Legislative acts | 0 | ||||
Regulations and policies | Confidentiality | O | 22, 27, 28, 29 | 4 | ||
Access and sharing of to medical records | O | 22, 29 | 2 | |||
Intellectual property regulations | O | 28 | 1 | |||
Self-referral prohibitions regarding sharing of technology | O | 25 | 1 | |||
Government regulation requiring mandatory reporting of patient information | O | 28 | 1 | |||
Governance bodies | Vendor certification and accreditation | O | 21, 25, 38 | 3 | ||
Legal liability | O | 22 | 1 | |||
Societal, political and economic trends | Societal trends | Competitive peer pressure in terms of more practices becoming computerized | O | 28 | 1 | |
Recommendations of colleagues | O | 38 | 1 | |||
Public or patient views for computerization | O | 26, 28, 33 | 3 | |||
Political trends | 0 | |||||
Economic trends | 0 | |||||
Micro | System | Functionality | Features available and functions computerized | O | 21, 22, 25, 26, 27, 35, 39 | 9 |
Intention to computerize functions | O | 26 | 1 | |||
Features desired and functions that should be computerized | O | 21, 26, 29, 31, 32 | 10 | |||
Features used | O | 22, 26, 35, 37, 38 | 5 | |||
Features for patient use | O | 22 | 5 | |||
Performance | Reliability of system | I | 22, 34 | 2 | ||
System downtime | I | 27, 33 | 2 | |||
Frequency of potential drug interaction alerts | I | 32 | 1 | |||
How good system is in alerting for significant interactions | I | 32 | 1 | |||
Concern system would become obsolete | O | 22 | 1 | |||
Security |
|
I | 22, 25, 26, 27, 29, 33, 34, 35, 36 | 11 | ||
Information | Availability | Information storage and retrieval | I | 30 | 1 | |
Reliability of information | I | 32 | 1 | |||
|
I | 21, 22, 24, 25, 27, 35, 36, 38 | 11 | |||
Content | Value of clinical records | I | 26 | 1 | ||
Accuracy of records | I | 21, 25, 38 | 3 | |||
Drug interaction alerts providing information that is irrelevant to the patient | I | 32 | 3 | |||
Amount of information provided | I | 32 | 1 | |||
Reason for overriding alert: more faith in other sources of information | I | 32 | 3 | |||
Grading interaction alerts according to severity | I | 32 | 1 | |||
Service | Responsiveness | Training | I | 24, 29, 31, 34, 38 | 8 | |
Level of support | I | 28, 31, 36, 37 | 4 | |||
Use | Use behavior and pattern | 0 | ||||
Self-reported Use | Use of information technology for clinical management activities | O | 27 (also see functionality) | 1 | ||
Overriding alerts | I | 32 | 4 | |||
Intention to use | 0 | |||||
Satisfaction | Competency | Learning curve | O | 21, 25, 27, 28, 33 | 6 | |
User satisfaction | Overall satisfaction | I | 21, 22, 39 | 4 | ||
Annoyance caused by drug interaction alert messages | I | 32 | 1 | |||
Usefulness in prescribing | I | 23, 32 | 2 | |||
Ease of use of system or clinical module | I | 22, 23, 31, 33 | 5 | |||
Ease of use | Data entry | I | 25, 27, 29, 33, 38 | 5 | ||
Interface and customization | I | 39 | 1 | |||
Net benefits | Quality: patient safety | Primary care and medical errors | I | 27, 29 | 3 | |
Medication-related errors | I | 22, 24, 25, 35, 36, 38 | 8 | |||
Quality: appropriateness and effectiveness | Disease prevention or management | I | 22, 30, 38 | 5 | ||
Clinical decision making | I | 22, 25 | 3 | |||
Clinical functions | I | 26 | 1 | |||
Prescriptions | I | 22, 25, 30 | 3 | |||
Legibility | I | 21 | 1 | |||
Frequency of change in initial prescribing decision due to drug interaction alerts | I | 32 | 1 | |||
Awareness of information provided by drug interaction alerts | I | 32 | 2 | |||
Effect of computer use on patients’ satisfaction with care received | I | 34 | 1 | |||
|
I | 21, 22, 24, 25, 26, 27, 28, 34, 35, 36 | 10 | |||
Documentation | I | 27 | 2 | |||
Effect on medical practice; practice style | I | 39 | 1 | |||
Health outcomes |
|
I | 21, 24, 26, 27, 28, 29, 31, 35, 36 | 12 | ||
Access: ability of patients and providers to access services | Remoteness in the provision of medical care | I | 30 | 1 | ||
Patient or customer base and area of coverage | I | 30 | 1 | |||
Access: patient and caregiver participation | 0 | |||||
Productivity: efficiency | Accounting and billing or charge capture | I | 21, 25, 27, 30 | 7 | ||
Assistance in test ordering and management | I | 22, 24 | 3 | |||
Documentation time | I | 21, 24 | 3 | |||
|
I | 21, 27, 28, 30, 33, 34, 35, 36, 39 | 10 | |||
Time for medication refills | I | 38 | 1 | |||
Time for patient care | I | 24, 26, 30 | 3 | |||
Workload | I | 27, 30 | 4 | |||
Productivity: care coordination |
|
I | 22, 24, 27, 30, 35, 36 | 8 | ||
Workflow | I | 21, 25, 27, 33, 37 | 5 | |||
Productivity: net cost |
|
I | 21, 25, 27, 28, 30, 35, 36 | 10 |
a B = background, O = other, I = impact-specific area.
b Health information system.
Our mapping of metrics to the Clinical Adoption Framework resulted in no metrics for the category patient and caregiver participation, and only 2 metrics for the ability of patients and providers to access services. Together, these make up the access category of net benefits. However, 1 paper [
We found appropriateness or effectiveness and efficiency to be the most-addressed areas of impact. The italicized areas in
We synthesized reported data for the 7 top impact-specific areas using the meta-analysis approach described above. The estimated log odds for users and nonusers are graphed in
Estimated log odds for selected impact-specific areas.
Our meta-analysis showed that both users and nonusers viewed EMRs as having a positive impact on accessibility of information. For this area, 8 papers [
El-Kareh et al [
Both users and nonusers perceived a positive effect on communication with other providers. A total of 6 papers [
Reduction in costs or savings was generally seen as an important impact of implementation in the majority of papers we reviewed. For the meta-analysis we only included metrics that reported on impact on practice costs after implementation for net benefits. Several others assessed views on costs to implement and maintain systems, but we categorized these under return on value. For the meta-analysis we looked at 10 metrics across 7 papers [
In the small number of papers reviewed, improvement in business or practice efficiency was seen as a benefit of implementation. There were a total of 10 metrics reported in 9 papers [
Of all papers reporting on patient–physician relationships and communication, views appeared to be generally positive, with two exceptions. For this area, 9 papers [22,24-28,34-36,] reported on 9 related metrics. The odds ratios for 2 papers produced mixed results. One paper [35] showed positive results for both users and nonusers and a higher likelihood of a positive view for users. The other paper [26] reported more negative views for users and nonusers. However, users were about 3.5 times more likely to have a less-negative view, which aligned with the authors’ conclusion that clinical users (ie, those with one or more clinical functions computerized) were less negative about potential interference with the doctor–patient encounter. The mean scores reported for users supported more positive views.
For nonusers, we calculated negative views in 2 papers. As mentioned above, in the study of Johnston et al [
A total of 8 papers [
Nonusers had more positive views for 6 out of 7 metrics according to our calculations. The exception was in the study of Kemper et al [
One paper [
Privacy and security appeared to be an area of mixed perceptions regarding the impact of EMRs. A total of 9 papers [
When looking at only the odds of positive views for users, our calculated results for 4 out of 6 metrics were positive. For nonusers, 3 out of our 9 calculated results were positive. Mean scores were reported in 2 papers, and when these were transformed into comparable scales, they reflected positive views for both users and nonusers. For example, Russell and Spooner [
The 19 papers reviewed provided valuable insights into the state of evaluation of perceived EMR impacts through survey research methods. It is clear that evaluation from the user’s perspective is needed alongside objective measures of impact.
The first aim of this review was to determine which areas of EMR implementation in office practices have been addressed in survey-based research studies. The majority of background areas corresponded to the meso level, and other areas looking at aspects of implementation corresponded to the macro level. A possible explanation for the lack of metrics for individuals and groups and for roles and responsibilities is that these can be considered the basis for selecting respondents and would therefore not have specific metrics related to them. In most cases, the researchers predetermined that respondents would be physicians who may be users of the EMR. Added values, legislative acts, political trends, and economic trends are all in line with the macro level according to the Clinical Adoption Framework and certainly do affect EMR use but didn’t seem to be the main objective for surveys evaluating more localized views of implementation in the office. It may be no surprise that the expense of implementation is a major consideration for office practices, and so this was a common area addressed.
The impact-specific areas we focused on were mostly contained within the net benefits dimension at the micro level of the framework. While functionality was frequently addressed in the surveys, the questions seemed to mainly pertain to availability of features rather than impact. Future surveys may wish to ask not only what is available and desired but also how it had an impact on practice. For example, this could drill further down into the efficiency areas in that improvements in efficiency may be associated with specific functionality, such as electronic transfer of laboratory results into the record, which may eliminate paper or fax transmission and manual entry time. One might expect user satisfaction to be the most-addressed category from the Clinical Adoption Framework, as surveys do generally obtain views on satisfaction. However, to understand the specific areas of satisfaction, we teased out the aspects of user satisfaction into more specific categories so that this particular category only included overall satisfaction. For example, user satisfaction with respect to the system’s effect on their efficiency was mapped to productivity. Use behavior or pattern and intention to use may be encompassed by functions used and use status. Appropriateness or effectiveness and efficiency seemed to be well addressed through surveys, but there were areas of net benefits that would be expected to have had more metrics than were found. The reasons for the lack of addressed areas found for patient and caregiver participation aren’t apparent. Either this specific aspect of EMR use hasn’t been studied in depth or perhaps there is a degree of overlap between this category and others such as care coordination. This particular category may warrant further exploration.
For the second contribution of this review, to compare the perceived impacts between users and nonusers, we looked at the 7 most-addressed areas of impact in the set of papers: security and privacy, quality of patient care or clinical outcomes, patient-physician relationship and communication, communication with other providers, accessibility of records and information, business or practice efficiency, and costs or savings. For these areas, the views of users were generally more positive than those of nonusers, but even when looking at the two groups separately, we found mostly positive views for most of the impact-specific areas. In reviewing computer-based patient record systems (including EMRs), Delpierre et al [
Privacy and security was an inconclusive area, which returned the most mixed results for users and nonusers and may therefore warrant further exploration. According to Boonstra and Broekhuis [
Given that there were some noted differences in perceptions between users and nonuser with respect to most of the areas we looked at, it may be prudent to look at some of the associated background and other factors we identified in our mapping, which may account for some of the differences. Delpierre et al [
An interesting point noted in some of the papers we reviewed mirrors the classic chicken-and-egg puzzle. That is, did the more-positive views seen in users exist before they adopted the systems or did they develop them as a result of adoption? El-Kareh et al [
Lastly, in assessing the state of knowledge regarding survey-based research in HIS evaluation, there appears to be a lack of clear methodological guidance. Regarding design quality, the papers varied in terms of methods used and how they were reported. The items with the highest-quality scores were reporting a profile of the sample frame, a response rate, and a profile of respondents. The item that scored most poorly across all papers was performing a reliability or validity analysis of item measurement or adopting a validated instrument. Only 1 paper specifically reported having a test–retest reliability rate for each item. The other item that generally scored low was the use of a combination of personal, telephone, and mail data collection. In most cases, survey data were collected solely through a mailed questionnaire.
In terms of constructs for evaluation, we identified an issue related to neutrality for our review. We aimed to determine whether there was an overall positive or negative perceived impact with respect to each selected area but found that, in many cases, the individual survey items seemed to lean in one direction or the other—for example, a survey item asking respondents whether security is a barrier versus an item asking whether the respondent sees a benefit with regard to security. Both items address the construct of security, but the responses elicited by each may be affected by how they are posed. Therefore, it is possible that the constructs for evaluation used in each study may have had an impact on the negative and positive responses collected, which in turn affects how the results are reconciled across papers. In designing surveys, Trochim [
We experienced several challenges common to meta-analyses of survey-based research. At the paper selection stage, a major challenge was determining whether the paper discussed EMR use, which relied on descriptions of functionality provided in the papers. This review is based on a small set of 19 papers, and we had to further narrow down the set of papers for the meta-analysis based on the type of data reported. The biggest limitation in our review is related to the heterogeneity of all the papers included. Both Rao et al [
Although based on a small set of papers and estimated calculations, the results of this review are promising in terms of clinicians’ views for adoption of systems and suggest that clinicians are beginning to see benefits in certain areas. However, there are additional factors (eg, organizational and system) that influence perceptions, so it is important to consider a wide range of contextual factors when adopting an EMR. Our mapping identified areas corresponding to categories of the Clinical Adoption Framework that have been addressed most and other areas that haven’t been looked at yet for evaluation through surveys. Although practices with electronic record systems already in use may have more positive views of impact, our review found that those without the systems still generally have a perceived positive impact with respect to some key areas, with the exception of mixed views toward privacy and security. The findings of this review have the potential to highlight areas of concern or benefit for adoption and should be considered in future implementations and evaluations. One hope is that nonusers can look to the areas where users saw more positive perceptions as areas where they can expect to see potential benefits for adoption. As well, associations between the most-addressed areas and least-addressed areas may help practices determine where they can focus effort during implementation planning, taking into account some of the key background and other areas we’ve identified.
Survey-based research studies are a valuable way to collect users’ views for HIS evaluation. They offer data and findings that can make a significant contribution to the field. However, careful effort should be made to ensure methodological rigor and consider potential future syntheses. Our review demonstrated an approach for reconciling results presented in different ways across heterogeneous survey-based studies, which is a recognized challenge. In terms of design quality, researchers should ensure that important survey-based research study design elements are present and clearly reported using a guide such as the survey methodological attributes. As well, the constructs for evaluation should draw from an established framework or tool when possible and be expressed in a neutral manner to elicit peoples’ views.
Paper selection flow diagram.
The Clinical Adoption Framework.
Estimated odds and mean calculations for selected impact-specific areas.
electronic medical record
health information system
We wish to thank Heidi Bell, Morgan Price, Jeanette Boyd, and Colin Partridge for their earlier work and contributions to initial paper selection. Funding support for this review was provided by the Canadian Institutes for Health Research and Canada Health Infoway through the eHealth Chair Award. Funding sources had no role in the conduct of this review.
None declared.