Published on 16.05.16 in Vol 5, No 2 (2016): Apr-Jun
An Observational Study to Evaluate the Usability and Intent to Adopt an Artificial Intelligence–Powered Medication Reconciliation Tool
Background: Medication reconciliation (the process of creating an accurate list of all medications a patient is taking) is a widely practiced procedure to reduce medication errors. It is mandated by the Joint Commission and reimbursed by Medicare. Yet, in practice, medication reconciliation is often not effective owing to knowledge gaps in the team. A promising approach to improve medication reconciliation is to incorporate artificial intelligence (AI) decision support tools into the process to engage patients and bridge the knowledge gap.
Objective: The aim of this study was to improve the accuracy and efficiency of medication reconciliation by engaging the patient, the nurse, and the physician as a team via an iPad tool. With assistance from the AI agent, the patient will review his or her own medication list from the electronic medical record (EMR) and annotate changes, before reviewing together with the physician and making decisions on the shared iPad screen.
Methods: In this study, we developed iPad-based software tools, with AI decision support, to engage patients to “self-service” medication reconciliation and then share the annotated reconciled list with the physician. To evaluate the software tool’s user interface and workflow, a small number of patients (10) in a primary care clinic were recruited, and they were observed through the whole process during a pilot study. The patients are surveyed for the tool’s usability afterward.
Results: All patients were able to complete the medication reconciliation process correctly. Every patient found at least one error or other issues with their EMR medication lists. All of them reported that the tool was easy to use, and 8 of 10 patients reported that they will use the tool in the future. However, few patients interacted with the learning modules in the tool. The physician and nurses reported the tool to be easy-to-use, easy to integrate into existing workflow, and potentially time-saving.
Conclusions: We have developed a promising tool for a new approach to medication reconciliation. It has the potential to create more accurate medication lists faster, while better informing the patients about their medications and reducing burden on clinicians.
Interact J Med Res 2016;5(2):e14
Medication error is one of the most common patient safety issues in the health care system [- ]. Medication error is a major contributor to preventable adverse drug events (ADEs), which cause more than 3.5 million physician office visits, an estimated 1 million emergency department visits, and approximately 125,000 hospital admissions each year [ ]. The national cost of ADEs is estimated to be $3.5 billion dollars every year [ ]. Medication reconciliation is an intervention designed to reduce medication errors and ADEs. It is a process of creating the most accurate list possible of all medications that a patient is taking—including drug name, dosage, frequency, and route—and comparing that list against the existing medication list in the patient record [ ].
Due to its early promise and large potential impact, medication reconciliation is mandated and reimbursed throughout the health care system. The Joint Commission specified medication reconciliation across the care continuum as a National Patient Safety Goal . The Institute for Healthcare Improvement has medication reconciliation as part of its 5 Million Lives Campaign [ ]. Medicare reimburses for medication reconciliation (Current Procedural Terminology code 111F) and has it as part of the requirement for Electronic Medical Record (EMR) Meaningful Use certification [ ].
However, despite the high hopes, 2 large meta reviews revealed that medication reconciliation only had limited success in reducing ADEs in hospital settings [, ]. Further investigations indicated that multidisciplinary team-based medication reconciliation approaches tend to work best. For example, a study showed a reduction of medication discrepancy from 89% to 49% by involving everyone, including patient, front desk staff, nurse, and physician, in the medication reconciliation process [ ]. Specifically, 2 factors have emerged as important to medication reconciliation success. It includes the following:
- Patient’s knowledge about his own medication use is of crucial importance for successful reconciliation [ ]. In multiple studies, patients have demonstrated that they can identify discrepancies in their own medication lists with assistance from electronic tools [ , ].
- The clinician team’s clinical knowledge gap is a major barrier for successful medication reconciliation [ ]. For instance, nurses are inadequately trained on pharmacy subjects [ ]. Pharmacist-led or nurse-pharmacist medication reconciliation demonstrated greater improvement in clinical outcomes [ , , ]. It is also shown that providing specialized medication reconciliation training to medical residents could reduce medication discrepancy [ ].
A promising approach to address both factors is to engage patients and supplement clinical knowledge using artificial intelligence (AI)-based electronic systems. The AI system can guide the patient to review his own medication lists and then to flag potential issues for the physician to review. For instance, the AI can understand hundreds of nonstandard abbreviations in handwritten prescriptions and can discern medications with multiple names that could confuse even expert clinicians . It is shown that computerized systems have the capability to process medication terminologies [ ].
The technology solution could also facilitate patient education by automatically showing the medication indications, side effects, prices, and other relevant information to patients. Such information is crucial for patients to make informed decisions on their medications. It could also flag potential discrepancies and prompt the patient to ask the clinicians in the care team to explain.
Electronic decision support tools have already shown promise in improving medication reconciliation [- ]. A portable AI-powered decision support tool can not only help the patient identify and manage medications but also enable better team work because everyone can review the information on the screen together. Incorporating patients in the medical decision-making process has resulted in increased patient’s commitment and understanding of treatment plans, improved adherence, and increased patient satisfaction [ , ]. A recent survey indicated that patients themselves are very interested in using tablet devices in clinics to exchange information with the clinicians [ ]. In primary care setting, having the patient do the work also has added benefits of saving clinician time and minimizing workflow interruptions.
In this paper, we will discuss how we built an AI-powered iPad tool to improve medication reconciliation and then evaluate the solution in real-world primary care settings.
The software we developed for medication reconciliation is a Web-based app optimized for touch screen tablets, such as the Apple iPad. It takes a team approach to organize its workflow: The patient provides information on his up-to-date medication use and flags medications that he wants to discuss. The clinicians review the list together with the patient to determine what the most accurate list is. During the shared-screen review process, the clinicians can answer patient's questions about each medication and provide an opportunity to adjust certain medications to address patient concerns such as cost and side effects.
The app allows the front desk administrator to load a list of currently scheduled patients on the screen. The front desk administrator can tap on a patient name and then hand the tablet device to the patient at the time of check-in together with other medical or financial forms the patient needs to fill out.
The Use of AI Assistance
The intervention tool uses AI in 3 ways. First, it contains a machine learning (ML) module to recognize and parse prescription instructions written in natural language. The module is trained on over 2000 real-world prescription records from EMRs and achieved an error rate of less than 2%. Because all parsed medication records are again reviewed by both the patient and physician, the AI is truly an assistant and will not endanger patient safety. The details of the ML and natural language processing algorithms are beyond the scope of this paper.
Second, the tool aggregates multiple medication databases from National Institutes of Health, Food and Drug Administration (FDA), and commercial vendors to figure out equivalent or conflicting prescriptions in its medication reconciliation algorithm. The ability to organize data into knowledge representations and apply the knowledge to deduce meaning from real-world data is a hallmark of AI algorithms.
Finally, the tool can converse with the patient to go through the medication review process. The conversation is semistructured and rule-based.
The patient receives the tablet and confirms that his or her name is indeed on the screen. Following the screen instructions, the patient now starts the medication reconciliation process for the clinic visit.
The patient’s current medication list is loaded by the app from the clinic’s connected EMR. The AI parses each natural language prescription record to break it into medication name, formulation, dose, and frequency information. The app will then present the medications to the patient one screen at a time (). The screen shows the medication name, dose, instructions, and an image of the pill or package, if available. The patient has the options to do the following:
- Confirm that the patient is taking this medication as instructed.
- Remove the medication from the list because the patient is not taking it, or no longer taking it, or has not filled it.
- Mark the medication as uncertain if the patient is unsure whether he or she is taking it.
- Edit the strength and dose of the medication if those are incorrect.
After the patient reviews all his or her current medications in the EMR, the app asks the patient if there are more to add. If the user chooses to add more medications, the app will show an auto-complete text field for the patient to add medications by just spelling out a few letters in the name. Once the patient chooses a medication name, the app lists all brand names and generic variations of this medication, with pill pictures if available. The user can select a medication from the list and then specify the number of pills taken every time and number of times the pills are taken per day. This process is depicted in. The patient repeats this process until he or she has no more medications to add.
Once the patient is done, the app shows a “reconciled” list on the screen (). When building this list, the AI goes through each medication, determines its generic active ingredients, and then compares them with other medications on the list. All duplicates will be automatically flagged by the AI. The AI-built reconciled list highlights medications that have been deleted, altered, or added. It also flags medications that the patient has marked as “unsure.” The patient can review the list for accuracy, and he or she can still add, delete, or make changes to any medication on the list. Notice that the screen shown in follows the patient into the clinical interview or encounter. The physician will work on this screen with the patient later.
After the patient is done reviewing his or her medication list, he or she might be still waiting to see the physician. The reconciled list provides a content page of educational material for each medication for the patient to learn more about his or her own medications, if the patient is interested. The educational page contains the following details:
- Link to FDA-structured label of the medication.
- Link to Medline Plus consumer content for selected medications.
- Link to GoodRx for selected medication so that the patient can look up cheapest prices for the medication in nearby local market.
Next, the patient brings the reconciled list on the tablet into the examination room. When the physician comes in, he or she will review the list with the patient together on the shared tablet screen (). They will discuss why certain medications are stopped (eg, side effect, cost, and so forth) and make shared decisions on whether to change or discontinue certain medications. Once they are finished, the physician can close this screen and send the updated list back to the EMR.
After medication reconciliation, the clinical encounter happens as usual. The physician will examine the patient and potentially adjust medications based on new patient complaints. After the clinical encounter is over, the patient is likely to have an updated prescription. The patient will then follow on-screen instructions on the tablet to load the updated list from the EMR. The screen now shows the difference between the patient’s previous reconciled list and the new prescription list. It shows which medications are removed or added. The patient should now review this list and confirm whether it is indeed correct to his or her understanding. If the patient spots any issue at this time, he or she should request to clarify with the physician or a nurse before leaving the office.
The patient also has the option to access his or her medication lists from home. The patient will receive a secure message containing his or her login credentials in the clinic’s patient portal.
To evaluate the usability of the solution, an observational study was conducted in a South Austin Family Practice clinic. We followed and observed 10 patients with complex medications over a span of 2 days. We examined whether they were able to complete the core medication reconciliation tasks on the tablet device and structured the observation using a software usability heuristic checklist.
The patients are selected by the office administrator the day before the study based on the current appointment schedule. The patient selection criteria are as follows:
- Each patient must have at least 5 active prescribed medications.
- The patient is previously scheduled to see Dr. Poonawala in the clinic on the 2 study dates.
- Medicare patients are prioritized for recruitment.
The selected patient is recruited at the time to check-in at the front desk. If the patient indicates that he or she is willing to participate, reads or writes English, and knows how to use a tablet device, the study coordinator will go over the informed consent with her. After the patient reviews and agrees to the consent, a tablet device will be provided to the patient with his or her personal and medical history information already loaded to start the study.
The study coordinator stays with the patient and passively observes how the patient is using the device. The study coordinator observes the patient’s interaction with the app and then notes any problems or difficulties the patient encounters. The patient issues are categorized according to 10 heuristics  commonly used to evaluate computer user interfaces.
In addition to the observational study conducted by a heuristic evaluator, the patient was directly asked at the end of the session how he or she perceives the usability of the product. We used the standard and widely used Computer System Usability Questionnaire  for this purpose. The patients answer those questions from SurveyMonkey Web site on the same tablet device they just used.
To evaluate the patients’ readiness to adopt and use the solution in future clinic visits, we surveyed them using a customized unified theory of acceptance and use of technology (UTAUT) instrument . The UTAUT is designed to evaluate factors that could facilitate or impede future adoption of the technology. Our small sample of users does not have the statistical power to evaluate the entire model, but the survey answers gave us important suggestions on how to improve the product and deployment process to foster future adoption.
At the end of the clinical days, the researchers conducted one-to-one interviews with the physician and nurse who participated in the study. The one-to-one interview is unstructured , and it is designed to prompt free-form suggestions from the clinicians on how the tool impacted their clinical workflow. Specifically, we asked questions about whether the tool saved time for the clinicians and whether the clinicians feel the tool helped improve patient care or reduce potential medication errors.
The clinician responses were recorded in interview notes and are summarized in the Results section.
The medication reconciliation tool is successfully deployed in the clinic and used by both patients and clinicians in the 2 days. In this section, we report results from the observation study, patient survey, and clinician interviews.
We collected valid data from 10 patients in the study. The patients have the following characteristics:
- Between 50 and 87 years old.
- Seven are women, and 3 are men.
- Have between 5 and 16 active medications each before the appointment.
The 10 patients have a combined 92 active medications before the appointment. After review, they changed 26 medication records, representing 28% of the total records. All the patients changed at least 1 medication record. This indicates widespread problem in medication discrepancy and highlights the patient’s essential role in providing accurate medication information.
After the initial medication review and consultation with the physician, only 1 patient identified problem with the new prescription list at discharge. This indicates that the reconciliation has successfully brought the patient and clinicians on the same page with regard to medications.
Learning About Medications
Although the software tool provides extensive materials for the patient to learn more about their medications, including indications, side effects, and prices, no patient has taken advantage of these features. It appears that the clinic environment is not well suited for reading long articles and research reports. Several patients remarked that short video clips will be much more appealing while they wait.
In addition, patients have also indicated that they are more likely to read about their medications after they go home. This indicates the need to follow-up with the patient with email or other content after they go home to complete the patient education cycle.
Heuristic Evaluation of the User Interface
Key usability problems identified in the heuristic evaluation are categorized by the heuristics ().
|Aesthetic and minimalist design|
|Consistency and standards|
|Documentation and help|
|Flexibility and efficiency of use|
|Help user recognize, diagnose, and recover from errors|
|Match between system and the real world|
|Recognition rather than recall|
|User control and freedom|
|Visibility of system status|
Almost all patients saw themselves as "not good at technology" and were initially uneasy about using the iPad. However, all the patients were able to use the core features of the tool with minimal help.
On the scale from 1 to 7 (7 being the most easy-to-use), the patients rated the tool a 6.5—very easy to use. The user satisfaction score for the tool is 6.0 of 7. Patients strongly agreed that the tool is a good idea (score 6.5 of 7).
Intent for Future Adoption
The patients showed strong interests in using the tool for future medical visits. When the patients were asked whether they will use the tool again in this clinic, 9 of 10 patients answered Yes. Of 10 patients, 8 answered that they will use it in a different clinic. However, only 5 of 10 patients would use the app to manage medications at home. We dictate that this reflects the user’s perception of limited utility for medication management at home without professional help nearby.
The tool is thought to be easy-to-use and have high perceived usefulness. However, at least 1 patient questioned the value of the tool to patients. She remarked that the tool seems to save time for clinicians but does not save time for the patient because the patient now needs to do more work. We dictate that this objection can be mitigated by emphasizing the tool’s benefit to patients in terms of reducing potential medication errors and harms.
Seven of 10 patients indicated that having someone to help in the clinic is a critical factor for them to adopt the tool. It is also true that the patient can more easily act on the results from the tool inside a clinic because the prescribing physician is on hand to review the results and potentially make changes to the prescription.
Clinician Interview Results
The general consensus is that the tool saved the nurse time by reducing their questions about patient medications. However, making changes to medication on the tool is harder than doing it on paper. But, that is probably a reflection of the nurses being unfamiliar with the tool with only 10 patients using it.
During the pilot, the tool does not save time for the physician although increased familiarity with the tool could result in time-saving in the future. The physician reported more confidence in the accuracy of the patients’ medication list reported by the tool.
The tablet-based medication reconciliation tool in medical clinic is well accepted by patients and clinicians. The AI component facilitated the patients themselves to recognize their own medications and report discrepancies for the clinicians to review. It has potentials to improve medication accuracy and reduce medication errors in the clinic.
Conflicts of Interest
Michael Yuan is the CEO of Ringful Health, which creates the software tool and could benefit from its commercial use.
- Bates DW, Spell N, Cullen DJ, Burdick E, Laird N, Petersen LA, et al. The costs of adverse drug events in hospitalized patients. Adverse Drug Events Prevention Study Group. JAMA 1997;277(4):307-311. [Medline]
- Barnsteiner J. Medication Reconciliation. In: Hughes RG, editor. Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville, MD: Agency for Healthcare Research & Quality; 2008.
- Harrison R, Walton M, Manias E, Smith-Merry J, Kelly P, Iedema R, et al. The missing evidence: a systematic review of patients' experiences of adverse events in health care. Int J Qual Health Care 2015 Dec;27(6):424-442. [CrossRef] [Medline]
- U.S. Department of Health and Human Services. 2014. National Action Plan for Adverse Drug Event Prevention URL: http://health.gov/hcq/pdfs/ade-action-plan-508c.pdf [accessed 2016-04-20] [WebCite Cache]
- Institute OM. Preventing medication errors: quality chasm series. The National Academies Press 2006.
- The JC. sentinel event alert. Medication reconciliation URL: http://www.jointcommission.org/assets/1/18/SEA_35.pdf [accessed 2016-04-20] [WebCite Cache]
- The Joint Commission. 2015. National Patient Safety Goals URL: http://www.jointcommission.org/assets/1/6/2015_NPSG_HAP.pdf [accessed 2016-04-20] [WebCite Cache]
- Institute for Healthcare Improvement. 2006. 5 million lives campaign URL: http://www.ihi.org/about/Documents/5MillionLivesCampaignCaseStatement.pdf [accessed 2016-04-20] [WebCite Cache]
- CMS. 2014. Eligible professional meaningful Use menu set measures: Measure 6 of 9 URL: https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/downloads/7_Medication_Reconciliation.pdf [accessed 2016-04-20] [WebCite Cache]
- Mueller SK, Sponsler KC, Kripalani S, Schnipper JL. Hospital-based medication reconciliation practices: a systematic review. Arch Intern Med 2012 Jul 23;172(14):1057-1069 [FREE Full text] [CrossRef] [Medline]
- Kwan JL, Lo L, Sampson M, Shojania KG. Medication reconciliation during transitions of care as a patient safety strategy: a systematic review. Ann Intern Med 2013 Mar 5;158(5 Pt 2):397-403. [CrossRef] [Medline]
- Bayoumi I, Howard M, Holbrook AM, Schabort I. Interventions to improve medication reconciliation in primary care. Ann Pharmacother 2009 Oct;43(10):1667-1675. [CrossRef] [Medline]
- Nassaralla CL, Naessens JM, Hunt VL, Bhagra A, Chaudhry R, Hansen MA, et al. Medication reconciliation in ambulatory care: attempts at improvement. Qual Saf Health Care 2009 Oct;18(5):402-407. [CrossRef] [Medline]
- Lesselroth BJ, Felder RS, Adams SM, Cauthers PD, Dorr DA, Wong GJ, et al. Design and implementation of a medication reconciliation kiosk: the Automated Patient History Intake Device (APHID). J Am Med Inform Assoc 2009 May;16(3):300-304 [FREE Full text] [CrossRef] [Medline]
- Heyworth L, Paquin AM, Clark J, Kamenker V, Stewart M, Martin T, et al. Engaging patients in medication reconciliation via a patient portal following hospital discharge. J Am Med Inform Assoc 2014 Feb;21(e1):e157-e162 [FREE Full text] [CrossRef] [Medline]
- van SN, Zegers M, Natsch S, Wollersheim H. Medication reconciliation at hospital admission and discharge: insufficient knowledge, unclear task reallocation and lack of collaboration as major barriers to medication safety. BMC Health Serv Res 2012;12:170 [FREE Full text] [CrossRef] [Medline]
- Adhikari R, Tocher J, Smith P, Corcoran J, MacArthur J. A multi-disciplinary approach to medication safety and the implication for nursing education and practice. Nurse Educ Today 2014 Feb;34(2):185-190. [CrossRef] [Medline]
- Feldman LS, Costa LL, Feroli ER, Nelson T, Poe SS, Frick KD, et al. Nurse-pharmacist collaboration on medication reconciliation prevents potential harm. J Hosp Med 2012 Feb;7(5):396-401. [CrossRef] [Medline]
- Graabaek T, Kjeldsen LJ. Medication reviews by clinical pharmacists at hospitals lead to improved patient outcomes: a systematic review. Basic Clin Pharmacol Toxicol 2013 Jun;112(6):359-373 [FREE Full text] [CrossRef] [Medline]
- Arundel C, Logan J, Ayana R, Gannuscio J, Kerns J, Swenson R. Safe Medication Reconciliation: An Intervention to Improve Residents' Medication Reconciliation Skills. J Grad Med Educ 2015 Sep;7(3):407-411. [CrossRef] [Medline]
- Grissinger M. Multiple brand names for the same generic drug can cause confusion. P T 2013 Jun;38(6):305 [FREE Full text] [Medline]
- Saitwal H, Qing D, Jones S, Bernstam EV, Chute CG, Johnson TR. Cross-terminology mapping challenges: a demonstration using medication terminological systems. J Biomed Inform 2012 Aug;45(4):613-625 [FREE Full text] [CrossRef] [Medline]
- Hesselink G, Schoonhoven L, Barach P, Spijker A, Gademan P, Kalkman C, et al. Improving patient handovers from hospital to primary care: a systematic review. Ann Intern Med 2012 Sep 18;157(6):417-428. [CrossRef] [Medline]
- Bozzo S, PA, Bernstam, EV, Markowitz E, Johnson, TR, Herskovic, JR. Automated medication reconciliation and complexity of care transitions. 2011 Presented at: AMIA Annual Symposium Proceedings; 2011; Washington D.C p. 1252-1260.
- Plaisant C, Chao T, Wu J, Hettinger, AZ, Herskovic J, Johnson, TR, et al. Twinlist: Novel user interface designs for medication reconciliation, Proc. American Medical Informatics Assn (AMIA 2013), Washington, DC (November 2013) 2013 Nov:1150.
- Parchman ML, Zeber JE, Palmer RF. Participatory decision making, patient activation, medication adherence, and intermediate clinical outcomes in type 2 diabetes: a STARNet study. Ann Fam Med 2010;8(5):410-417 [FREE Full text] [CrossRef] [Medline]
- Wilson SR, Strub P, Buist AS, Knowles SB, Lavori PW, Lapidus J, Better Outcomes of Asthma Treatment (BOAT) Study Group. Shared treatment decision making improves adherence and outcomes in poorly controlled asthma. Am J Respir Crit Care Med 2010 Mar 15;181(6):566-577 [FREE Full text] [CrossRef] [Medline]
- Patel V, Hale TM, Palakodeti S, Kvedar JC, Jethwani K. Prescription Tablets in the Digital Age: A Cross-Sectional Study Exploring Patient and Physician Attitudes Toward the Use of Tablets for Clinic-Based Personalized Health Care Information Exchange. JMIR Res Protoc 2015 Oct;4(4):e116 [FREE Full text] [CrossRef] [Medline]
- Nielsen J. ,Molich, R. Heuristic evaluation of user interfaces, Proc. ACM CHI'90 Conf. (Seattle, WA. 1-5 April) 1990:249-256.
- Lewis J. R. IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use.International Journal of Human-Computer Interaction 1995;7(1):57-78.
- Venkatesh V. , Morris, M. , Davis, G., Davis, F. User Acceptance of Information Technology, Toward a Unified View. MIS Quarterly 2003;27:425-478.
- Usability.gov. Individual Interviews URL: http://www.usability.gov/how-to-and-tools/methods/individual-interviews.html [accessed 2016-04-12] [WebCite Cache]
|AI: artificial intelligence|
|ADEs: adverse drug events|
|EMR: electronic medical record|
|ML: machine learning|
|UTAUT: unified theory of acceptance and use of technology|
Edited by G Eysenbach; submitted 19.12.15; peer-reviewed by L McCullagh, V Patel; comments to author 04.02.16; revised version received 07.03.16; accepted 16.03.16; published 16.05.16
©Ju Long, Michael Juntao Yuan, Robina Poonawala. Originally published in the Interactive Journal of Medical Research (http://www.i-jmr.org/), 16.05.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.