Journal of the American Medical Informatics Association, 26(12), 2019, 1560–1565

doi: 10.1093/jamia/ocz135
Advance Access Publication Date: 7 August 2019
Research and Applications

Research and Applications

Reducing drug prescription errors and adverse drug

events by application of a probabilistic, machine-learning
based clinical decision support system in an inpatient
setting
G Segal ,1 A Segev,1 A Brom,1 Y Lifshitz,1 Y Wasserstrum,1 and E Zimlichman2
1
Internal Medicine “T,” Chaim Sheba Medical Center, Tel-Hashomer, Ramat Gan, Sackler Faculty of Medicine, Tel Aviv University,
Tel Aviv, Israel and 2Management Wing, Chaim Sheba Medical Center, Tel-Hashomer, Ramat Gan, Sackler Faculty of Medicine,
Tel Aviv University, Tel Aviv, Israel

Corresponding Author: Gad Segal, MD, Internal Medicine “T,” Tel-Hashomer, 2 Sheba Road, Ramat Gan 5265601, Israel
([email protected])
Received 18 December 2018; Revised 4 June 2019; Accepted 10 July 2019

ABSTRACT
Background: Drug prescription errors are made, worldwide, on a daily basis, resulting in a high burden of mor-
bidity and mortality. Existing rule-based systems for prevention of such errors are unsuccessful and associated
with substantial burden of false alerts.
Objective: In this prospective study, we evaluated the accuracy, validity, and clinical usefulness of medication
error alerts generated by a novel system using outlier detection screening algorithms, used on top of a legacy
standard system, in a real-life inpatient setting.
Materials and Methods: We integrated a novel outlier system into an existing electronic medical record system,
in a single medical ward in a tertiary medical center. The system monitored all drug prescriptions written during
16 months. The department’s staff assessed all alerts for accuracy, clinical validity, and usefulness. We recorded
all physician’s real-time responses to alerts generated.
Results: The alert burden generated by the system was low, with alerts generated for 0.4% of all medication
orders. Sixty percent of the alerts were flagged after the medication was already dispensed following changes
in patients’ status which necessitated medication changes (eg, changes in vital signs). Eighty-five percent of the
alerts were confirmed clinically valid, and 80% were considered clinically useful. Forty-three percent of the
alerts caused changes in subsequent medical orders.
Conclusion: A clinical decision support system that used a probabilistic, machine-learning approach based on
statistically derived outliers to detect medication errors generated clinically useful alerts. The system had high
accuracy, low alert burden and low false-positive rate, and led to changes in subsequent orders.

Key words: drug prescription errors, adverse drug events, outlier system, drug safety, patient safety, in-patient setting

INTRODUCTION billion and liability cost of more than $13 billion.2–4 It is recognized
Preventable prescription errors and adverse drug events (ADEs) are that while prescription errors and ADEs are ultimately caused by
estimated to account for 1 out of 131 outpatient deaths and 1 out of errors made by individuals, they are failures in computerized health
854 inpatient deaths in the US,1 with a direct cost of more than $20 information systems.5

C The Author(s) 2019. Published by Oxford University Press on behalf of the American Medical Informatics Association.
V
All rights reserved. For permissions, please email: [email protected]
1560
Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12 1561

Current approaches to minimize such errors include various clin- • Clinical outliers: an alert flagged when a certain medication does
ical decision support (CDS) alerting systems, but they often identify not fit the patient’s clinical profile (eg, when a hypoglycemic
only a small fraction of the errors and suffer from high incidence of drug prescribed to a patient without a diagnosis of diabetes
false alerts, resulting in “alert fatigue,” inevitably disrupting work- mellitus nor indices indicative of such a disease [such as hyper-
flows.6–8 In addition, being based on a predetermined database and glycemia or previous hypoglycemic drugs]).
rules, such CDS systems (CDSS) inherently miss error types that • Dosage outliers: an alert flagged when a certain medication dos-
have not been anticipated or programmed into the decision support age is considered as an outlier with respect to the machine-
software rules.9 Moreover, in contrast to rule-based CDSS that mon- learned dosage distribution of that medication in the population
itor the actual instance of drug prescription and no longer affect the and/or the patient’s own history (eg, rare dose, rare dosage unit,
ongoing treatment of patients, there is a need for systems that could rare frequency, rare route).
actively monitor and identify emerging ADEs along the evolution of • Drug overlap: an alert flagged when parallel treatment with 2
the hospitalization to enable early intervention and reduce harm.10 medications of the same group (or the same indication) pre-
The dynamic nature of patient status during hospitalization, due to scribed in circumstances that defy the usage of such regimens (eg,
either clinical improvement or deterioration, raises a need for more 2 types of statins).
dynamic CDSS in order to better ensure patient safety.
Asynchronous alert types include:
MedAware (Raanana, Israel) is a commercial software screening
system developed for identification and prevention of prescription • Time-dependent irregularities (asynchronous): an alert flagged
errors. It uses machine-learning algorithms to identify and intercept when changes in the patient’s profile occur after the prescription
potential medication prescription errors. In a previous study per- was made, rendering a certain medication as inappropriate or
formed on retrospective clinical data extracted from an electronic dangerous to continue (eg, when the blood pressure drops and
health system, the system generated alerts that might otherwise be continuation of anti-hypertensive medications becomes poten-
missed with existing CDS systems. The majority (75%) of these tially harmful).
alerts were found to be clinically useful.11
In this study, we evaluated the system’s performance in a real-
life setting, identifying medication errors and ADEs in a 38-bed in- Study setting and patient population
patient department of internal medicine in a tertiary medical center. In April 2016, after Sheba Medical Center’s ethics committee ap-
The system was operating on top of a legacy standard CDSS.12 The proved this study, the system was installed in a single 38-bed inter-
metrics measured were: 1) alert burden, 2) alert accuracy and clini- nal medicine department at Sheba Medical Center, a 1800-bed
cal relevance, and 3) physician’s response to the alerts. academic medical center in Israel. Initially, the system operated in a
“silent mode” for several months. During this period, drug prescrip-
tions were monitored, patients’ profiles analyzed, outliers detected,
MATERIALS AND METHODS and appropriate alerts generated. Still, alerts were not evident to the
department’s physicians. Once performance level was acceptable, in
The computerized decision support system
July 1 2016, the system switched to live mode, and the physicians
The CDSS studied in this study uses machine-learning algorithms to
started receiving the alerts in the electronic health record (EHR) en-
identify and intercept potential medication prescription errors in
vironment (Chameleon by Elad Systems, Tel Aviv, Israel) and could
real time. After analyzing historical electronic medical records, the
respond to them (ie, accept or reject both synchronous and asyn-
system automatically generates, for each medication, a computa-
chronous scenarios).
tional model that captures the population of patients that are most
The first 6 months of live operation were considered a “run-in”
likely to be prescribed a certain medication and the clinical environ-
period, in which data consistency problems were identified and han-
ment and temporal circumstances in which it is likely to be pre-
dled. These included identification of abnormal laboratory results
scribed. This model can then be used to identify prescriptions that
due to technical issues such as a hemolytic blood samples and ana-
are significant statistical outliers given patients’ clinical situations.
lytics of bodily fluids other than blood (eg, pleural and peritoneal
Examples for such outliers are medications rarely or never pre-
fluids), identification, and temporarily overlooking of medications
scribed to patients in certain situations, such as birth control pills to
on “hold”. Following the “run-in” period, the system was declared
a baby boy, or an oral hypoglycemic medication to a patient without
operational and has been active ever since. For this study, data were
diabetes. Such prescriptions are flagged by the system as potential
collected and analyzed from all inpatient EHR files for patients ad-
medication errors during real-time prescribing events. The system
mitted to Internal Medicine “T” in Sheba Medical Center, Israel, be-
intervenes at 2 points in the physician’s workflow: 1) synchronous
tween July 1, 2016 and April 30, 2018 (Figure 1).
alerts—the alert pops up during the prescribing process if the physi-
cian chooses a medication which is an outlier to the patient’s clinical
general characteristic and current clinical situation, and 2) asynchro- Types of alerts and their process of validation
nous alerts—generated after the medication order was already en- Data on the alert types, alert burden, and physicians’ responses,
tered into the system following a relevant change in the patient’s were extracted from the system’s database. Assessment of the alerts’
profile (eg, new laboratory test results or a change in vital signs that accuracy and clinical relevance were based on the physician’s
have rendered 1 of the active medications an outlier).
Synchronous alert types include:

• Time-dependent irregularities (synchronous): an alert flagged

when existing data in the patient’s profile render the prescribed
medication inappropriate or dangerous (eg, prescribing an anti-
hypertensive medication to a patient in septic shock). Figure 1. Study duration.
1562 Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

response to the alerts in real time and then validated in biweekly RESULTS
interviews with the clinical champion (GS) in the medical depart-
Alert timing and burden
ment, in which all alerts were manually reviewed and ranked
During the period of this study, there were 4533 admissions of 3160
according to their:
patients, all of whom were included in the study. Physicians pre-
• Accuracy—were there any data-related issues that caused a false scribed 78 017 medication orders. The alert burden generated by the
alarm? If yes, then accuracy ¼ zero; if not, accuracy ¼ one. For system throughout the duration of the study was low with 315 alerts
example, initially, antiplatelet agents were flagged as inappropri- on 282 prescriptions, which were 0.4% of all prescriptions or an av-
ate after thrombocytopenia was detected in fluids other than erage of 4.5 alerts every week for the whole department.
blood (accuracy ¼ 0, validity ¼ 0, usefulness ¼ 0). Such accuracy Of the alerts generated, 40% were flagged synchronously during
issues were handled by assimilating the difference between differ- the medication order process, and 60% were flagged later during
ent origins of laboratory samples into the CDSS algorithms. As a the monitoring phase, after the medication orders were already ac-
result, such inaccuracies diminished significantly. tive, following a change in the patient clinical state (ie, new lab re-
• Clinical Validity—was there clinical justification, as docu- sult, vital signs, etc).
mented in the medical record, for this medication to be pre- Time-dependent alerts were the most common (64.8%), fol-
scribed to the patient? If yes, then validity ¼ zero; if not (ie, the lowed by dosage outliers (30.2%), clinical outliers (3.8%), and drug
medication had no clinical justification and, hence, is a likely overlap (1.3%). A detailed list of the synchronous and asynchronous
error), then validity ¼ one. For example, anti-thyroid medica- alerts is presented in Tables 1 and 2, respectively. It is notable that
tions prescribed to a patient without any documentation of thy- clinical outlier alerts were generated on a wide range of medications,
roid disease and flagged as inappropriate (accuracy ¼ 1, without a “common alert.”
validity ¼ 1, usefulness ¼ 2).
• Clinical Usefulness—was the alert clinically useful to the physi- Alert accuracy and clinical relevance
cian? (Zero ¼ alert irrelevant to this patient; 1 ¼ no clinical rele- Of the alerts generated during the study period, 89% were accurate
vance; 2 ¼ clinically relevant alert even if physician overrides it; (EHR data support the alert), 85% were clinically valid (no justifica-
3 ¼ clinically relevant alert and the physician should modify tion found in the EHR to support the medication), and 80% were
treatment accordingly). For example, many patients who suffer clinically useful (alert clinically justified).
from chronic lung diseases have a high partial pressure of CO2 in
their blood. For such patients, sedative medications are poten- Physician’s response to the alerts
tially harmful since they might worsen respiratory failure. In During the 16-month duration of the study, 135 medication orders
many instances, the warning against using such medications in (48% of the accurate alerts) were stopped or modified within a short
the face of high CO2 was considered valid and clinically relevant time (median of 1 hour, interquartile range: [0.07, 4] hours) follow-
but still overridden by the attending physician who considered ing alert generation by MedAware’s system. Of these, 39% of the er-
the medication’s effects to be safely tolerated (accuracy ¼ 1, roneous medication orders were modified during the order of the
validity ¼ 1, usefulness ¼ 2). medication (synchronous flags), and 61% were modified during
monitoring phase (asynchronous flags) following a change in labora-
Thus, each alert received 3 scores (ie, [accuracy ¼ one, validity ¼
tory results or vital signs.
zero, usefulness ¼ zero] ¼ no data issues were found; however, we
The most common alerts causing a change in physician’s behav-
found a justification for the medication in the patient records. For
ior (ie, alerted medication stopped or modified) were dosage
example, an alert on a new prescription for anticoagulation, while a
alerts—followed by time-dependent alerts triggered by bradycardia,
physician’s note states that the patient has atrial fibrillation: [Accu-
elevated liver function tests, and hypotension (Table 3).
racy ¼ 1, validity ¼ 1, usefulness ¼ 3] ¼ a clinically relevant alert,
for example, a drug name mix-up resulting in prescribing chemo-
therapy to a healthy individual). Comparison to the medical center’s legacy CDSS
To accurately rank the algorithm-generated alerts, the patient’s The performance of Sheba Medical Center’s legacy CDSS was
complete longitudinal electronic medical record, was manually assessed in a recent study by Zenziper Straichman et al.12 The legacy
reviewed by the clinical champion, including structured data as CDSS alerts mostly on drug-to-drug interactions, dosage, and drug
well as unstructured data, such as physician’s notes, textual overlap. Compared to the legacy CDSS, the system generated almost
problem list, etc. a 100 times fewer alerts, which were 5 times more clinically relevant
and caused more than 8 times as many changes in prescribing
(Table 4).
The legacy CDSS generated a high alert burden (37% of pre-
Data analysis and statistical methods scriptions were flagged) with a high dismissal rate by physicians
Patient IDs were anonymized: a random number was generated for (95% of the alerts were ignored). Most of the alerts generated by
each ID and a conversion table saved in a secured location on Sheba the legacy CDSS were related to drug-to-drug interactions and dos-
Medical Center’s servers. The anonymized file was then subjected to ages. The MedAware system was deployed as an add-on to the leg-
statistical analysis, including 1) distribution of the number of alerts acy CDSS (ie, the system’s alerts were displayed in addition to the
per type (time-dependent, dosage, etc), 2) alert accuracy and clinical legacy CDS system’s, they did not replace them) and differentiated
relevance, calculated according to the clinical champion’s classifica- itself from the legacy system by using a different look-and-feel user
tion of each alert; 3) physician’s response to the alert assessed by an- interface. As the system’s alerts did not relate to drug-drug interac-
alyzing changes in prescription (hold/stop/dosage change) tions, there was minimal overlap between the systems involving only
immediately following synchronous alerts and within a few hours high-dosage alerts. In these few cases, physicians received 2 alerts on
following asynchronous alerts. the same prescription.
Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12 1563

Table 1. Synchronous alert distribution

Alert Type Most common Clinical scenario Most common medications flagged Percent
Time dependent alerts (synchronous) 47%

Hypercarbia Respiratory failure, sedation Sedatives, opioid narcotics 22%

Hyponatremia SIADH thiazide diuretics and SSRI’s 5%
Disrupted liver function Hepatitis, Sepsis and septic shock Statins 3%
Bradycardia Cardiac arrhythmia Beta blockers, calcium channel blockers 3%
Hyperkalemia Acute kidney injury Potassium sparing diuretics, Angiotensin- 3%
Converting-Enzyme inhibitors
Thrombocytopenia Sepsis and DIC Anticoagulants, anti-aggregates 3%
Hypotension Sepsis and septic shock, sedation Vasodilators 3%
Disrupted coagulation sepsis Anticoagulants 2%
tests (prolonged PT/INR)
Hypokalemia sepsis Diuretics, potassium associating resins 2%
Hypoglycemia Sepsis and septic shock Insulin, oral hypoglycemic drugs 1%
Dosage alerts 42%
High dosage 23%
Low dosage 18%
Rare unit 1%
Clinical alerts 11%

Abbreviations: DIC, disseminated intravascular coagulation; SIADH, syndrome of inappropriate antidiuretic hormone secretion; SSRI, selective serotonin reup-
take inhibitor.

Table 2. Asynchronous time-dependent alert distribution

Time-Dependent Alert Typea Most common Clinical scenario Most common medications flagged Percent

Hypercarbia Respiratory failure, sedation Sedatives, opioid narcotics 11.52%

Bradycardia Cardiac arrhythmia Beta blockers, calcium channel blockers 10.29%
Hypotension Sepsis and septic shock, sedation Vasodilators 4.90%
Thrombocytopenia Sepsis and DIC Anticoagulants, anti-aggregates 4.66%
Disrupted liver function Hepatitis, sepsis, and septic shock Statins 4.41%
Hypoglycemia Sepsis and septic shock Insulin, oral hypoglycemic drugs 4.17%
Hyperkalemia Acute kidney injury Potassium sparing diuretics, Angiotensin- 2.94%
Converting-Enzyme inhibitors
Disrupted coagulation Sepsis Anticoagulants 2.45%
tests (prolonged PT/INR)
Hypokalemia Sepsis Diuretics, potassium associating resins 1.72%
Hyponatremia SIADH Thiazide diuretics and SSRI’s 0.49%
Acute kidney injury Sepsis and septic shock Angiotensin-Converting-Enzyme inhibitors 0.25%
(elevated creatinine)
Hypercalcemia Malignancy calcium and vitamin D derivatives 0.25%
Elevated CPK levels Hepatitis, convulsions, rhabdomyolysis Statins 0.25%

Abbreviations: DIC, disseminated intravascular coagulation; SIADH, syndrome of inappropriate antidiuretic hormone secretion; SSRI, selective serotonin reup-
take inhibitor.
a
Asynchronous event/change in the patient’s profile necessitating certain medications to be flagged.

Table 3. Most common alerts causing physician behavior change There were no conflicts between the systems as each reflected
different knowledge types: the legacy system reflected what is de-
Alert Type Most common Most common Percent scribed in the drug leaflet, and MedAware reflected the statistics
Clinical scenario medications flagged based on physician’s practice.
High dosage 26%
Low dosage 18% DISCUSSION
Bradycardia Cardiac Beta blockers 9%
Prescription errors and ADEs are associated with substantial mor-
arrhythmia
Disrupted Hepatitis, sepsis Statins 8% bidity and mortality and with a significant preventable wasteful
liver and septic health care cost. Currently available CDS systems targeted to ad-
function shock dress this problem suffer from several flaws: 1) low coverage (ie,
Hypotension Sepsis and septic Vasodilators 8% they identify only a small predefined subset of errors, such as drug
shock, sedation interactions and allergies); 2) high alert burden (flagging more than
10% of prescriptions in best cases); and 3) high false-alarm rate,
1564 Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12

Table 4. Comparison between the probabilistic, machine-learning tems do not flag already active prescriptions, they would likely not
approach-based system and the legacy CDS have identified these ADEs.
As the system is based on probabilistic outliers, and, as most of
Legacy CDS The System
the alerts generated by the system are unique and not addressed by
Alert Burden (% of prescriptions) 37.10% 0.40% legacy CDS systems, it is difficult to estimate its sensitivity (ie, what
Clinically relevant (% of alerts) 16% 85% errors are missed by the system). We assume that not a few errors
Caused a change in practice (% of alerts) 5.30% 43% are missed and there is no “perfect” system that can catch all, but
Post prescribing surveillance (% of alerts) 0% 60% our belief is that getting physicians to pay attention to the alerts is
Alert types:
key to reducing harm—even at the cost of missing a few relevant
Clinical outliers No Yes
alerts. Moreover, the system does not currently intend to replace the
Time-dependent No Yes
High dosage Yes Yes
legacy systems but be an add-on to current CDS systems, adding an
Low dosage No Yes additional layer of safety.
Rare dosage unit No Yes We anticipate that the main unique impact of the system in addi-
Drug-drug interaction Yes No tion to the legacy CDSS will mostly be in:
Drug overlap Yes Yes
1. Addition of clinical outlier alerts (ie, wrong drug to the wrong
patient)
2. Postprescribing surveillance (ie, continuously evaluating the risk
exceeding 90% in most cases.12–14 These result in “alert fatigue,”
of a medication after the medication was ordered, following a
causing physicians to ignore these alerts altogether.12,13 Moreover,
change in the patient’s lab profile, vitals, etc)
most systems focus on identifying the potential errors at the pre-
3. Optimization of the alerts, resulting in low alert burden, high
scribing event but fail to identify ADEs that occur in ongoing active
clinical relevance, and physician response
prescriptions due to the frequent changes, which are characteristic
of patients during their in-hospital stay. Moreover, we anticipate that probabilistic analysis will be imple-
Indeed, research has shown that a wide variability exists regarding mented in the next-generation rule-based systems to optimize and
the ability of common vendor-developed computerized physician or- personalize their alerts based on their performance on large-scale
der entry systems with CDS to flag errors in medication prescriptions. real-time clinical data.
One study used a simulation tool designed to assess how well safety Our study is limited in scope as it was conducted in a single in-
decision support worked when applied to medication orders in com- ternal medicine department in 1 hospital. In addition, the clinical ac-
puterized order entry.15 The researchers found a large variability in curacy of alerts was determined by a single “clinical champion,” a
performance both between vendor systems and between hospitals us- potential drawback of the validation process. Moreover, the legacy
ing similar vendor systems—ranging from an ability to flag 80% of CDSS was assessed in a different study using different methodolo-
errors to some systems being able to flag only 10%. In another study, gies, which might influence the results of the comparison between
researchers found that, in community hospitals implementing com- them. Further analysis of this CDS system’s potential should be
puterized physician order entry with some level of CDS, only a mod- sought after deployment in other departments and several hospitals.
est decrease in ADEs with an increase in potential ADEs (errors that
did not cause harm) occurred.16 A return on investment analysis done
for these hospitals did not show significant financial returns.17 CONCLUSION
MedAware is a CDSS that uses statistically derived outliers to To conclude, in this study we have shown that outlier detection,
detect potential medication errors and ADEs. Here, for the first based on machine learning and statistical methods, generates clini-
time, we describe the performance of such a system in a live inpa- cally relevant alerts with good physician response and low alert fa-
tient setting following integration with the local EHR. This novel tigue in a live, busy, hospital inpatient setting.
approach targets the challenges of current CDS systems: 1) Wider
span/coverage of potential errors flagged: the system is not limited
to predefined types of errors and identifies a wide range of outliers, FUNDING
including unanticipated types of errors;11 2) Reduced burden of This research received no specific grant from any funding agency in
alerts: the system was associated with a very low alert burden and the public, commercial, or not-for-profit sectors.
flagged 0.4% of medical orders as potentially dangerous prescrip-
tion errors; 3) Decreased false-alarm rate: 80% of alerts were con-
sidered clinically justified by the physicians, and only 20% were CONTRIBUTIONS
regarded as false alarms. In subsequent analysis, most of the errors SG, EZ, and YW engaged with study design and methodology, data
related to data acquisition issues; 4) Diminished alert fatigue: due to analysis, manuscript writing, and editing. AS, AB, and YL engaged
its low alert burden and low false-alarm rate, physicians heeded with data acquisition, data analysis, and manuscript proofing.
alerts provided by the system (evidently, they differentiated between
such alerts and alerts flagged by the legacy system that continued
working in the background). This is evident by the fact that prescrib- CONFLICT OF INTEREST STATEMENT
ing physicians changed their behavior and modified/stopped the None declared.
medical orders following 43% of the alerts; 5) Surveillance of post
prescribing events: the system monitors changes in the patient record
to identify potentially emerging ADEs which may be associated with REFERENCES
an active medication. This resulted in 60% of the alerts generated 1. National Academies Press. To Err Is Human. Washington, DC: National
after the medication was already prescribed. As most current sys- Academies Press; 2000.
Journal of the American Medical Informatics Association, 2019, Vol. 26, No. 12 1565

2. James JT. A new, evidence-based estimate of patient harms associated 11. Schiff GD, Volk LA, Volodarskaya M, et al. Screening for medication
with hospital care. J Patient Saf 2013; 9 (3): 122–8. errors using an outlier detection system. J Am Med Inform Assoc 2017; 24
3. Mello MM, Chandra A, Gawande AA, Studdert DM. National costs of (2): 281–7.
the medical liability system. Health Aff (Millwood) 2010; 29 (9): 12. Zenziper Straichman Y, Kurnik D, Matok I, et al. Prescriber response to
1569–77. computerized drug alerts for electronic prescriptions among hospitalized
4. Bishop TF, Ryan AM, Ryan AK, Casalino LP. Paid malpractice claims for patients. Int J Med Inform 2017; 107: 70–5.
adverse events in inpatient and outpatient settings. JAMA 2011; 305 (23): 13. Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates
2427–31. in the meaningful use era: no evidence of progress. Appl Clin Inform
5. Velo GP, Minuz P. Medication errors: prescribing faults and prescription 2014; 5 (3): 802–13.
errors. Br J Clin Pharmacol 2009; 67 (6): 624–8. 14. Tolley CL, Slight SP, Husband AK, Watson N, Bates DW. Improving
6. McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision medication-related clinical decision support. Am J Heal Pharm 2018; 75
support alert appropriateness: a review and proposal for improvement. (4): 239–46.
Ochsner J 2014; 14 (2): 195–202. 15. Metzger J, Welebob E, Bates DW, Lipsitz S, Classen DC. Mixed results in
7. Hoffman S, Podgursky A. Drug-drug interaction alerts: emphasizing the the safety performance of computerized physician order entry. Health Aff
evidence. St Louis Univ J Health Law Pol 2012; 5: 297–310. 2010; 29: 655–63.
8. Glassman PA, Simon B, Belperio P, Lanto A. Improving recognition of 16. Leung AA, Schiff G, Keohane C, et al. Impact of vendor computerized
drug interactions: benefits and barriers to using automated drug alerts. physician order entry on patients with renal impairment in community
Med Care 2002; 40 (12): 1161–71. hospitals. J Hosp Med 2013; 27 (7): 801–7.
9. van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A. Turning off fre- 17. Zimlichman E, Keohane C, Franz C, et al. Return on investment for ven-
quently overridden drug alerts: limited opportunities for doing it safely. J dor computerized physician order entry in four community hospitals: the
Am Med Inform Assoc 2008; 15 (4): 439–48. importance of decision support. Jt Comm J Qual Patient Saf 2013; 39 (7):
10. AHRQ Agency for Healthcare Research and Quality. Reducing and Pre- 312–8.
venting Adverse Drug Events to Decrease Hospital Costs. Rockville, MD:
AHRQ Agency for Healthcare Research and Quality; 2001.