Ahrq 15 Hie PDF

Evidence Report/Technology Assessment
Number 220
Health Information
Exchange
Evidence-Based Health Information

Practice Technology
Evidence Report/Technology Assessment
Number 220
Health Information Exchange
Prepared for:
Agency for Healthcare Research and Quality
U.S. Department of Health and Human Services
540 Gaither Road
Rockville, MD 20850
www.ahrq.gov
Contract No. 290-2012-00014-I
Prepared by:
Pacific Northwest Evidence-based Practice Center
Portland, OR
Investigators:
William Hersh, M.D.
Annette Totten, Ph.D.
Karen Eden, Ph.D.
Beth Devine, Ph.D., Pharm.D., M.B.A.
Paul Gorman, M.D.
Steve Kassakian, M.D.
Susan S. Woods, M.D.
Monica Daeges, B.A.
Miranda Pappas, M.A.
Marian S. McDonagh, Pharm.D.
AHRQ Publication No. 15(16)-E002-EF

December 2015
This report is based on research conducted by the Pacific Northwest Evidence-based Practice
Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ),
Rockville, MD (Contract No. 290-2012-00014-I). The findings and conclusions in this document
are those of the authors, who are responsible for its contents; the findings and conclusions do not
necessarily represent the views of AHRQ. Therefore, no statement in this report should be
construed as an official position of AHRQ or of the U.S. Department of Health and Human
Services.
None of the investigators have any affiliations or financial involvement that conflicts with
the material presented in this report.
The information in this report is intended to help health care decisionmakers—patients and
clinicians, health system leaders, and policymakers, among others—make well informed
decisions and thereby improve the quality of health care services. This report is not intended to
be a substitute for the application of clinical judgment. Anyone who makes decisions concerning
the provision of clinical care should consider this report in the same way as any medical
reference and in conjunction with all other pertinent information, i.e., in the context of available
resources and circumstances presented by individual patients.
This report is made available to the public under the terms of a licensing agreement between the
author and the Agency for Healthcare Research and Quality. This report may be used and
reprinted without permission except those copyrighted materials that are clearly noted in the
report. Further reproduction of those copyrighted materials is prohibited without the express
permission of copyright holders.
AHRQ or U.S. Department of Health and Human Services endorsement of any derivative
products that may be developed from this report, such as clinical practice guidelines, other
quality enhancement tools, or reimbursement or coverage policies, may not be stated or implied.
This report may periodically be assessed for the currency of conclusions. If an assessment is
done, the resulting surveillance report describing the methodology and findings will be found on
the Effective Health Care Program Web site at www.effectivehealthcare.ahrq.gov. Search on the
title of the report.
Persons using assistive technology may not be able to fully access information in this report. For
assistance contact [email protected].
Suggested citation: Hersh W, Totten A, Eden K, Devine B, Gorman P, Kassakian S, Woods SS,
Daeges M, Pappas M, McDonagh MS. Health Information Exchange. Evidence
Report/Technology Assessment No. 220. (Prepared by the Pacific Northwest Evidence-based
Practice Center under Contract No. 290-2012-00014-I.) AHRQ Publication No. 15(16)-E002-EF.
Rockville, MD: Agency for Healthcare Research and Quality; December 2015.
www.effectivehealthcare.ahrq.gov/reports/final.cfm.
ii
Preface
The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based
Practice Centers (EPCs), sponsors the development of systematic reviews to assist public- and
private-sector organizations in their efforts to improve the quality of health care in the United
States. These reviews provide comprehensive, science-based information on common, costly
medical conditions, and new health care technologies and strategies.
Systematic reviews are the building blocks underlying evidence-based practice; they focus
attention on the strength and limits of evidence from research studies about the effectiveness and
safety of a clinical intervention. In the context of developing recommendations for practice,
systematic reviews can help clarify whether assertions about the value of the intervention are
based on strong evidence from clinical studies. For more information about AHRQ EPC
systematic reviews, see www.effectivehealthcare.ahrq.gov/reference/purpose.cfm.
AHRQ expects that these systematic reviews will be helpful to health plans, providers,
purchasers, government programs, and the health care system as a whole. Transparency and
stakeholder input are essential to the Effective Health Care Program. Please visit the Web site
(www.effectivehealthcare.ahrq.gov) to see draft research questions and reports or to join an
email list to learn about new program products and opportunities for input.
We welcome comments on this systematic review. They may be sent by mail to the Task
Order Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither Road,
Rockville, MD 20850, or by email to [email protected].
Richard G. Kronick, Ph.D. Arlene S. Bierman, M.D., M.S.

Director Director
Agency for Healthcare Research and Quality Center for Evidence and Practice Improvement
Agency for Healthcare Research and Quality
Stephanie Chang, M.D., M.P.H. Edwin Lomotan, M.D.

Director, EPC Program Task Order Officer
Center for Evidence and Practice Improvement Center for Evidence and Practice Improvement
Agency for Healthcare Research and Quality Agency for Healthcare Research and Quality
iii
Acknowledgments
The authors gratefully acknowledge the following individuals for their contributions to this
project: Andrew Hamilton, M.L.S., M.S., for conducting literature searches and Spencer Dandy,
B.S., for assistance with preparing this report (both are located at the Oregon Health & Science
University) and Jon White, M.D., Task Order Officer at the Agency for Healthcare Research and
Quality.
Key Informants
In designing the study questions, the EPC consulted several Key Informants who represent
the end-users of research. The EPC sought the Key Informant input on the priority areas for
research and synthesis. Key Informants are not involved in the analysis of the evidence or the
writing of the report. Therefore, in the end, study questions, design, methodological approaches,
and/or conclusions do not necessarily represent the views of individual Key Informants.
Key Informants must disclose any financial conflicts of interest greater than $10,000 and any
other relevant business or professional conflicts of interest. Because of their role as end-users,
individuals with potential conflicts may be retained. The TOO and the EPC work to balance,
manage, or mitigate any conflicts of interest.
The list of Key Informants who participated in developing this report follows:
Julia Adler-Milstein, Ph.D. Deven McGraw, J.D., M.P.H.

Assistant Professor, School of Information Director, Health Privacy Project
Health Management and Policy, School of Center for Democracy & Technology
Public Health Washington, DC
University of Michigan
Ann Arbor, MI Kory Mertz, B.A.
Project Officer, Policy Analyst
Michael Barr, M.D., M.B.A., FACP Office of the National Coordinator for
Executive Vice President of the Quality Health Information Technology
Measurement & Research Group Department of Health and Human Services
National Committee for Quality Assurance Washington, DC
Washington, DC
Christopher Muir, M.A.
Richard Dave deBronkart, S.B. Director, State Health Information Exchange
Patient Advocate Program
Nashua, NH Office of the National Coordinator for
Health Information Technology
Erin Holve, Ph.D., M.P.P., M.P.H. Department of Health and Human Services
Senior Director Washington, DC
AcademyHealth
Washington, DC
Gilad Kuperman, M.D., Ph.D.

Associate Professor, Biomedical Informatics
Columbia University
New York, NY
iv
Susan Otter, B.A. Bret Schillingstad, M.D.
Oregon State Coordinator for Health Physician/Clinical Informatics
Information Technology Epic
Oregon Health Authority, Office of the Madison, WI
Director
Portland, OR Lee Stevens, B.A.
Program Manager for the State Health
Vaishali Patel, Ph.D., M.P.H. Information Exchange Program
Senior Advisor, Office of Economic Office of the National Coordinator for
Analysis, Evaluation and Modeling Health Information Technology
Office of the National Coordinator for Department of Health and Human Services
Health Information Technology Washington, DC
Department of Health and Human Services
Washington, DC
Technical Expert Panel

In designing the study questions and methodology at the outset of this report, the EPC
consulted several technical and content experts. Broad expertise and perspectives were sought.
Divergent and conflicted opinions are common and perceived as healthy scientific discourse that
results in a thoughtful, relevant systematic review. Therefore, in the end, study questions, design,
methodologic approaches, and/or conclusions do not necessarily represent the views of
individual technical and content experts.
Technical Experts must disclose any financial conflicts of interest greater than $10,000 and
any other relevant business or professional conflicts of interest. Because of their unique clinical
or content expertise, individuals with potential conflicts may be retained. The TOO and the EPC
work to balance, manage, or mitigate any potential conflicts of interest identified.
The list of Technical Experts who participated in developing this report follows:
Julia Adler-Milstein, Ph.D. Melinda Buntin, Ph.D.

Assistant Professor Professor and Chair, Department of Health
School of Information Health Management Policy
and Policy, School of Public Health Vanderbilt University School of Medicine
University of Michigan Nashville, TN
Ann Arbor, MI
Patricia Dykes, Ph.D.*
Thomas Agresta, M.D., M.B.I.* Assistant Professor, Harvard Medical
Professor of Family Medicine, Department School
of Family Medicine The Center for Patient Safety Research and
University of Connecticut School of Practice
Medicine The Center for Nursing Excellence
Farmington, CT Brigham & Women's Hospital
Boston, MA
v
Erin Holve, Ph.D., M.P.P., M.P.H. Daniel Vreeman, P.T., D.P.T., M.S.C.*
Senior Director Associate Director of Terminology Services
AcademyHealth Regenstrief Institute, Inc.
Washington, DC Associate Research Professor
Indiana University School of Medicine
Vaishali Patel, Ph.D., M.P.H. Indianapolis, IN
Senior Advisor, Office of Economic
Analysis, Evaluation and Modeling Johanna Westbrook, Ph.D., M.H.A.,
Office of the National Coordinator for FACHI, FACMI,
Health Information Technology Grad.Dip.App.Epidemiol, B.App.Sc.*
Department of Health and Human Services Professor of Health Informatics, Australian
Washington, DC Institute of Health Innovation
Faculty of Medicine and Health Sciences
Joshua R Vest, Ph.D., M.P.H.* Macquarie University
Assistant Professor, Department of Sydney, Australia
Healthcare Policy and Research
Weill Cornell Medical College
New York, NY
*This TEP member also provided review of the draft report.
Peer Reviewers
Prior to publication of the final evidence report, EPCs sought input from independent Peer
Reviewers without financial conflicts of interest. However, the conclusions and synthesis of the
scientific literature presented in this report do not necessarily represent the views of individual
reviewers.
Peer Reviewers must disclose any financial conflicts of interest greater than $10,000 and any
other relevant business or professional conflicts of interest. Because of their unique clinical or
content expertise, individuals with potential nonfinancial conflicts may be retained. The TOO
and the EPC work to balance, manage, or mitigate any potential nonfinancial conflicts of interest
identified.
The list of Peer Reviewers follows:
Jessica S. Ancker, Ph.D., M.P.H. Christopher Chute, M.D., D.P.H.

Associate Professor, Department of Bloomberg Distinguished Professor of
Healthcare Policy and Research Health Informatics
Weill Cornell Medical College Professor of Medicine, Public Health, and
New York, NY Nursing
Chief Health Research Information Officer,
Patricia Flatley Brennan, Ph.D., R.N. Johns Hopkins Medicine
Moehlman Bascom Professor School of Johns Hopkins University
Nursing Baltimore, MD
Industrial & Systems Engineering
Wisconsin Institute for Discovery
University of Wisconsin–Madison
Madison, WI
vi
Keven Johnson, M.D., M.S. Christopher Muir, M.A.
Professor and Chair, Department of Director, State Health Information Exchange
Biomedical Informatics Program
Vanderbilt University School of Medicine Office of the National Coordinator for
Nashville, TN Health Information Technology
Department of Health and Human Services
Rainu Kaushal, M.D., M.P.H. Washington, DC
Professor and Chairman, Department of
Healthcare Policy and Research Matthew Swain, M.P.H.
Weill Cornell Medical College Senior Analyst, Strategy and Performance
New York, NY Office of the National Coordinator for
Health Information Technology
Clement McDonald, M.D. Department of Health and Human Services
Director, Lister Hill National Center for Washington, DC
Biomedical Communications
National Library of Medicine, National
Institutes of Health
Bethesda, MD
Ann McKibbon, B.Sc., M.L.S., Ph.D.

Associate Professor, Department of Clinical
Epidemiology & Biostatistics
Director, eHealth MSc Program
Member, Health Information Research Unit
McMaster University
Ontario, Canada
vii
Health Information Exchange
Structured Abstract
Objectives. This review sought to systematically review the available literature on health
information exchange (HIE), the electronic sharing of clinical information across the boundaries
of health care organizations. HIE has been promoted as an important application of technology in
medicine that can improve the efficiency, cost-effectiveness, quality, and safety of health care
delivery. However, HIE also requires considerable investment by sponsors, which have included
governments as well as health care organizations. This review aims to synthesize the currently
available research addressing HIE effectiveness, use, usability, barriers and facilitators to actual
use, implementation, and sustainability, and to present this information as a foundation on which
future implementation, expansion, and research can be based.
Data sources. A research librarian designed and conducted searches of electronic databases,
including MEDLINE® (1990 to February 2015), PsycINFO® (1990 to February 2015),
CINAHL® (1990 through February 2015), the Cochrane Central Register of Controlled Trials
(through January 2015), the Cochrane Database of Systematic Reviews (through January 2015),
the Database of Abstracts of Reviews of Effects (through the first quarter of 2015), and the
National Health Sciences Economic Evaluation Database (through the first quarter of 2015). The
searches were supplemented by reviewing reference lists and the table of contents of journals not
indexed in the databases we searched.
Review methods. Two investigators reviewed abstracts and the selected full-text articles for
inclusion based on predefined criteria. Discrepancies were resolved through discussion and
consensus, with a third investigator making the final decision as needed. Data were abstracted
from each included article by one person and verified by another. All analyses were qualitative,
and they were customized according to the topic.
Results. We included 136 studies overall, with 34 on effectiveness, 26 of which reported

intermediate clinical, economic, or patient outcomes, and 8 that reported on clinical perceptions
of HIE. We also found 58 studies on the use of HIE, 22 on usability and other facilitators and
barriers to actual use of HIE, 45 on facilitators or barriers to HIE implementation, and 17 on
factors related to sustainability of HIE.
No studies of HIE effectiveness reported impact on primary clinical outcomes (e.g., mortality
and morbidity) or identified harms. Low-quality evidence somewhat supports the value of HIE
for reducing duplicative laboratory and radiology test ordering, lowering emergency department
costs, reducing hospital admissions (less so for readmissions), improving public health reporting,
increasing ambulatory quality of care, and improving disability claims processing. In studies of
clinician perceptions of HIE, most respondents attributed positive changes to HIE, such as
improvements in coordination, communication, and knowledge about the patient. However in
one study clinicians reported that the HIE did not save time and may not be worth the cost.
Studies of HIE use found that HIE adoption has increased over time, with 76 percent of U.S.
hospitals exchanging information in 2014, an 85-percent increase since 2008 and a 23-percent
increase since 2013. HIE systems were used by 38 percent of office-based physicians in 2012,
while use remains low, less than 1 percent, among long-term care providers.
viii
Within organizations with HIE, the number of users or the number of visits in which the HIE
was used was generally very low. The degree of usability of an HIE was associated with
increased rates of use but was not associated with effectiveness outcomes. The most commonly
cited barriers to HIE use were lack of critical mass electronically exchanging data, inefficient
workflow, and poorly designed interface and update features. Information was insufficient to
allow us to assess usability by HIE function or architecture.
Studies provided information on both external environmental and internal organizational
characteristics that affect implementation and sustainability. General characteristics of the HIE
organization (e.g., strong leadership) or specific characteristics of the HIE system were the most
frequently cited facilitators, while disincentives such as competition or lack of a business case
for HIE were the most frequently identified barriers.
Limitations. The scope of studies identified was limited compared with the actual uses and
capabilities of HIE. The outcomes measured and methods of measurement and analysis, for
example, were limited and narrowly defined; the issue of potential confounders was not
addressed in most studies of effectiveness, and harms were not adequately studied. There was a
high degree of heterogeneity in study designs, outcomes, HIE types, and settings across the
studies, limiting the ability to synthesize the evidence; no quantitative analyses were possible.
The applicability of this evidence base is uncertain because the HIE systems studied were so
diverse, and many in existence have not contributed to research in this field.
Conclusions. The full impact of HIE on clinical outcomes and potential harms is inadequately
studied, although evidence provides some support for benefit in reducing use of some specific
resources and achieving improvements in quality-of-care measures. Use of HIE has risen over
time, and is highest in hospitals and lowest in long-term care settings. However, use of HIE
within organizations that offer it is still low. Barriers to HIE use include lack of critical mass
participating in the exchange, inefficient workflow, and poorly designed interface and update
features. Studies have identified numerous facilitators and barriers to implementation and
sustainability, but the studies have not ranked or compared their impact. To advance our
understanding of HIE, future studies need to address comprehensive questions, use more
rigorous designs, use a standard for describing types of HIE, and be part of a coordinated
systematic approach to studying HIE.
ix
Contents
Executive Summary .................................................................................................................ES-1
Introduction ....................................................................................................................................1
Background ................................................................................................................................1
Scope of Review and Key Questions .........................................................................................2
Methods ...........................................................................................................................................7
Topic Development and Refinement .........................................................................................7
Literature Search Strategy..........................................................................................................7
Process for Study Selection .......................................................................................................8
Populations...........................................................................................................................8
Intervention and Comparators..............................................................................................8
Outcomes by Key Question .................................................................................................8
Timing ..................................................................................................................................9
Settings .................................................................................................................................9
Study Design ........................................................................................................................9
Data Abstraction and Data Management .................................................................................10
Assessment of Methodological Risk of Bias of Individual Studies .........................................10
Data Synthesis and Organization of Report .............................................................................11
Grading the Body of Evidence for Each Key Question ...........................................................11
Assessing Applicability ...........................................................................................................12
Peer Review and Public Commentary .....................................................................................12
Results ...........................................................................................................................................13
Results of Literature Searches .................................................................................................13
Description of Included Studies ...............................................................................................13
Key Question 1. Is HIE effective in improving clinical, economic, and population
outcomes? ..........................................................................................................................14
Key Question 2. What harms have resulted from HIE?...........................................................14
Key Question 3. Is HIE effective in improving intermediate outcomes such as patient
and provider experience, perceptions, or behavior; health care processes; or the
availability, completeness, or accuracy of information? ...................................................14
Key Points ..........................................................................................................................14
Detailed Synthesis ..............................................................................................................14
Key Question 4. What are the current level of use and primary uses of HIE? ........................26
Key Points ..........................................................................................................................26
Key Question 5. How does the usability of HIE impact effectiveness or harms
for individuals and organizations? .....................................................................................57
Key Question 6. What facilitators and barriers impact use of HIE? ........................................57
Key Points ..........................................................................................................................57
Key Question 7. What facilitators and barriers impact implementation of HIE? ....................70
Key Question 8. What factors influence sustainability of HIE? ..............................................70
Key Points ..........................................................................................................................70
Discussion......................................................................................................................................80
Key Findings ............................................................................................................................80
x
Strength of Evidence ................................................................................................................81
Findings in Relationship to What Is Already Known ..............................................................82
Applicability ............................................................................................................................83
Limitations of the Evidence Base ............................................................................................85
Future Research Needs ............................................................................................................85
Conclusions ..............................................................................................................................88
References .....................................................................................................................................89
Abbreviations and Acronyms ...................................................................................................100
Tables
Table A. Summary of evidence ................................................................................................ES-2
Table 1. Studies of HIE included for assessing outcomes .............................................................16
Table 2. Patient and clinician perceptions of HIE .........................................................................24
Table 3. Factors that may affect outcomes ....................................................................................25
Table 4. Level of use and primary uses of HIE: participation in HIE, types of data
exchanged, and characteristics of successfully participating organizations
(United States–wide studies)..........................................................................................................31
Table 5. Level of use and primary uses of HIE: transfer of records between integrated
delivery systems .............................................................................................................................38
Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use,
by regional or statewide initiatives ................................................................................................42
Table 7. Level of use and primary uses of HIE: extent of use, types of information
exchanged, and adoption in international or multinational settings ..............................................53
Table 8. Summary of evidence addressing usability, barriers, and facilitators to use ...................58
Table 9. Barriers and facilitators of actual HIE use grouped by theme .........................................67
Table 10. Facilitators to implementation and sustainability of HIE ..............................................73
Table 11. Barriers to implementation and sustainability of HIE ...................................................77
Table 12. Summary of evidence ....................................................................................................80
Figures
Figure 1. Analytic framework ..........................................................................................................4
Figure 2. Literature flow diagram ..................................................................................................13
Figure 3. Rasmussen sociotechnical analysis framework ..............................................................87
Appendixes
Appendix A. Search Strategies
Appendix B. Inclusion and Exclusion Criteria
Appendix C. List of Included Studies
Appendix D. List of Excluded Studies
Appendix E. Study Design Terminology
Appendix F. Evidence Table
Appendix G. Risk of Bias Assessment Criteria
Appendix H. Strength of Evidence Criteria
Appendix I. Quality Assessment Tables
xi
Executive Summary
Background
Health information exchange (HIE) is the sharing of electronic clinical data across
organizations.1 The idea that records should follow patients wherever they receive care has been
promoted as a cornerstone of efforts to improve the coordination, efficiency, and effectiveness of
health services. The underlying belief is that ultimately patients would benefit if all relevant
information were available to the various health care providers involved in treating them and
working to maintain their health. However, realizing this vision is challenging because health
care is currently provided by a diversity of organizations and providers with disparate
information systems. A substantial investment of resources is needed to develop an environment
that allows health care information to follow the patient.
Governments at all levels, as well as health systems and individual organizations, have and
are continuing to make the significant investment of time and resources to achieve the goals of
HIE. For example, in the United States, the Health Information Technology for Economic and
Clinical Health (HITECH) Act, part of the American Recovery and Reinvestment Act of 2009, is
providing up to $29 billion in incentive funding for the adoption and “meaningful use” of
electronic health records by hospitals and health professionals. The HITECH Act designated an
additional $564 million for investment by States or State-designated entities to establish HIE
capability among health care providers and hospitals in their jurisdictions. Understandably, all
stakeholders are interested in assuring that there is a return on this investment. These efforts have
resulted in substantial growth of HIE across the United States.2
The purpose of this review was to identify, summarize, and synthesize the available research
about HIE. The scope of the review was purposely broad and includes studies about four topics:
(1) effectiveness, (2) use of HIE, (3) usability and barriers and facilitators to use, and
(4) implementation and sustainability.
Methods
This review was completed by the Pacific Northwest Evidence-based Practice Center in
fulfillment of a contract from the Agency for Healthcare Research and Quality through the
Effective Health Care Program. We used the Program’s standard methods and procedures,3
which are similar to those established by the Institute of Medicine for systematic reviews.4 A
detailed description of the methods is available in the review protocol and in the full report, both
available at www.effectivehealthcare.ahrq.gov.
After finalizing the Key Questions to be considered in our review, we looked for reports of
research on HIE. We searched several bibliographic citation databases (e.g., MEDLINE®) with
support from two specialized reference librarians, and we searched Web sites and tables of
contents of publications that are not indexed in citation databases. Studies identified through
these searches were reviewed for eligibility by two investigators. We included any study with
data about an actual HIE designed to be used for clinical or public health decisionmaking. We
included many different types of studies in order to provide a comprehensive review of research
on HIE effectiveness, use, usability, implementation, and sustainability. Given this broad scope,
the included studies varied widely in design and quality. We did not include studies of exchanges
of data for research only, or studies about hypothetical or future HIEs. Data from included
ES-1
studies were abstracted from the articles, and this information was summarized in tables and
narratives.
Results
Overview
The major results are summarized in Table A and described in this section.
Table A. Summary of evidence

Number and Type of Primary Limitations of the
Topic Main Findings
Included Studies Evidence
Effectiveness 34 total: Low-quality evidence somewhat Studies were of a small number of the
20 retrospective cohort supports the value of HIE for functioning HIE implementations, with
3 RCT reducing duplicative laboratory similarity to unstudied ones unknown,
2 cross-sectional and radiology test ordering, possibly limiting generalizability.
2 case series lowering ED costs, reducing
8 survey (1 survey study hospital admissions (less so for Studies looked at limited outcomes,
was an RCT) readmissions), improving public considering the intended scope of the
health reporting, increasing impact of HIE.
ambulatory quality of care, and
improving disability claims
processing. No studies of harm
were reported.
Use 58 total: The proportion of hospitals and While there are relatively high-quality
25 survey ambulatory care practices that national and regional surveys and
13 audit log have adopted HIE is increasing. reports that track the expansion of
9 retrospective HIE among health care organizations,
database Currently, rates of HIE use within there is not a corresponding
7 mixed methods organizations with HIE are comprehensive effort to track changes
2 focus groups generally low. in rates of use within organizations.
1 time-motion
1 geocoding
Usability and 22 total: The most commonly cited barriers Studies of usability did not relate it to
factors affecting 9 multiple-site case to HIE use were lack of critical effectiveness and do not permit
use study mass electronically exchanging comparisons across settings or types
11 cross-sectional data (8 studies); inefficient of HIE.
2 before-after workflow (10 studies); poorly
designed interface and update Studies had limitations, such as
features (7 studies). incomplete description of the
functionality and architecture of the
systems, making comparison by type
difficult.
ES-2
Table A. Summary of evidence (continued)
Number and Type of Primary Limitations of the
Topic Main Findings
Included Studies Evidence
Implementation 52 total: Most facilitators of implementation The research has not been designed
and 26 cross-sectional cited in research were to allow ranking or comparisons of the
sustainability 17 multiple-site case characteristics of HIE projects or relative impact of different barriers
study the internal environment of the and facilitators.
2 before-after organizations implementing HIE,
3 retrospective cohort such as leadership. Most of the The definition and appropriate
2 prospective cohort identified barriers to measures of sustainability of HIE are
2 time series implementation were external not yet agreed upon, and the majority
environmental factors, such as of projects are relatively recent.
concerns about competition.
Factors related to sustainability

were similar to those identified for
implementation.
ED = emergency department; HIE = health information exchange; RCT = randomized controlled trial
We reviewed 5,211 abstracts and 849 full-text articles. Of these, we included 136 studies that
addressed one or more of our Key Questions. The data in the following sections come from a
body of literature in which studies of 12 different HIE implementations are the most frequent
even though they represent a small proportion of the HIEs functioning in the United States.
Fewer studies were based on national surveys/datasets, and a comparatively small number of
studies were conducted in other countries. Most of this literature has been published since 2006.
Most studies were retrospective cohort studies (analysis of existing data comparing a certain
outcome with and without HIE) or cross-sectional studies. We included several multisite case
studies that consisted of qualitative analysis of data from several sources, including responses
from interviews, questionnaires, or focus groups. Other less common research designs included
before-and-after studies and time-series studies, which looked at what happened before and after
HIE implementation. Only two randomized trials (in 3 publications) were identified. In general,
the risk of bias for these studies was high, with some rated as moderate, although not all study
designs were rated, and the overall strength of evidence was assessed as low or insufficient for
most outcomes.
Effectiveness
We identified 34 studies that associated HIE with various outcomes, with 26 assessing the
impact of HIE on resource use and 8 reporting on user perceptions of HIE impact. Studies that
examined whether HIE improved resource use defined this as: (1) reduced ordering of laboratory
tests, radiology exams, and costs, especially in the emergency department (ED); (2) reduced
hospital admissions, hospital readmissions, and consultations; (3) successful public health use; or
(4) improvement in quality of care or service delivery. The overall strength of evidence was low,
as most studies were retrospective and reported on narrow questions, such as reduction in test
ordering or consultations, and not larger overall clinical and financial impacts. Furthermore, the
retrospective design of most of the studies raised the potential for confounding factors impacting
their conclusions.
Studies of reduced laboratory tests, radiology exams, and costs showed the most consistent
associated benefits. Four U.S. studies found reductions in ED orders of lab tests and radiology
exams,5-8 and three more found reductions in radiology alone.9-11 A United States–based
ES-3
ambulatory study found a reduced rate of increase in laboratory testing and no impact on
imaging,12 while a Finland-based study found that orders for lab tests increased while orders for
imaging decreased.13 Two studies found that HIE reduced overall ED costs.5,6
The studies of admissions and readmissions had inconsistent findings, with some reporting
that HIE reduced admissions6,7,14-16 or readmissions,17 while others reported no effect.18-21
Similarly, the findings related to consultations or referrals were mixed, with one study reporting
fewer consultations and cost savings7 and another reporting an increase in referrals by both
primary care physicians and specialists.13 We did not pool the results using meta-analysis, as the
patient populations differed across studies.
Studies of other resource-use outcomes more consistently identified benefits. Studies of
quality of care found that physicians providing preventive services who used HIE performed
better on quality measures.22,23 Studies also reported that HIE could help identify frequent ED
users24 but did not lead to improvement of medication adherence.25 One study found that HIE
reduced the time needed to evaluate Social Security claims.26 Another found a positive
association between general patient satisfaction in hospitals and whether the hospital had
implemented HIE.27
In studies that asked users of HIE to report on their perception of its impact, all found at least
some benefit, although some uncovered negative aspects as well. Physicians were more satisfied
with electronic than paper lab reports;28 more physicians preferred HIE that pushed data to them
than HIE that required them to pull the data with a query;29 and physicians believed electronic
reports of ED use improved followup30,31 and that HIE improved ambulatory care practice
efficiency.32,33 However, physicians in one study responded that having HIE provide pharmacy
information in the ED improved knowledge but did not reduce time spent to provide service and
was not worth the cost.34 Patients reported that they preferred having records transferred via HIE
over transferring paper records themselves.35
Although most studies of the effectiveness of HIE reported positive results, the literature as a
whole was not comprehensive and few studies were of high quality. HIE is usually broad based
and designed to affect practice and numerous outcomes; however, evaluation studies have
focused on only one or a small number of uses or potential effects. Additionally, even in cases in
which the results were positive, the effect sizes were not large or able to be assessed given the
information provided. For example, ED savings are hard to evaluate if the overall budget for the
ED is not known. (See evidence tables in Appendix F of the full report for detailed results.)
Additionally, many studies employed simple study designs that impede risk-of-bias assessment
(thus lowering our confidence in the study results). Given these limitations, it is not possible to
conclude with any certainty that HIE has consistently been effective in improving health
outcomes.
Use of Health Information Exchange

We identified 58 studies that described either the level of use of HIE or the primary uses of
HIE. Of these, 15 studies evaluated HIE use nationally in the United States and 2 studies
evaluated HIE use across integrated delivery systems. About half (30 studies) of these studies
analyzed the extent to which HIE was implemented in a State or across a region, but these were
concentrated in New York (10 studies), Texas (5 studies), and Tennessee (5 studies). Six studies
evaluated HIE in other countries and three in multiple countries, two of which included the
United States.
ES-4
Nationwide surveys in the United States suggest that HIE use has risen substantially among
hospitals since 2008. Use of HIE was reported by 11 percent of hospitals in 2009,36 while more
current estimates range from 30 to 58 percent.37-39 Recent data from the Office of the National
Coordinator for Health Information Technology (ONC) suggest that more than three-quarters
(76%) of non-Federal acute care hospitals electronically exchanged laboratory results, radiology
reports, clinical care summaries, and/or medication lists with an outside provider.2 This
represents an 85-percent increase since 2008 and a 23-percent increase since 2013. Close to 7 in
10 hospitals (69%) electronically exchanged health information with ambulatory providers
outside of their organization, representing a 92-percent increase since 2008 and a 21-percent
increase since 2013. Results from the National Ambulatory Medical Care Survey (2013)
concluded that 39 percent of office-based physicians reported having HIE capability with other
providers or hospitals.40 Limited data suggest that use of technology in general and HIE
specifically is very low (> 1%) in long-term care settings.41,42
Between 2004 and 2009, regional health information organization (RHIO) was the term used
to describe HIE organizations; several of the included studies used this term. All RHIOs are
involved in HIE by definition, but both their reach and composition vary. In 2008 and 2009,
RHIOs included 14 percent of U.S. hospitals and 3 percent of ambulatory care practices.43 A
study of public health departments found that 36 percent had no RHIO in their jurisdiction and
12 percent had no relationship with the RHIO in their area.44 Of those with a RHIO in their area,
40 percent were actually exchanging information.44 In RHIOs, the entities most commonly
providing data are hospitals (83%), followed by ambulatory settings (60%); the entities most
commonly receiving data are ambulatory settings (95%), followed by hospitals (83%), public
health departments (50%), and payers (44%).45
Studies of HIE in integrated delivery systems included exchanges among the Department of
Defense, Department of Veterans Affairs (VA), and the private sector. In an initial test in one
city, 73 percent of patients could be located across the system and exchanges were executed two
to three times a week.46 A larger 12-site expansion experiment resolved some issues in matching
patients but reported that the VA received information from private organizations for 9 percent
of the matched patients.47
While organizational involvement and capacity for HIE are increasing, the data about actual
use of HIE when it is possible were limited and suggested that HIE is still not integrated into
usual care. For example, studies from the MidSouth e-Health Alliance suggested low use of HIE
overall (from 2.6% to 9.5% of visits in 2008 and 2009),48 with higher use for ED visits (15%)
and return clinic visits (19%).49 In another example, data collected in the Central Texas HIE
from 2006 to 2011, HIE use was low—used in only 2.3 percent of encounters.50
Usability and Other Barriers and Facilitators to Use

We reviewed 22 studies that examined either usability or other barriers and facilitators to
actual HIE use. The evidence was insufficient to compare usability by type of HIE function
(query-based, or pull, vs. directed, or push, exchange) or by type of architecture (centralized or
not).
We found five surveys on HIE usability, and most defined usability as it relates to function
and/or measured satisfaction with exchanging health information.29,32,51-53 Perceptions of
usability were related to actual use. One study reported higher scores on a measure of satisfaction
with user interface related to more frequent use,52 and another reported that users endorsing
statements that the HIE was useful and easy to learn to operate had higher levels of weekly HIE
ES-5
use.54 Providers who used HIE also reported increased satisfaction and improved relationships
with care partners.53,55A related negative finding was that providers had high expectations for
HIE before implementation and reported some ongoing unmet needs once HIE was operational.53
Barriers and facilitators to use of HIE were identified using cross-sectional and multiple-site
case studies that drew on data from several sources (e.g., interviews, focus groups, and
observations). Barriers and facilitators identified fell under three broad topics: lack of critical
mass electronically exchanging data, workflow, and interface. Several facilitators showed
promise in promoting electronic health data exchange: obtaining more complete patient
information; thoughtful implementation and workflow; and well-designed user interface and data
presentation.
Lack of critical mass was a key issue: if providers do not find useful data from HIE, they are
less likely to use HIE in the future. Data were incomplete because of issues of incomplete patient
information that related to the setting (more complete in an ED and less in a homeless center) or
challenges in matching patients across systems.46,47,56-61 Privacy, legal concerns, and
requirements that patients opt in or opt out to sharing data all reduced the completeness of data,
and approaches to address these factors could lead to more comprehensive data and increased
use. Differences in how HIE was incorporated into workflow and daily operations also affected
use.32,47,49,51,53,54,56,60-62 Studies found that when proxy nonphysician users accessed the system
and provided relevant information to the doctors, the system was used more frequently.48,49
Studies based on observations found that different providers used the exchange differently, with
nurses seeking information on hospital admissions or other care mentioned by the patients, while
physicians also used the exchange to complete their understanding of the patient history and to
facilitate decisionmaking.63 The interface and features of the systems were also cited as
encouraging or hindering use. User opinions differed in terms of whether they wanted more or
less information, based both on desire for more content61 and on interface issues, such as the
need to scroll or click through multiple pages.54,56,60 In addition, users reported that the systems
slowed down as data were expanded to include more patients and information or that new
information was not added to centralized systems quickly enough (so that going to records in
separate systems was quicker).54
Implementation and Sustainability

We identified 52 studies that aimed to identify factors that affect implementation and
sustainability. Forty-five studies identified facilitators to implementation (which we grouped into
8 categories) and barriers (which we grouped into7 categories). While fewer studies (17 studies)
considered sustainability, we sorted the positive and negative influences on sustainability so that
they overlapped with our categories of facilitators and barriers to implementation. Studies were
not designed to rank factors and did not provide enough data to allow us to assess the
comparative impact of different factors on implementation and sustainability.
Facilitators for implementation focused predominately on the characteristics of the
implementing organization or of the HIE system the organizations were planning to implement.
The most frequently cited category we labeled General Structure of the organizations
implementing HIE, and included specifics such as leadership26,64-66 and prior experience with or
readiness for information technology (IT) projects.53,67 Another category that facilitated
implementation, HIE-Specific Structures, included governance26 and participatory
approaches.23,68-71 Organizations implementing HIE shifted their mission or focus (category
labeled Orientation Shift) toward collaboration72 and continuity of care,73 and those that were
ES-6
successful were able to shift from piloting minimal HIE functions to a robust system quickly.74
Organizations successful in implementing HIE also provided support for the implementation,
such as training,75,76 and focused on selected outcomes, such as meeting a community need.77
Key Functions is our category of facilitators that included HIE designs that reflected workflow,69
and functions that could be integrated into care processes47,76,78,79 were also considered
facilitators for implementation. The one type of external factor cited as a facilitator was policy in
the form of Federal and State laws and mandates,78,80 as well as grants from Federal and State
governments that supported preliminary HIE activities and subsidized participating
organizations.67
Barriers to implementation overlapped with facilitators but included more categories of
external factors. External Policy included laws and grants that were identified as barriers when
their timelines or changes in requirements imposed burdens on organizations that could mitigate
the support they provided for implementation.65,81 The most frequently cited category of barriers
was Disincentives, including the issue of financial viability67,75,78,82,83 and the mismatch between
those who invest in HIE and those who benefit.67,84,85 The Technology Environment was another
category; characteristics that hindered implementation included lack of standards44,86 and limited
interoperability across organizations.78,87,88 Three categories of barriers were related to the
organization and its efforts to establish HIE: the Lack of Necessary Components, such as
physician engagement;72 the Fit between the goals and timeline of the organization and HIE
projects;89,90 and the need for resources to address complex problems with User Interface and
Functionality.47
Fewer studies considered sustainability. Positive influences included factors identified as
being associated with both implementation and sustainability, such as leadership by a health
information organization91 and provision of direct financial benefit to HIE participants.84,92 The
most commonly cited negative influences on sustainability were competition and the difficulty in
making the business case for HIE.93-96 Other hindrances to sustainability identified were
structural factors, such as a mismatch between the geographic coverage of the HIE and the
service area,96 governance issues and lack of trust,96,97 and lack of engagement of participating
organizations and their providers.77 One study documented that most HIE projects have overly
optimistic timelines and that the lack of time and missed deadlines worked against
sustainability.74
Implications
HIE represents a significant component of health care reform efforts. HIE is one of the major
applications of health IT and requires significant resources. Thus it is not surprising that
numerous studies have been published about HIE. However, this body of literature is limited in
several ways. Most of the studies are not designed to sufficiently control for risk of bias, and
they focus on relatively narrow outcomes when assessing the impact of a broad-based, complex,
systemic intervention such as HIE. While the studies of use, usability, implementation, and
sustainability provide information on context and allow some insight into trends, in general they
do not permit any comparative assessment or ranking of the importance of different barriers or
facilitators. Additionally these studies do not provide sufficient technical detail to compare HIE
systems by function or architecture.
Although it may not be the purview of research to decide if HIE should be funded as
infrastructure (as with a utility) or as a part of business operations, the notion that HIE should
improve efficiency and quality of care, including clinical and economic benefits, is not
ES-7
overwhelmingly supported by the available evidence. Positive findings are encouraging, but both
the level of the impact and some inconsistencies in results preclude any definitive conclusion.
Additionally, while surveys suggest that use of HIE is spreading, the scope of use within
organizations is still limited, implementation is slow, and sustainability seems less than assured.
Exactly what is needed for HIE to be effective is also difficult to discern from a body of
literature that does not include many comparative studies and that does not seem to build on prior
results to create a succession of increasingly relevant studies. We hope that this will improve as
HIE implementations become more mature and more robust study designs are used. Future
research should consist of prospective studies, carried out in mature HIE settings, assessing
patients who are likely to benefit from HIE and comparing appropriate outcomes for the use or
nonuse of HIE. The prospective collection of data from diverse settings where HIE is used,
classified by a detailed taxonomy of research type, system implementation, and usage type, could
allow for prospective cohort studies that could identify aspects of HIE associated with beneficial
outcomes.
Despite these concerns, expansion of HIE seems likely, and research could better serve this
effort by developing and pursuing a more deliberate research agenda designed to capture the full
potential impact of HIE and identify the comparative role of specific factors related to use,
usability, implementation, and ultimately, sustainability.
Conclusions
The full impact of HIE on clinical outcomes and potential harms is inadequately studied,
although evidence provides some support for benefit in reducing use of some specific resources
and improving quality-of-care measures. Use of HIE has risen over time, and is highest in
hospitals and lowest in long-term care settings. However, use of HIE within organizations that
offer it is still low. Barriers to HIE use include lack of critical mass exchanging data, inefficient
workflow, and poorly designed interface and update features. Factors we identified as facilitating
HIE implementation included general characteristics of the organization and specific
characteristics of the HIE system. Barriers focused more on the external environment, and
disincentives made up the largest category of barriers. Sustainability was less frequently studied;
the most frequently cited negative influences were competition and the lack of a business case
for HIE.
To advance our understanding of HIE, future studies need to address comprehensive
questions, use more rigorous designs, and be part of a coordinated systematic approach to
studying HIE.
ES-8
References
1. Health Information Exchange.What is HIE? 8. Winden TJ, Boland LL, Frey NG, et al. Care
Washington, DC: Department of Health and everywhere, a point-to-point HIE tool:
Human Services. www.healthit.gov/ utilization and impact on patient care in the
providers-professionals/health-information- ED. Appl Clin Inform. 2014;5(2):388-401.
exchange/what-hie. Accessed April 18, PMID: 25024756.
2014.
9. Bailey JE, Wan JY, Mabry LM, et al. Does
2. Swain M, Charles D, Patel V, et al. Health health information exchange reduce
Information Exchange among U.S. Non- unnecessary neuroimaging and improve
federal Acute Care Hospitals: 2008-2014. quality of headache care in the emergency
ONC Data Brief No. 24. Washington DC: department? J Gen Intern Med.
Office of the National Coordinator for 2013;28(2):176-83. PMID: 22648609.
Health Information Technology; 2015.
10. Bailey JE, Pope RA, Elliott EC, et al. Health
www.healthit.gov/sites/default/files/data-
information exchange reduces repeated
brief/ONC_DataBrief24_HIE_Final.pdf.
diagnostic imaging for back pain. Ann
Accessed April 19, 2015.
Emerg Med. 2013;62(1):16-24. PMID:
3. Methods Guide for Effectiveness and 23465552.
Comparative Effectiveness Reviews. AHRQ
11. Lammers EJ, Adler-Milstein J, Kocher KE.
Publication No. 10(14)-EHC062-EF.
Does health information exchange reduce
Rockville, MD: Agency for Healthcare
redundant imaging? Evidence from
Research and Quality. January 2014.
emergency departments. Med Care.
Chapters available at
2014;52(3):227-34. PMID: 24374414.
www.effectivehealthcare.ahrq.gov.
Accessed April 18, 2014. PMID: 21433403. 12. Ross SE, Radcliff TA, Leblanc WG, et al.
Effects of health information exchange
4. National Research Council. Finding What
adoption on ambulatory testing rates. J Am
Works in Health Care: Standards for
Med Inform Assoc. 2013;20(6):1137-42.
Systematic Reviews. Washington, DC: The
PMID: 23698257.
National Academies Press; 2011.
13. Mäenpää T, Asikainen P, Gissler M, et al.
5. Tzeel A, Lawnicki V, Pemble KR. The
Outcomes assessment of the regional health
business case for payer support of a
information exchange: a five-year follow-up
community-based health information
study. Methods Inf Med. 2011;50(4):308-18.
exchange: a Humana pilot evaluating its
PMID: 21336419.
effectiveness in cost control for plan
members seeking emergency department 14. Ben-Assuli O, Shabtai I, Leshno M. The
care. Am Health Drug Benefits. impact of EHR and HIE on reducing
2011;4(4):207-15. PMID: 25126351. avoidable admissions: controlling main
differential diagnoses. BMC Med Inform
6. Frisse ME, Johnson KB, Nian H, et al. The
Decis Mak. 2013;13:49. PMID: 23594488.
financial impact of health information
exchange on emergency department care. J 15. Vest JR, Kern LM, Campion TR Jr, et al.
Am Med Inform Assoc. 2012;19(3):328-33. Association between use of a health
PMID: 22058169. information exchange system and hospital
admissions. Appl Clin Inform.
7. Carr CM, Gilman CS, Krywko DM, et al.
2014;5(1):219-31. PMID: 24734135.
Observational study and estimate of cost
savings from use of a health information 16. Ben-Assuli O, Shabtai I, Leshno M. Using
exchange in an academic emergency electronic health record systems to optimize
department. J Emerg Med. 2014;46(2):250- admission decisions: the Creatinine case
6. PMID: 24071033. study. Health Informatics J. 2015;21(1):73-
88. PMID: 24692078.
ES-9
17. Vest JR, Kern LM, Silver MD, et al. The 27. Vest JR, Miller TR. The association between
potential for community-based health health information exchange and measures
information exchange systems to reduce of patient satisfaction. Appl Clin Inform.
hospital readmissions. J Am Med Inform 2011;2(4):447-59. PMID: 23616887.
Assoc. 2015 Mar;22(2):435-42. Epub 2014
28. Chang KC, Overhage JM, Hui SL, et al.
Aug 6. PMID: 25100447.
Enhancing laboratory report contents to
18. Vest JR. Health information exchange and improve outpatient management of test
healthcare utilization. J Med Syst. results. J Am Med Inform Assoc.
2009;33(3):223-31. PMID: 19408456. 2010;17(1):99-103. PMID: 20064809.
19. Tzeel A, Lawnicki V, Pemble KR. "Hidden" 29. Campion TR Jr, Ancker JS, Edwards AM, et
value: how indirect benefits of health al. Push and pull: physician usage of and
information exchange further promote satisfaction with health information
sustainability. Am Health Drug Benefits. exchange. AMIA Annu Symp Proc.
2012;5(6):333-40. PMID: 24991331. 2012;2012:77-84. PMID: 23304275.
20. Lang E, Afilalo M, Vandal AC, et al. Impact 30. Altman R, Shapiro JS, Moore T, et al.
of an electronic link between the emergency Notifications of hospital events to outpatient
department and family physicians: a clinicians using health information
randomized controlled trial. CMAJ. exchange: a post-implementation survey.
2006;174(3):313-8. PMID: 16399880. Inform Prim Care. 2012;20(4):249-55.
PMID: 23890336.
21. Jones SS, Friedberg MW, Schneider EC.
Health information exchange, Health 31. Afilalo M, Lang E, Léger R, et al. Impact of
Information Technology use, and hospital a standardized communication system on
readmission rates. AMIA Annu Symp Proc. continuity of care between family physicians
2011;2011:644-53. PMID: 22195120. and the emergency department. CJEM.
2007;9(2):79-86. PMID: 17391577.
22. Kern LM, Barrón Y, Dhopeshwarkar RV, et
al. Health information exchange and 32. Machan C, Ammenwerth E, Schabetsberger
ambulatory quality of care. Appl Clin T. Evaluation of the electronic transmission
Inform. 2012;3(2):197-209. PMID: of medical findings from hospitals to
23646072. practitioners by triangulation. Methods Inf
Med. 2006;45(2):225-33. PMID: 16538293.
23. Nagykaldi ZJ, Yeaman B, Jones M, et al.
HIE-i-health information exchange with 33. Maass MC, Asikainen P, Mäenpää T, et al.
intelligence. J Ambulatory Care Manage. Usefulness of a Regional Health Care
2014;37(1):20-31. PMID: 24309392. Information System in primary care. A case
study. Comput Methods Programs Biomed.
24. Shapiro JS, Johnson SA, Angiollilo J, et al.
2008;91(2):175-81. PMID: 18514363.
Health information exchange improves
identification of frequent emergency 34. Kaushal R, Dhopeshwarkar R, Gottlieb L, et
department users. Health Aff. al. User experiences with pharmacy benefit
2013;32(12):2193-8. PMID: 24301405. manager data at the point of care. J Eval
Clin Pract. 2010;16(6):1076-80. PMID:
25. Willis JM, Edwards R, Anstrom KJ, et al.
20666888.
Decision support for evidence-based
pharmacotherapy detects adherence 35. Park H, Lee S-I, Kim Y, et al. Patients'
problems but does not impact medication perceptions of a health information
use. Stud Health Technol Inform. exchange: a pilot program in South Korea.
2013;183:116-25. PMID: 23388267. Int J Med Inf. 2013;82(2):98-107. PMID:
22658777.
26. Feldman SS, Horan TA. Collaboration in
electronic medical evidence development: a 36. Adler-Milstein J, Bates DW, Jha AK. U.S.
case study of the Social Security regional health information organizations:
Administration's MEGAHIT System. Int J progress and challenges. Health Aff.
Med Inf. 2011;80(8):e127-40. PMID: 2009;28(2):483-92. PMID: 19276008.
21333588.
ES-10
37. Adler-Milstein J, Jha AK. Health 46. Bouhaddou O, Bennett J, Cromwell T, et al.
information exchange among U.S. hospitals: The Department of Veterans Affairs,
who's in, who's out, and why? Healthcare. Department of Defense, and Kaiser
2014;2(1):26-32. Permanente Nationwide Health Information
Network exchange in San Diego: patient
38. Audet A-M, Squires D, Doty MM. Where
selection, consent, and identity matching.
are we on the diffusion curve? Trends and
AMIA Annu Symp Proc. 2011;2011:135-43.
drivers of primary care physicians' use of
PMID: 22195064.
health information technology. Health Serv
Res. 2014;49(1 Pt 2):347-60. PMID: 47. Byrne CM, Mercincavage LM, Bouhaddou
24358958. O, et al. The Department of Veterans
Affairs' (VA) implementation of the Virtual
39. Furukawa MF, Patel V, Charles D, et al.
Lifetime Electronic Record (VLER):
Hospital electronic health information
findings and lessons learned from Health
exchange grew substantially in 2008-12.
Information Exchange at 12 sites. Int J Med
Health Aff. 2013;32(8):1346-54. PMID:
Inf. 2014;83(8):537-47. PMID: 24845146.
23918477.
48. Johnson KB, Gadd CS, Aronsky D, et al.
40. Furukawa MF, King J, Patel V, et al.
The MidSouth eHealth Alliance: use and
Despite substantial progress in EHR
impact in the first year. AMIA Annu Symp
adoption, health information exchange and
Proc. 2008:333-7. PMID: 18999184.
patient engagement remain low in office
settings. Health Aff. 2014:1-8. PMID: 49. Johnson KB, Unertl KM, Chen Q, et al.
25104827. Health information exchange usage in
emergency departments and clinics: the
41. Hamann DJ, Bezboruah KC. Utilization of
who, what, and why. J Am Med Inform
technology by long-term care providers:
Assoc. 2011;18(5):690-7. PMID: 21846788.
comparisons between for-profit and
nonprofit institutions. J Aging Health. 50. Vest JR, Zhao H, Jasperson J, et al. Factors
2013;25(4):535-54. PMID: 23509114. motivating and affecting health information
exchange usage. J Am Med Inform Assoc.
42. Caffrey C, Park-Lee E. Use of electronic
2011;18(2):143-9. PMID: 21262919.
health records in residential care
communities. NCHS Data Brief. 51. Hyppönen H, Reponen J, Lääveri T, et al.
2013(128):1-8. PMID: 24152578. User experiences with different regional
health information exchange systems in
43. Adler-Milstein J, Bates DW, Jha AK. A
Finland. Int J Med Inf. 2014;83(1):1-18.
survey of health information exchange
PMID: 24200753.
organizations in the United States:
implications for meaningful use. Ann Intern 52. Gadd CS, Ho Y-X, Cala CM, et al. User
Med. 2011;154(10):666-71. PMID: perspectives on the usability of a regional
21576534. health information exchange. J Am Med
Inform Assoc. 2011;18(5):711-6. PMID:
44. Hessler BJ, Soper P, Bondy J, et al.
21622933.
Assessing the relationship between health
information exchanges and public health 53. Messer LC, Parnell H, Huffaker R, et al. The
agencies. J Public Health Manag Pract. development of a health information
2009;15(5):416-24. PMID: 19704310. exchange to enhance care and improve
patient outcomes among HIV+ individuals
45. Adler-Milstein J, McAfee AP, Bates DW, et
in rural North Carolina. Int J Med Inf.
al. The state of regional health information
2012;81(10):e46-55. PMID: 22898321.
organizations: current activities and
financing. Health Aff. 2008;27(1):w60-9. 54. Myers JJ, Koester KA, Chakravarty D, et al.
PMID: 18073225. Perceptions regarding the ease of use and
usefulness of health information exchange
systems among medical providers, case
managers and non-clinical staff members
working in HIV care and community
settings. Int J Med Inf. 2012;81(10):e21-9.
PMID: 22854159.
ES-11
55. Massy-Westropp M, Giles LC, Law D, et al. 65. Merrill JA, Deegan M, Wilson RV, et al. A
Connecting hospital and community care: system dynamics evaluation model:
the acceptability of a regional data linkage implementation of health information
scheme. Aust Health Rev. 2005;29(1):12-6. exchange for public health reporting. J Am
PMID: 15683350. Med Inform Assoc. 2013;20(e1):e131-8.
PMID: 23292910.
56. Kierkegaard P, Kaushal R, Vest JR. How
could health information exchange better 66. Phillips AB, Wilson RV, Kaushal R, et al.
meet the needs of care practitioners? Appl Implementing health information exchange
Clin Inform. 2014;5(4):861-77. PMID: for public health reporting: a comparison of
25589903. decision and risk management of three
regional health information organizations in
57. McCullough JM, Zimmerman FJ, Bell DS,
New York state. J Am Med Inform Assoc.
et al. Electronic health information exchange
2014;21(e1):e173-7. PMID: 23975626.
in underserved settings: examining
initiatives in small physician practices & 67. Dullabh P, Hovey L. Large scale health
community health centers. BMC Health information exchange: implementation
Serv Res. 2014;14:415. PMID: 25240718. experiences from five states. Stud Health
Technol Inform. 2013;192:613-7. PMID:
58. Hincapie AL, Warholak TL, Murcko AC, et
23920629.
al. Physicians' opinions of a health
information exchange. J Am Med Inform 68. Herwehe J, Wilbright W, Abrams A, et al.
Assoc. 2011;18(1):60-5. PMID: 21106994. Implementation of an innovative, integrated
electronic medical record (EMR) and public
59. Ozkaynak M, Brennan PF. Revisiting
health information exchange for HIV/AIDS.
sociotechnical systems in a case of
J Am Med Inform Assoc. 2012;19(3):448-
unreported use of health information
52. PMID: 22037891.
exchange system in three hospital
emergency departments. J Eval Clin Pract. 69. Nykänen P, Karimaa E. Success and failure
2013;19(2):370-3. PMID: 22420774. factors in the regional health information
system design process--results from a
60. Thorn SA, Carter MA, Bailey JE.
constructive evaluation study. Methods Inf
Emergency physicians' perspectives on their
Med. 2006;45(1):85-9. PMID: 16482376.
use of health information exchange. Ann
Emerg Med. 2014;63(3):329-37. PMID: 70. Dullabh P, Ubri P, Hovey L. The State HIE
24161840. Program Four Years Later: Key Findings on
Grantees’ Experiences from a Six-State
61. Rudin R, Volk L, Simon S, et al. What
Review. Case Study Report. (Prepared by
affects clinicians' usage of health
NORC under Contract No.
information exchange? Appl Clin Inform.
HSSP2337010T/OS33547.) Office of the
2011;2(3):250-62. PMID: 22180762.
National Coordinator for Health Information
62. Nohr C, Kristensen M, Andersen SK, et al. Technology; 2014.
Shared experience in 13 local Danish EPR www.healthit.gov/sites/default/files/CaseStu
projects: the Danish EPR Observatory. Stud dySynthesisGranteeExperienceFinal_12101
Health Technol Inform. 2001;84(Pt 1):670- 4.pdf. Accessed April 22, 2014.
4. PMID: 11604822.
71. Pagliari C, Gilmour M, Sullivan F.
63. Unertl KM, Johnson KB, Lorenzi NM. Electronic Clinical Communications
Health information exchange technology on Implementation (ECCI) in Scotland: a
the front lines of healthcare: workflow mixed-methods programme evaluation. J
factors and patterns of use. J Am Med Eval Clin Pract. 2004;10(1):11-20. PMID:
Inform Assoc. 2012;19(3):392-400. PMID: 14731147.
22003156.
72. Saff E, Lanway C, Chenyek A, et al. The
64. Feldman SS, Schooley LB, Bhavsar PG. Bay Area HIE. A case study in connecting
Health information exchange stakeholders. J Healthc Inf Manag.
implementation: lessons learned and critical 2010;24(1):25-30. PMID: 20077922.
success factors from a case study. JMIR
Med Inform. 2014;2(2):e19.
ES-12
73. Unertl MK, Johnson BK, Gadd SC, et al. 82. Adler-Milstein J, Bates DW, Jha AK.
Bridging organizational divides in health Operational health information exchanges
care: an ecological view of health show substantial growth, but long-term
information exchange. JMIR Med Inform. funding remains a concern. Health Aff
2013;1(1):e3. PMID: 25600166. (Millwood). 2013;32(8):1486-92. PMID:
23840051.
74. Morris G, Afzal S, Bhasker M, et al. Query-
Based Exchange: Key Factors Influencing 83. Dixon B, Miller T, Overhage M. Barriers to
Success and Failure. Office of the National achieving the last mile in health information
Coordinator for Health Information exchange: a survey of small hospitals and
Technology; 2012. physician practices. J Healthc Inf Manag.
www.healthit.gov/sites/default/files/query_b 2013;27(4):55-8.
ased_exchange_final.pdf. Accessed April
84. Grossman JM, Kushner KL, November EA.
10, 2015.
Creating sustainable local health information
75. Ross SE, Schilling LM, Fernald DH, et al. exchanges: can barriers to stakeholder
Health information exchange in small-to- participation be overcome? Res Briefs.
medium sized family medicine practices: 2008(2):1-12. PMID: 18496926.
motivators, barriers, and potential
85. Vest JR. More than just a question of
facilitators of adoption. Int J Med Inf.
technology: factors related to hospitals'
2010;79(2):123-9. PMID: 20061182.
adoption and implementation of health
76. Silvester BV, Carr SJ. A shared electronic information exchange. Int J Med Inf.
health record: lessons from the coalface. 2010;79(12):797-806. PMID: 20889370.
Med J Aust. 2009;190(11 Suppl):S113-6.
86. Schabetsberger T, Ammenwerth E,
PMID: 19485857.
Andreatta S, et al. From a paper-based
77. Goldwater J, Jardim J, Khan T, et al. transmission of discharge summaries to
Emphasizing public health within a health electronic communication in health care
information exchange: an evaluation of the regions. Int J Med Inf. 2006;75(3-4):209-15.
District of Columbia’s Health Information PMID: 16112892.
Exchange Program. EGEMS (Wash DC).
87. Dobalian A, Claver ML, Pevnick JM, et al.
2014;2(3):1090. PMID: 25848607.
Organizational challenges in developing one
78. Fontaine P, Zink T, Boyle RG, et al. Health of the Nationwide Health Information
information exchange: participation by Network trial implementation awardees. J
Minnesota primary care practices. Arch Med Syst. 2012;36(2):933-40. PMID:
Intern Med. 2010;170(7):622-9. PMID: 20703640.
20386006.
88. Overhage JM, Evans L, Marchibroda J.
79. Steward WT, Koester KA, Collins SP, et al. Communities' readiness for health
The essential role of reconfiguration information exchange: the National
capabilities in the implementation of HIV- Landscape in 2004. J Am Med Inform
related health information exchanges. Int J Assoc. 2005;12(2):107-12. PMID:
Med Inf. 2012;81(10):e10-20. PMID: 15561785.
22841703.
89. Genes N, Shapiro J, Vaidya S, et al.
80. Adjerid I, Padman R. Impact of health Adoption of health information exchange by
disclosure laws on health information emergency physicians at three urban
exchanges. AMIA Annu Symp Proc. academic medical centers. Appl Clin
2011;2011:48-56. PMID: 22195054. Inform. 2011;2(3):263-9. PMID: 23616875.
81. Fairbrother G, Trudnak T, Christopher R, et 90. Lobach DF, Kawamoto K, Anstrom KJ, et
al. Cincinnati Beacon Community Program al. Proactive population health management
highlights challenges and opportunities on in the context of a regional health
the path to care transformation. Health Aff. information exchange using standards-based
2014;33(5):871-7. PMID: 24799586. decision support. AMIA Annu Symp Proc.
2007:473-7. PMID: 18693881.
ES-13
91. Kern LM, Wilcox AB, Shapiro J, et al. 95. Miller AR, Tucker C. Health information
Community-based health information exchange, system size and information silos.
technology alliances: potential predictors of J Health Econ. 2014;33:28-42. PMID:
early sustainability. Am J Manag Care. 24246484.
2011;17(4):290-5. PMID: 21615199.
96. Vest JR, Grinspan ZM, Kern LM, et al.
92. Kern LM, Wilcox A, Shapiro J, et al. Which Using a health information exchange system
components of health information for imaging information: patterns and
technology will drive financial value? Am J predictors. AMIA Annu Symp Proc.
Manag Care. 2012;18(8):438-45. PMID: 2013;2013:1402-11. PMID: 24551416.
22928759.
97. Rudin RS, Simon SR, Volk LA, et al.
93. Kern LM, Barrón Y, Abramson EL, et al. Understanding the decisions and values of
HEAL NY: promoting interoperable health stakeholders in health information
information technology in New York State. exchanges: experiences from Massachusetts.
Health Aff. 2009;28(2):493-504. PMID: Am J Public Health. 2009;99(5):950-5.
19276009. PMID: 19299671.
94. McGowan JJ, Jordan C, Sims T, et al. Rural
RHIOs: common issues in the development
of two state-wide health information
networks. AMIA Annu Symp Proc.
2007:528-32. PMID: 18693892.
ES-14
Introduction
Background
The use of health information technology (IT) has the potential to improve the quality, safety,
and efficiency of health care in the United States and around the world.1 Health IT can support
patient care delivery activities such as communications, results reporting, order entry, care
planning, and documentation. Examples of health IT applications include electronic health
records (EHR), clinical decision support such as alerts and reminders, computerized provider
order entry, electronic access to clinical practice guidelines and evidence databases, consumer
health informatics applications, telemedicine, and electronic exchange of health information.
In recent years, the Health Information Technology for Economic and Clinical Health
(HITECH) Act has accelerated EHR adoption in ambulatory and hospital settings across the
United States. The HITECH Act, part of the American Recovery and Reinvestment Act of 2009,
is providing up to $29 billion in incentive funding for the adoption and “meaningful use” of
EHRs by hospitals and physicians. As a result of HITECH funding, 94 percent of non-Federal
hospitals,2 78 percent of hospital-based physicians,3 84 percent of emergency departments, and
73 percent of hospital outpatient departments in the United States have adopted EHRs.4 The
motivation to increase the use of EHRs is grounded in evidence that health IT may improve the
quality, safety, efficiency, and satisfaction with care, as has been reported in recent systematic
reviews.5-8
A key challenge to effective use of health IT, however, is that most U.S. residents, especially
those with multiple conditions, receive care across a number of settings. Among 3.7 million
patients hospitalized in Massachusetts during a 5 year period, 31 percent were admitted to two or
more hospitals (57% of all visits) and 1 percent were admitted to five or more hospitals (10% of
all visits).9 Similarly, an analysis of 2.8 million patients seen by an emergency department in
Indiana found that 40 percent had data at multiple institutions.10 These data silos present a
challenge if we are to meet the goal stated by former Agency for Healthcare Research and
Quality (AHRQ) Director Dr. Carolyn Clancy that, “data should follow the patient” wherever
they get their care.11
To enable data to follow patients wherever they receive care, attention is now focused on
health information exchange (HIE), defined as the reliable and interoperable electronic sharing
of clinical information among physicians, nurses, pharmacists, other health care providers, and
patients across the boundaries of health care institutions, health data repositories, states, and
other entities who are not within a single organization or among affiliated providers.12 The
HITECH Act recognized that EHR adoption alone is insufficient to realize the full promise of
health IT, allocating $563 million for States or State-designated entities to establish HIE
capability among health care providers and hospitals in their jurisdictions.13 In the meantime, a
growing number of private organizations have undertaken HIE.14 Ideally, HIE across health care
organizations should facilitate care coordination and transitions between settings, improve
patient safety, and reduce duplicate testing.
The Office of the National Coordinator for Health IT (ONC) has defined three forms of
HIE:13
1
• Directed exchange: sending and receiving secure information electronically between care
providers
• Query-based exchange: provider-initiated requests for information on a patient from other
providers
• Consumer-mediated exchange: patients aggregating and controlling the use of their health
information among care providers.
ONC also uses the words “push” to describe directed exchange and “pull” to describe query-
based exchange.15
In general, HIE is defined as the electronic exchange of patient data across health care
organizations. This excludes exchange of information that is predominantly paper-based as well
as queries of remotely accessed systems (e.g., a clinician in one health care system seeking
information residing in a system of another health care organization accessed over the Internet).
Many also advocate that HIE be used as a verb or activity-based noun, and not as an entity or
organization, even though many HIE implementations and/or the organizations implementing
them call themselves “HIEs.”16,17
An early example of HIE was the work of Dr. Clement McDonald, who pioneered HIE in
Indiana starting in the 1990s.18 This led to the formation of the Indiana Health Information
Exchange, one of the largest and most successful HIE efforts in the United States.19 Other early
efforts to implement HIE, including some high-profile efforts, were less successful.20 Although
the rationale for HIE has been viewed as critical,21 the path to achieve it has in some respects
been more difficult than EHR adoption,22,23 in no small part due to the lack of sustainable
business models.24,25 Nonetheless, HIE adoption has grown as a result of the HITECH Act.26
Another barrier to HIE has been the development and adoption of health IT standards to
ensure interoperability among systems. This has driven ONC, the lead U.S. government agency
for health IT, to prioritize interoperability in its most recent strategic plan for health IT.27 ONC
has also launched a process to establish an interoperability roadmap for guiding implementation
of standards and interoperability, which also has the potential to facilitate adoption and
improvement of HIE.28 An additional barrier to HIE described by ONC is “information
blocking,” which is the unintentional or deliberate prevention of information exchange between
health IT systems.29
Evaluating the effectiveness of HIE (and health IT generally) has been challenging.30 HIE is
a technology that is intermediate to improving care delivery, allowing clinicians and others
improved access to patient data to inform decisions and facilitate appropriate use of testing and
treatment. HIE is not specific to any health issue or diagnosis. HIE implementations have often
been supported by one-time start-up funding, without long-term support to sustain the programs
long enough for evaluation.
The promise for HIE to improve health care delivery is substantial, but adoption in its various
forms has been complex and costly. It is therefore critical to be able to determine if HIE does
improve health or intermediate outcomes as well as to systematically assess comparative
approaches, barriers, return on investment, and sustainability of HIE.
Scope of Review and Key Questions

The review undertaken is timely and necessary—our knowledge of and experience with the
HIE literature demonstrates an evidence base that is scattered across disciplines and in various
formats. There are three previously published systematic reviews that focus exclusively on
HIE.31-33 One of these reviews is almost a half-decade old,31 another focused only on U.S.-based
2
and clinical-only (i.e., not public health) activities,32 and a third assessed only care outcomes and
not larger issues of facilitators, barriers, and sustainability.33
In requesting this review, AHRQ’s goal is a report focused on systematically identifying and
synthesizing evidence on the extent to which HIE can effectively improve a variety of outcomes,
and to determine if it is possible to say how the impact varies by different approaches to HIE.
This is due in part to AHRQ having funded a large portfolio of research in health IT and HIE,34
and having published an extensive guide to evaluating HIE projects.35 This report also aims to
identify evidence on levels of use, and usability of HIE, as well as facilitators of and barriers to
implementation, use, and sustainability of HIE. The analytic framework (Figure 1) and Key
Questions used to guide this review are shown below. The analytic framework shows the target
populations, interventions, and health outcomes examined, with numbers corresponding to the
Key Questions.
3
Figure 1. Analytic framework
KQ = Key Question
4
This report focuses on the following Key Questions:
Key Question 1. Is HIE effective in improving clinical (e.g., mortality and morbidity), economic
(e.g., costs and resource use, the value proposition for HIE), and population (e.g., syndromic
surveillance) outcomes?
Key Question 1a. Does effectiveness vary by type of HIE?
Key Question 1b. Does effectiveness vary by health care settings and systems?
Key Question 1c. Does effectiveness vary by IT system characteristics?
Key Question 1d. What evidence exists that the lack of HIE leads to poorer outcomes?
Key Question 2. What harms have resulted from HIE? (e.g., violations of privacy, errors in
diagnosis or treatment from too much, too little or inaccurate information, or patient or provider
concerns about HIE)?
Key Question 2a. Do harms vary by type of HIE?
Key Question 2b. Do harms vary by health care settings and systems?
Key Question 2c. Do harms vary by the IT system characteristics?
Key Question 3. Is HIE effective in improving intermediate outcomes such as patient and
provider experience, perceptions, or behavior; health care processes; or the availability,
completeness, or accuracy of information?
Key Question 3a. Does effectiveness in improving intermediate outcomes vary by type of
HIE?
Key Question 3b. Does effectiveness in improving intermediate outcomes vary by health care
settings and systems?
Key Question 3c. Does effectiveness in improving intermediate outcomes vary by IT system
characteristics?
Key Question 3d. What evidence exists that the lack of HIE leads to poorer intermediate
outcomes?
Key Question 4. What are the current level of use and primary uses of HIE?
Key Question 4a. Do level of use and primary uses vary by type of HIE?
Key Question 4b. Do level of use and primary uses vary by health care settings and systems,
or provider type?
Key Question 4c. Do level of use and primary uses vary by IT system characteristics?
Key Question 4d. Do level of use and primary uses vary by data source?
Key Question 5. How does the usability of HIE impact effectiveness or harms for individuals
and organizations?
Key Question 5a. How usable are various types of HIE?
Key Question 5b. What specific usability factors impact the effectiveness or harms from
HIE?
Key Question 5c. How does usability vary by health care settings or systems?
Key Question 6. What facilitators and barriers impact use of HIE?

Key Question 6a. Do facilitators and barriers that impact use vary by type of HIE?
5
Key Question 6b. Do facilitators and barriers that impact use vary by health care settings and
systems?
Key Question 6c. Do facilitators and barriers that impact use vary by IT system
characteristics?
Key Question 7. What facilitators and barriers impact implementation of HIE?

Key Question 7a. Do facilitators and barriers that impact implementation vary by type of
HIE?
Key Question 7b. Do facilitators and barriers that impact implementation vary by health care
settings and systems?
Key Question 7c. Do facilitators and barriers that impact implementation vary by IT system
characteristics?
Key Question 8. What factors influence sustainability of HIE?
6
Methods
This systematic review follows the methods of the Agency for Healthcare Research and
Quality (AHRQ) “Methods Guide for Effectiveness and Comparative Effectiveness Reviews.”36
Topic Development and Refinement

The initial draft Key Questions were first provided by AHRQ, who requested this review as
part of its effort to assess the impact of the AHRQ’s health information technology (IT) portfolio
and set future direction for the field. The Key Questions and scope were further revised and
developed by the review team with input from a group of stakeholders (Key Informants)
convened for this review to provide diverse perspectives as well as content and methodological
expertise. The Key Informants consisted of experts in health IT, applied informatics, clinical
care, health policy and patient advocacy. Key Informants disclosed financial and other conflicts
of interest prior to participation. The AHRQ Task Order Officer and the investigators reviewed
the disclosures and determined that the Key Informants had no conflicts of interest that precluded
participation.
The project team, with input from a Technical Expert Panel (TEP) convened for this review,
further developed the approach to this review. The TEP added expertise in informatics research
and systematic reviews to the perspectives that were represented by the Key Informants. The
Key Informants and TEP members are listed in the front matter. The protocol was then posted
for public comment from February 6 to February 26, 2014. Based on public comments, we
further revised the Key Questions and scope. The final protocol was developed and posted on
July 21, 2014 on the AHRQ Web site at: http://effectivehealthcare.ahrq.gov/index.cfm/search-
for-guides-reviews-and-reports/?productid=1943&pageaction=displayproduct. The protocol was
subsequently revised to document a change in the numbering of the Key Questions and reposted.
The original protocol was also registered in the PROSPERO international database of
prospectively registered systematic reviews.37
Literature Search Strategy

A research librarian conducted searches in Ovid MEDLINE (1990 to February 2015),
PsycINFO (1990 to February 2015), CINAHL (1990 through February 2015), the Cochrane
Central Register of Controlled Trials (through January 2015), Cochrane Database of Systematic
Reviews (through January 2015), the Database of Abstracts of Reviews of Effects, and the
National Health Sciences Economic Evaluation Database (through the first quarter of 2015). See
Appendix A for the detailed search strategies. Searches were peer reviewed by a second librarian
with systematic review experience who offered suggestions and confirmed accuracy. Searches
were designed to retrieve publications from January 1, 1990 forward, which reflects the timing of
initial implementations of health information exchange (HIE) in the United States. Our search
strategy was based on a broad terms and we evaluated this approach in several ways including
determining if it successfully identified examples of several types of studies. During our
literature scan we screened a sample of citations from two additional databases: Business
Premier and the Institute of Electrical and Electronics Engineers (IEEE) Xplore Digital Library;
neither screen resulted in identification of relevant articles and the databases were not searched
further. Searches were supplemented with hand searches of reference lists of relevant studies and
the table of contents of journals not indexed in the databases searched (e.g., Generating Evidence
7
and Methods to improve patient outcomes [eGEMs]), as well as searches of gray literature
sources (e.g., reports and analyses on Web sites of key organizations).
In addition, Scientific Information Packets were requested from organizations likely to have
data on research or evaluations of health information exchange (HIE) that have not been
published or indexed in citation databases. These organizations had the opportunity to submit
data using the portal for submitting Scientific Information Packets on the Effective Health Care
Program Web site. One submission was received from the California Health Care Foundation.
Process for Study Selection

The criteria for inclusion and exclusion of studies was based on the Key Questions and the
populations, interventions, comparators, outcomes, timing, types of studies and setting (PICOTS)
defined in the protocol (Appendix B). Papers were selected for review if they reported data about
HIE (as defined below), had data relevant to a Key Question, and met the other pre-specified
inclusion criteria. Studies of nonhuman subjects and studies with no original data were excluded.
Abstracts were independently reviewed by two investigators for inclusion. Full-text articles were
obtained for all studies that any investigator identified as potentially meeting inclusion criteria.
Two investigators independently reviewed all full-text articles for final inclusion. Sample sets of
abstracts and full text articles were reviewed by the entire team at key points in the review
process to establish norms. Inclusion was restricted to English-language articles. A list of the
included studies appears in Appendix C; a list of excluded studies and primary reasons for
exclusion can be found in Appendix D. Discrepancies were resolved through discussion and
consensus during team meetings with investigators.
Populations
Study population included any individual or group of health care providers, patients,
managers, health care institutions, or regional organizations.
Intervention and Comparators

We defined HIE as the electronic sharing of clinical information among users such as
clinicians, patients, administrators, or policymakers, across the boundaries of health care
institutions, health data repositories, States, and others, typically not within a single organization
or among affiliated providers, while protecting the integrity, privacy, and security of the
information. We did not include in this definition of HIE the exchange of information within a
single organization or entity (e.g., exchange within a network such as Kaiser Permanente or the
Veteran’s Administration or exchange across roles such as patient and clinician communications
within a provider organization).
Comparators included were time period prior to HIE implementation, different locations
(geographic or organizational without HIE) or situations in which HIE is not available (akin to
“usual care” in a clinical study), comparisons across types of HIE, and comparisons of the
characteristics of the different settings, health care system, and IT systems in which HIE is used.
Outcomes by Key Question

Key Question 1: Effectiveness was defined in terms of clinical outcomes (e.g., mortality and
morbidity), economic outcomes (e.g., costs and resource use, the value proposition for HIE) and
8
population outcomes (e.g., syndromic surveillance for the identification of trends or clusters).
Each study was assessed for its type of outcome and results in terms of the following attributes:
• Location – geographic
• Health care setting – e.g., emergency department, outpatient, health system
• HIE type – query versus directed
• Outcome category
• Direction of result – benefit versus mixed versus none.
Key Question 2: Harms included unintended negative consequence or adverse events
experienced by individuals, institutions, or organizations. Harms from HIE may include negative
outcomes or the risk of negative outcomes resulting from information that is wrong, not provided
in a timely manner, or in formats that inhibit its identification, comprehension, and use. Harms
may result from too much information or insufficient information, or include negative impacts on
attitudes (e.g., patient privacy concerns or clinician liability concerns).
Key Question 3: Intermediate outcomes included clinician and patient experiences and
perceptions; changes in individual behavior or care delivery processes; and changes in the
availability, completeness or accuracy of information.
Key Question 4: Level of use was a measure of the usage of HIE use by individuals, health
care institutions, or regional organizations.
Key Question 5: Usability focused on the function of the HIE in terms of the interaction
between users and HIE and their ability or capacity to navigate and accomplish tasks.
Key Question 6: Facilitators and barriers were the drivers and challenges to use of HIE in
the workflow and decisions of patients, clinicians, or organizations.
Key Question 7: Implementation of HIE was defined as the realization of an HIE project
such that the exchange of data is operational.
Key Question 8: Sustainability was long-term maintenance, development, and improvement
or expansion of HIE, after the implementation period.
Timing
No prespecified minimum duration of time was required between implementation of HIE to
the measurement of outcomes.
Settings
Settings included any aspect of the location or venue in which health information is
exchanged for the purpose of improving health or health care that is hypothesized to impact
effectiveness, use, usability, or sustainability. This included the type(s) of clinical environments
(e.g., ambulatory care, hospital, nursing home), payment/reimbursement model(s) (e.g., fee-for-
service, managed care, risk/value-based model such as an accountable care organization), and
legislative requirements (e.g., participation in HIE required to participate in Medicaid). Also
included were studies in public health organizations and settings; those using HIE data for
clinical research were excluded.
Study Design
Our approach to decisions about what designs and units of analysis to include varied across
the Key Questions, reflecting the fact that different types of research was needed to answer
different types of questions.
9
For questions on efficacy, effectiveness, and harms a “best evidence” approach was used.
Randomized controlled trials (RCTs) were included as the top-tier evidence. If insufficient
evidence was found of this type, observational studies (defined as cohort studies comparing at
least two HIE systems, case-control studies, and time-series studies) were explored.
For questions on use, usability, implementation, and sustainability, observational studies and
qualitative research were included. We also included detailed case studies of multiple HIE
organizations or sites. For studies of use and usability we included examinations both on the
individual level and organizational level, while implementation and sustainability were defined
as organizational level activities.
Systematic reviews were considered as sources of studies to be reviewed for possible
inclusion. High quality reviews with information directly relevant to our Key Questions were
eligible for inclusion in this review as evidence. High-quality reviews were defined as those
assessed as being at low risk of bias, according to the Assessing the Methodological Quality of
Systematic Reviews-AMSTAR quality assessment tool.36,38
We excluded studies that modeled the potential impact of HIE or that presented, discussed, or
evaluated hypothetical situations about HIE not yet implemented. Also excluded were
descriptive narratives or “lessons learned” essays that were not based on collecting clinical,
survey, or interview data from identified users or stakeholders. We restricted inclusion to
English-language articles, but reviewed English language abstracts of non-English language
articles to identify studies that would otherwise meet inclusion criteria.
See Appendix E for the study design terminology used in this review.
Data Abstraction and Data Management

After studies were selected for inclusion, data were abstracted into categories including but
not limited to: (a) general information such as study design, year, setting, geographic location,
and duration; (b) characteristics of the HIE implementation such as the form (directed exchange,
query-based exchange, consumer-mediated exchange), the number and types of participating
organizations, the type of user interface (e.g., push vs. pull), and the types of information
included; and (c) key contextual information to be used to identify facilitators and barriers to
HIE use as well as to assess applicability of the results. At a minimum, we included details about
the type(s) of clinical environments (e.g., ambulatory care, hospital, nursing home),
payment/reimbursement model(s) (e.g., fee-for-service, managed care setting, risk/value-based
model such as an accountable care organization), and relevant outcomes. Abstracted information
is included in Appendix F and is also available in the Systematic Review Data Repository.
Assessment of Methodological Risk of Bias of Individual

Studies
Assessment of risk of bias of trials and observational studies was based on recommendations
in the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews.36 Two
investigators independently assessed risk of bias for all effectiveness studies. Differences were
resolved by discussion and consensus and reviewed by the team of investigators. Individual
studies were rated as “low,” “moderate,” or “high” risk of bias. The criteria and interpretation of
these ratings are described in our protocol and in Appendix G.
For studies of surveys, interviews, and focus groups we did not give a formal overall risk of
bias rating; however, we did record information about sampling, completion rates, the
10
development of the questions, and the appropriateness of the analysis. This information informed
our descriptions of the studies and assessment of both the strength of evidence and the specific
needs for future research. Appendix G includes a list of the information we recorded. Risk of
bias was not assessed for case studies, mixed methods studies, or studies based on computer
system logs.
Data Synthesis and Organization of Report

We constructed evidence tables identifying the study characteristics, results of interest, and
risk of bias assessment for all included studies with summary tables to highlight the main
findings. For each study, we recorded the type of HIE when described, information on the
sample and response rate when reported, and types of stakeholders. We reviewed and highlighted
studies by using a hierarchy of evidence approach, where the best evidence was the focus of our
synthesis for each Key Question.
We found heterogeneity in the interventions and outcomes measured, including how similar
outcomes were measured and reported, such that we did not conduct meta-analyses. We
combined studies in the synthesis of the results based on the similarity of the type of HIE, the
implementation of the HIE, outcomes measured, and results reported. Where studies were not
similar in these areas we provided the results of the invidvidual studies without combining them.
The evidence for Key Questions 1, 2, and 3 were summarized and presented together as there
were few studies that reported on primary clinical outcomes and no studies that explicitly
analyzed harms. Many studies that reported resource usage (primary economic outcomes) were
actually reporting on clinical process outcomes, such as use of testing or prevention of hospital
admissions. We included studies of perceptions of HIE only if an actual operational HIE
implementation was analyzed. For Key Question 4 there were two categories of studies: large,
mostly national surveys that examined HIE use on a macro level (e.g., which organizations did or
did not use HIE); and studies that examined how HIE was used within organizations. We
presented the evidence for Key Questions 5 (usability) and 6 (barriers and facilitators to use)
jointly as some studies addressed both sets of questions together.
Similarly, we presented the results for Key Questions 7 and 8 together because conceptually,
organizations consider sustainability when deciding whether or not to adopt an innovation or
implement a new practice and conversely sustainability is at least partially dependent on the form
and success of implementation. As a result, there is significant overlap in the research. Many of
the studies we identified either addressed implementation and sustainability, or addressed
implementation as well as the topics covered by other Key Questions – impact, use, or
usage/usability. The focus of the results section for Key Questions 7 and 8 is on categories of
facilitators and barriers. We grouped the factors identified in the literature into categories in
order to provide a summary.
Grading the Body of Evidence for Each Key Question

The strength of evidence for key outcomes was rated only for effectiveness and harms
outcomes in Key Questions 1, 2, and 3 using the four categories recommended in the AHRQ
Methods Guide.36
• A “high” grade indicates high confidence that the estimate of effect lies close to the true
effect for this outcome. The body of evidence has few or no deficiencies and the findings
are stable (i.e., another study would not change the conclusions).
11
• A “moderate” grade indicates moderate confidence that the estimate of effect lies close to
the true effect for this outcome. The body of evidence has some deficiencies and findings
are likely to be stable, but some doubt remains.
• A “low” grade indicates low confidence that the estimate of effect lies close to the true
effect for this outcome. The body of evidence has major or numerous deficiencies (or
both) and additional evidence is needed before concluding either that the findings are
stable or that the estimate of effect is close to the true effect.
• An “insufficient” grade indicates inability to estimate an effect or no confidence in the
estimate of effect for this outcome, no evidence is available or the body of evidence has
unacceptable deficiencies, precluding reaching a conclusion.
For a more detailed description of the methods and domains used to rate strength of evidence,
see Appendix H.
Other outcomes (e.g., perceptions in Key Question 3) and outcomes for Key Questions 4
through 8 were not formally evaluated for strength of evidence.
Assessing Applicability
Applicability is defined as the extent to which the effects observed in published studies are
likely to reflect the expected results when a specific intervention is applied to the population of
interest under “real-world” conditions.36 It is an indicator of the extent to which research
included in a review might be useful for informing clinical decisions in specific situations.
Applicability depends on the particular question and the needs of the user of a review. There is
no generally accepted universal rating system for applicability. In addition, applicability depends
in part on context. Therefore, a rating of applicability (such as “high” or “low”) was not assigned
because applicability may differ based on the user of a review. Rather, factors important for
understanding the applicability of studies were recorded, such as differences in the organizations
(e.g., payment/reimbursement model, range of services provided, governance structure, IT
systems) and people (e.g., profession, type of relationship with the organization, tenure) affected
by the creation and implementation of the HIE that was the subject of study, the scope of the
HIE, the clinical settings involved, and the geographic area (e.g., states, regions or countries) in
which the studies were performed.
Peer Review and Public Commentary

Experts in HIE, individuals representing important stakeholder groups, and Technical Expert
Panel members were invited to provide external peer review of this systematic review. The
AHRQ Task Order Officer and a designated Evidence-based Practice Center Associate Editor
also provided comments and editorial review. To obtain public comment, the draft report was
posted on the AHRQ Web site for 4 weeks from March 12 to April 8, 2015. A disposition of
comments report detailing the authors' responses to the peer and public review comments will be
made available after AHRQ posts the final systematic review on the public Web site.
12
Results
Results of Literature Searches
Results of the literature search and selection process are summarized in the literature flow
diagram (Figure 2). Database searches resulted in 5,211 potentially relevant citations. After dual
review of abstracts and titles, 849 articles were selected for full-text review. After dual review of
full text articles, 136 studies were included. Data extraction and risk of bias assessment tables for
included studies are available in Appendixes F and I.
Figure 2. Literature flow diagram
HIE = health information exchange; KQ= Key Question

*Cochrane databases include the Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials,
Database of Abstracts of Reviews of Effects, and National Health Sciences Economic Evaluation Database.
†Identified from reference lists, hand searching, suggested by experts, and other sources.
‡Publications may address more than one Key Question, studies may have multiple publications.
Description of Included Studies

Of the 136 studies included in this review, two randomized controlled trials (RCTs)
described in three papers and 32 observational and survey studies addressed Key Questions 1, 2,
13
and 3, pertaining to the effectiveness of improving clinical, economic, population, and
intermediate outcomes. Most were conducted in the United States, although eight were from
Europe, Canada, Israel, and South Korea. Most studies reported clinical or public health process,
economic, or population outcomes, while no studies reported harms of health information
exchange (HIE). The majority were assessed to be of low risk of bias but also contained low-
quality, mostly retrospective evidence. We identified 58 studies that addressed Key Question 4,
pertaining to the use of HIE. The majority were conducted in the United States and were low risk
of bias or could not be rated due to study design. Twenty-two studies were identified that
addressed Key Questions 5 and 6, pertaining to usability and facilitators and barriers to use. Most
were assessed to be of moderate risk of bias and were conducted in the United States, Austria,
and Australia. A total of 52 studies addressed Key Questions 7 and 8, related to HIE
implementation and sustainability. These studies used varying types of qualitative methods; for
those that could be assessed for risk of bias, most were found to have a high risk of bias.
Key Question 1. Is HIE effective in improving clinical, economic, and

population outcomes?
Key Question 2. What harms have resulted from HIE?
Key Question 3. Is HIE effective in improving intermediate outcomes such
as patient and provider experience, perceptions, or behavior; health care
processes; or the availability, completeness, or accuracy of information?
Key Points
• HIE has been studied in far fewer places than it has been implemented, resulting in a
research literature skewed toward a relatively small number of sites.
• Although the potential uses of HIE are broad, most studies reported on narrow questions,
such as reduction in test ordering or consultations, and not larger overall clinical and
financial impacts. Furthermore, most of these studies were conducted retrospectively,
making cause and effect difficult to ascertain.
• The strength of evidence for HIE in improving clinical, economic, or population
outcomes was low.
• Most studies also reported positive results, raising concerns about publication bias.
Detailed Synthesis
We identified 34 studies that assessed some sort of outcome from HIE use (Table 1).
Mapping to our original Key Questions, a total of 26 studies were deemed to report clinical
(intermediate), economic, or population outcomes (Key Question 1), while eight were found to
report on perceptions of outcomes (Key Question 3). However, no studies evaluated primary
clinical outcomes from HIE (e.g., mortality and morbidity - Key Question 1), and none explicitly
assessed harms (Key Question 2). Additionally, some studies reported outcomes for more than
one of the outcomes in the Key Questions. For these reasons, we present the results of Key
Questions 1 through 3 together below.
14
The most common study design for assessing outcomes was retrospective cohort, typically
with HIE use associated with some specific outcome factor.39-56 The next most common design
was survey, which was usually focused on perception of outcomes.57-64 Two studies were RCTs,
one of a particular directed information exchange (2 published papers, 1 on clinical outcomes65
and the other on perceptions66) and the other of a clinical decision support intervention using
data from an HIE implementation.67 Two studies used cross-sectional analyses of large databases
to compare those having access to HIE with those without access.68,69 Two other studies used a
case series methodology, one of which involved asking clinicians if HIE access avoided
undesirable resource use, and then calculating the costs saved70 and the other that retrospectively
analyzed data to determine duplicative testing averted.71
The identified studies were performed mostly in the United States, but we identified eight
studies from five other countries (Austria,62 Canada,65,66 Finland,46,61 Israel,41,72 and South
Korea63). Of the 26 U.S. studies, three assessed multiple HIE implementations in two states (1
study)69 and the entire country (2 studies).64,68 The remaining 23 studies were conducted (1 study
per State unless otherwise noted) in Colorado,50 Indiana (3 studies),42,49,59 Louisiana,47
Massachusetts,60 Minnesota71, North Carolina,67 New York (6 studies),45,51,55-58 Oklahoma,48
South Carolina,70 Tennessee (3 studies),39,40,44 Texas,54 Virginia,43 and Wisconsin (2 studies).52,53
The number of studies and their locations in the United States represent a small fraction of
those reporting to be operational, sustainable, or innovating according to the eHealth Initiative
Annual Data Exchange Survey, which reported a total of 84 such HIE implementations in 201373
and 106 in 2014.74 In other words, while a substantial number of HIE implementations exist in
the United States, only a small number have been subject to evaluation. This low number of
studies relative to HIE efforts also makes it difficult to generalize factors about aspects of them,
such as location, HIE type, and setting, with results of research.
In Table 1, we present the results of these studies by outcome category, classifying the
study’s geographic location, health care setting, HIE type (query vs. directed), and general
direction of the results. Due mainly to study design and performance or reporting limitations, and
the lack of ability to combine results, the strength of this body of evidence was rated as low.
15
Table 1. Studies of HIE included for assessing outcomes
Direction
Risk of
Study Location Setting HIE Type Study Type of Outcome(s) Assessed Results
Bias
Result(s)
Laboratory Testing or Cost of Testing
Mäenpää et al., Tampere, Outpatient Query Retrospective Low Negative Lab test ordering Increased lab
201146 Finland cohort testing
Ross et al., 201350 Mesa County, Outpatient Query Retrospective Low Beneficial Rate of increase in lab Reduced rate of
Colorado cohort testing increase in lab
testing
Carr et al., 201470 Charleston, ED Query Case series Moderate Beneficial Lab testing Reduced lab testing
South Carolina
Frisse et al., Memphis, ED Query Retrospective Moderate Beneficial Lab testing Reduced lab testing
201244 Tennessee cohort
Tzeel et al., 201152 Milwaukee, ED Query Retrospective Low Beneficial ED visit costs Decreased with HIE
Wisconsin cohort use; driven by
reduced testing
Winden, et al., Minnesota ED Query Case series Moderate Beneficial Lab testing Reduction of
201471 duplicate lab testing
Radiology Testing
Bailey et al., Memphis, ED Query Retrospective Low Beneficial Use of neuroimaging Reduced imaging
Bailey, et al., Memphis, ED Query Retrospective Low Beneficial Use of back imaging Reduced imaging
Carr et al., 201470 Charleston, ED Query Case series Moderate Beneficial Use of radiology testing Reduced imaging
South Carolina
Frisse et al., Memphis, ED Query Retrospective Moderate Beneficial Use of radiology testing Reduced imaging
Lammers, Adler- California and ED Varied Cross-sectional Low Beneficial Reimaging in ED Reduced imaging
Milstein, and Florida among those who
Kocher, 201469 implemented HIE
16
Table 1. Studies of HIE included for assessing outcomes (continued)
Direction
Risk of
Bias
Result(s)
Radiology Testing (continued)
Mäenpää et al., Tampere, Outpatient Query Retrospective Low Beneficial Use of radiology testing Reduced imaging
201146 Finland cohort
Ross et al., 201350 Mesa County, Outpatient Query Retrospective Low None Use of radiology testing No impact on
Colorado cohort imaging
reduced testing
Winden, et al., Minnesota ED Query Case series Moderate Beneficial Use of radiology testing Reduction of
201471 duplicate imaging
Hospital Admissions
Ben-Assuli, Israel HMO Query Retrospective Low Beneficial Hospital admissions Decreased with HIE
Shabtai, and cohort use
Leshno, 201341
Ben-Assuli, Israel HMO Query Retrospective Low Beneficial Hospital admissions Decreased with HIE
Shabtai and cohort use
Leshno, 201572
Frisse et al., Memphis, ED Query Retrospective Low Beneficial Hospital admissions Decreased with HIE
201244 Tennessee cohort use
Carr et al., 201470 Charleston South ED Query Case series Moderate Beneficial Hospital admissions Decreased with HIE
Carolina use
Tzeel et al., 201253 Milwaukee, ED Query Retrospective Low Mixed Hospital admissions Increased
Wisconsin cohort Length of Stay admissions but
decreased LOS
Vest, 200954 Austin, Texas ED Query Retrospective Low Beneficial Hospital admissions for Increased with use
cohort ambulatory-sensitive of HIE
diagnoses in indigent
patients
Vest et al., 2014 56 Rochester, New ED Query Retrospective Low Beneficial Hospital admissions Reduced with HIE
York cohort
17
Direction
Risk of
Bias
Result(s)
Hospital/ED Readmissions
Lang et al., 200665 Montreal, ED Directed RCT Moderate None ED return visits No difference
Canada
Vest et al., 2014 55 Rochester, New ED Query Retrospective Low Beneficial Hospital readmissions Decreased with HIE
York cohort use
Jones, Friedberg U.S. All Varied Cross-sectional Low None Hospital readmissions No difference
and Schneider,
201168
Referrals and/or Consultations
Carr et al., 201470 Charleston, ED Query Case series Moderate Beneficial Consultation Reduced with HIE
South Carolina use
Mäenpää et al., Tampere, Outpatient Query Retrospective Low Mixed Referral ordering Increased referrals
201146 Finland cohort with HIE
Emergency Department Costs
Frisse et al., Memphis, ED Query Retrospective Low Beneficial Overall cost Decreased with HIE
201244 Tennessee cohort use
reduced lab testing
Public Health Reporting
Magnus et al., Louisiana Public health Directed Retrospective Low Beneficial Followup care for HIV Improved with HIE
201247 cohort patients
Dixon, McGowan Indiana Public health Directed Retrospective Low None Completeness of public Incomplete due to
and Grannis, cohort health reporting poor quality of
201142 clinical data
Overhage et al., Indiana Public health Directed Retrospective Low Beneficial Identification and Increased notifiable
200849 cohort completeness of diseases found and
notifiable disease completeness of
reporting data for diseases
found
18
Direction
Risk of
Bias
Result(s)
Quality of Ambulatory Care
Kern et al., 201245 Hudson Valley, Outpatient Query Retrospective Low Beneficial Clinical quality Increased with HIE
New York cohort measures
Nagykaldi et al., Norman and Outpatient Query Retrospective Moderate Beneficial Clinical quality Increased with HIE
201448 Oklahoma City, cohort measures
Oklahoma
Willis et al., 201367 North Carolina Outpatient Query RCT Moderate Beneficial Documentation and Increased with HIE
medication
reconciliation
Other Aspects of HIE
Feldman and Virginia Government Directed Retrospective Moderate Beneficial Case processing time Decrease in mean
Horan, 201143 cohort for SSD determination case processing
time
Shapiro et al., New York ED Query Retrospective Moderate Beneficial Identification of Increased with HIE
201351 cohort frequent ED users
Vest and Miller, U.S. Hospital Varied Cross-sectional Low Beneficial Patient satisfaction with Higher in
201164 hospital care implemented than
adopted hospitals
CDS = clinical decision support; CQI = continuous quality improvement; ED = emergency department; HIE = health information exchange; HMO = health maintenance
organization; LOS = length of stay; PCP = primary care provider; PH = public health; RCT = randomized, controlled trial; SSD = Social Security Disability;
VA = Veterans Affairs; vs. = versus
19
With the exception of two RCTs (in 3 publications) and one other study with a prospective
design, most studies used retrospective designs, usually with an approach examining the
association of HIE use with one or more clinical variables. All of these studies focused on the
direct effect of HIE, usually in reducing resource use or costs, without determining its larger
impact (e.g., overall total or proportion of spending in an emergency department [ED] vs. the
total dollar amounts that HIE appeared to save). None of the studies analyzed individual episodes
of care to determine clinical appropriateness of possible changes brought about by HIE use.
The prospective studies also had limitations. The RCTs were focused on highly specific uses
of HIE, namely directed exchange of ED reports in one and pharmacotherapy clinical decision
support in another. Of note, however, was that neither study showed benefit of HIE. The other
prospective study was limited by methodology of physicians self-reporting of resources not
utilized when HIE was used, with no followup or validation of their decisions, or analysis of
more holistic views of clinical outcomes or costs.
While most of these studies had reasonable internal validity, questions of external validity
remain, especially since the intervention (HIE) was only one of many potential influences on
clinical outcome (i.e., many more factors go into clinical outcomes than the decision to consult
an HIE system on a patient). As a result, most studies with appropriate retrospective methods are
listed as having low or moderate risk of bias due to their proper internal validity but there are still
significant concerns about external validity.
Improving Resource Use

Laboratory Testing
Six studies addressed laboratory testing, with five finding a benefit of HIE in reducing
overall tests, although estimates of impact on cost were mixed.44,46,50,52,70,71 Four of these studies
took place in the ED setting, all showing some aspect of reduced testing and cost savings. Two
studies found overall reduced laboratory testing, with one reporting an odds ratio (OR) of testing
among patients for whom HIE was accessed to be 0.880 (95% confidence interval [CI], 0.828 to
0.935)44 and the other noting 23 percent fewer lab testing procedures (statistical significance not
reported) in a propensity-matched group of patients for whom HIE could have been used.52 A
third study logged physician self-reports of laboratory testing averted with use of HIE in the ED,
with savings over 3 months of $462 calculated from tests reportedly not ordered.70 A fourth
study found 96 instances of duplicate lab testing averted in 1,488 patient encounters that were
retrospectively analyzed.71Two studies were conducted in ambulatory settings, against a
backdrop of increased overall laboratory testing. One U.S. study found that after HIE
implementation, there was a reduction in the rising rate of testing, without overall cost savings.50
In contrast, a study in Finland found increased laboratory testing during the period of HIE
implementation (19.0% for primary care physicians and 7.0% for specialist physicians per total
patient appointments).46 As with all retrospective studies, the four studies of laboratory testing
could have been complicated by confounders, while the prospective study did not validate
physician self-reporting of tests avoided or measure overall costs of care for the ED encounter or
subsequent utilization.
20
Radiology Testing
Nine studies assessed radiology testing, with all but one reporting an association of reduced
testing with HIE.39,40,44,46,50,52,69-71 Six of these studies also examined laboratory testing and are
described previously,44,46,50,52,71,75 and three additional ED studies assessed only imaging.39,40,69
The ED studies showed a variety of findings. One study found that for all radiologic imaging,
there was reduction of head computed tomography (CT) imaging, (OR of 0.913, 95% CI, 0.842
to 0.991) as well as body CT imaging (OR 0.886, 95% CI, 0.828 to 0.948) but no significant
changes in echocardiogram, chest x-ray, or ankle x-ray testing across 12 EDs.44 Another study
demonstrated 22 percent decreased diagnostic radiology ordering and 52 percent reduced CT
scan ordering (statistical significance not reported) when HIE was used in the ED.52 Two
additional studies assessed neuroimaging for headache39 and repeat imaging for back pain in
EDs.40 For neuroimaging, HIE usage was associated with decreased diagnostic imaging (OR
0.38; 95% CI, 0.29 to 0.50) and increased adherence to evidence-based guidelines (OR 1.33;
95% CI, 1.02 to 1.73), although there was no significant change in overall costs. HIE usage was
associated with reduced repeat imaging for back pain (OR 0.36; 95% CI, 0.18 to 0.71), but no
change in cost due to higher use of CT scans with HIE access. A prospective case series study
reported $161K in savings over 3 months through averted radiologic testing in EDs,70 while a
retrospective case series found 453 duplicate radiology testing in 1,488 patient encounters
retrospectively analyzed.71
One cross-sectional study looked at repeat imaging in the ED in two states (California and
Florida), finding reduced probability of repeat CT (-8.7%; 95% CI, -14.7% to -2.7%), ultrasound
(-9.1%; 95% CI, -17.2% to -1.1%), and chest x-ray (-13.0%; 95% CI, -18.3% to -7.7%) ordering
in hospitals that had HIE participation as reported in the Healthcare Information and
Management Systems Society Analytics Database of hospital information technology (IT)
functionality.69
In ambulatory settings, one U.S. study showed no statistically significant reduction in the rate
of radiologic testing.50 However, a Finland-based study showed a reduction in radiologic testing
(16.4% reduction for primary care physicians and 11.0% reduction for specialist physicians).46
Hospital Admissions
Eight studies assessed the role of HIE in reducing hospital admissions, with inconsistent
findings.41,44,53,54,56,65,70,72 Two studies (described above) found a reduction in hospital admissions
and lower costs using methods previously described. The bulk of the $1.07 million annual
savings due to HIE found in one study resulted from reduced admissions.44 Another study also
reported $118K in savings from averted admissions over a 3-month period.70 Two studies in an
Israeli health maintenance organization found that viewing the medical history via an electronic
health record (EHR) decreased possibly redundant admissions, with even greater reductions
when information was accessed using HIE.41,72 A study in New York found that viewing
information reduced odds of admission (OR 0.70; 95% CI, 0.52 to 0.95).56
Other studies, however, found no benefit from HIE in terms of avoiding hospital admissions.
An RCT of directed HIE in Canada providing family physicians electronic reports of ED visits
versus paper-based reports resulted in no difference in hospital admissions or return visits to the
ED.65 Other studies found that HIE was associated with increased admissions for ambulatory-
sensitive diagnoses54 and a 28 percent increased rate of admissions, although such admissions
had reduced length of stay with 771 fewer bed days per 1,000 health plan members over 16
months.53
21
Two studies assessed HIE in reducing hospital readmissions. One study found that assessing
information in an HIE implementation was associated with reduced odds of hospital readmission
(OR 0.43; 95% CI, 0.27 to 0.70)55 while another found that U.S. hospitals participating in HIE in
2007 did not have lower readmission rates for acute myocardial infarction, pneumonia, or heart
failure.68
Referrals and Consultations

Two studies, described previously, assessed HIE for reducing referrals and/or consultations.
The prospective ED case series reported reduced consultations, leading to savings of $3,990 over
3 months.70 The Finland-based ambulatory study, however, found that HIE was associated with
increased referrals by primary care physicians (43.6%) and specialists (12.8%).46
ED Cost
Another two studies addressed reducing overall ED costs per patient, with both finding
reductions when HIE was available. One study found that an HIE system encompassing 12 EDs
resulted in net annual savings (total savings minus operating costs) of $1.07 million, with
reduced hospital admissions accounting for 97.6 percent of the reduction.44 Another study found
that for a propensity-matched group of patients for whom HIE could have been used, the group
for whom HIE was used had $29 per ED visit less expenditures.52 Neither study reported overall
ED expenditures, making it unknown what proportion of overall ED spending was impacted by
HIE.
Public Heath Reporting

Three studies assessed HIE in public health settings, all of which were conducted in the
United States.42,47,49 Two examined the completeness of notifiable disease reporting data. One
study compared usual (“spontaneous”) public health reporting with automated lab reporting
through the HIE, finding a 4.4-fold higher rate of reporting for the HIE-based approach, with
cases identified an average of 7.9 days earlier.49 The other study showed equal or improved
completeness of reporting for a variety of data fields in notifiable disease reports, although
completeness was reduced for some fields (e.g., laboratory units of measure, normal range, and
abnormal flag) due to inadequacies in the clinical data entering the HIE.42 Another study found
that a public health HIE led to increased identification of needed followup care of 419 HIV
patients and 85 percent of them having actual followup care.47
Quality of Care
Three studies looked at the value of HIE in improving quality of care in ambulatory
settings.45,48,67 One study assessed a benchmark group of clinical quality measures believed to be
amenable to HIE usage among users and nonusers of an HIE portal. Users of the portal had a
higher proportion of physicians exceeding mean clinical quality measure performance at baseline
(57% vs. 48%) that increased after the HIE became available (64% vs. 49%), with the increase
for portal users before and after availability of the HIE statistically significant (p<0.001).45 An
RCT of HIE data used in a clinical decision support intervention was able to detect medication
adherence problems in eight categories of drugs but did not show any benefit in improving
adherence by patients in taking medications prescribed based on evidence-based guidelines.67
Another study of six physician practices found improved documentation and delivery of
preventive services for mammography screening (21.1% to 57.1%, p<0.01), colonoscopy
22
screening (31.7% to 53.8%, p<0.01), pneumococcal vaccine administration (39.1% to 50.6%,
p<0.01), and influenza vaccine administration (22.7% to 41.7%, p<0.01).48 The study also found
that medication reconciliation completion improved from 35.3 percent to 44.9 percent (p<0.001).
Other Aspects of HIE

Three studies assessed other aspects of HIE. One study found a 30 percent reduction in
evaluation time for Social Security Disability claims.43 Another found that HIE data led to a 20.3
percent increase in identifying frequent ED users compared with site-specific data.51 An
additional study focused on hospital-based HIE, finding that communication and satisfaction
(based on the Hospital Consumer Assessment of Healthcare Providers and Systems survey) were
higher in hospitals that implemented HIE compared with those that proposed to implement
HIE.64
Although the risk of bias in most studies was low, the resulting evidence from them was
mostly of low quality. This low strength evidence mostly favored the value of HIE in reducing
resource use and costs, especially in the ED. However, these studies used mostly retrospective
designs that cannot account for how HIE was used and its impact on the overall care of the
patient beyond the immediate setting where it was used.
Perceptions
A number of studies evaluated clinician or patient perceptions of HIE (Table 2).57-64,66 Three
studies assessed clinician perceptions of HIE in the ED setting. One study followed up an RCT
on the provision of an electronic versus mailed report after an ED visit,65 with family physicians
reporting improved patient management and followup in ED settings. 66 Another study also
found that primary care physicians reported enhanced awareness and improved communication
and followup with primary care physicians after ED admission/discharge.57 An additional study
found that providing pharmacy information to physicians in the ED improved knowledge and
gaps but was not felt to reduce time or be worth the cost.60
Other studies assessed perceptions in the outpatient setting. Two studies found that HIE was
perceived to improve ambulatory care function, resulting in faster acquisition and treatment
decisions61 and improved care and decreased work for filing and archiving discharge reports that
were sent.62
Some studies looked at specific aspects of HIE. One study found that physicians were more
satisfied with electronic lab reports than with paper-based reports.59 Another queried physicians
on push versus pull HIE, with respondents reporting satisfaction with both, although more so
with push over pull.58 An additional study assessed patient satisfaction when records were
transferred via HIE, finding it to be improved over patients delivering paper records
themselves.63
Clinician perceptions of the value of HIE, where studied, were generally positive. How such
perceptions translate into improved care is unknown. This body of evidence was of low strength.
23
Table 2. Patient and clinician perceptions of HIE
Study
Direction
HIE Type, Risk of
Study Location Setting of Perception(s) Assessed Results
Type Data Bias
Result(s)
Source
Afilalo, et al., Montreal, ED Directed RCT, Moderate Beneficial Outcomes improved, better Improved with HIE
200766 Canada survey patient management
Altman et al., New York ED Directed Cross- Moderate Beneficial PCP notification of ED Enhanced awareness
2012 57 sectional admission/discharge and improved
survey communication and
followup
Campion et al., Rochester and Outpatient Both Cross- Moderate Beneficial Physician satisfaction of push Satisfied with both,
201258 Buffalo, New sectional, vs. pull more with push than
York survey pull
Chang et al., Indiana Outpatient Query Cross- Moderate Beneficial Physician satisfaction with Favorable, including
201059 sectional, electronic lab reports over traditional reports
survey
Kaushal et al., Massachusetts ED Directed Cross- High Mixed Impact of providing pharmacy Improved knowledge
201060 sectional, information and gaps but not felt
survey to reduce time or be
worth the cost
Maass et al., Finland Outpatient Query Cross- High Beneficial Improvements in care When HIE used,
200861 sectional, faster results
survey acquisition and
treatment decision
Machan, Tyrol, Austria Outpatient Directed Cross- Low Beneficial Physician satisfaction with Improved care and
Ammenwerth, sectional, discharge reports sent decreased work for
and survey filing and archiving
Schabetsberger,
200662
Park et al., South Korea Outpatient Directed Cross- Low Beneficial Patient perceptions of data Increased satisfaction
201363 sectional, transferred for patients whose
survey records transferred via
HIE
ED = emergency department; HIE = health information exchange; RCT = randomized controlled trial; U.S. = United States
24
Factors Associated With Outcomes
To determine whether effectiveness of HIE varied by location, health care setting, or
outcome type, we rated each study outcome by whether HIE was found to have some beneficial
effect or not. As shown in Table 3, the preponderance of studies showed that HIE use for
different functions, in various settings, and of varying types was mostly positive. While the
number of positive versus negative studies was not an indicator of the overall direction of the
evidence, we did note that for each “negative” study, there is at least one “positive one.
For “Type of HIE,” there was no clear pattern of findings to suggest that one type is clearly
better than another, even indirectly. The two RCTs we found were described in three papers.
Two of these reported outcomes, one for each RCT, both of which showed no benefit for the HIE
intervention.65,67 A perceptions study of one of the RCTs found perceptions of improved patient
outcomes and their management.66 These are in contrast with the observational study designs
where 96 percent found beneficial effects of HIE. This is somewhat typical in comparing RCT
and observational study results, likely due to confounding. For HIE setting, only ambulatory and
ED have enough studies to evaluate patterns, with outpatient settings less likely to find beneficial
results compared with studies in ED settings, but again based on indirect comparisons only.
The sparseness of studies across geographic settings does not allow for identification of patterns,
although across most studies in the United States, the findings were positive.
Table 3. Factors that may affect outcomes

Studies Studies
Studies of Studies of
Factor Reported as Reported as No Total
Outcomes Perceptions
Beneficial Benefit
Study Type
Retrospective cohort 20 19 1 20
Randomized controlled trial 2 1 1 2 3
Cross-sectional 2 1 1 2
Case series 2 2 2
Survey* 8 8 8
Setting
All 1 1 1
Emergency department 13 3 13 3 16
Government 1 1 1
HMO 2 2 2
Hospital 1 1 1
Outpatient 5 5 9 1 10
Public health 3 3 3
Location
U.S. multistate 3 2 1 3
Colorado 1 1 1
Indiana 2 1 3 3
Louisiana 1 1 1
25
Table 3. Factors that may affect outcomes (continued)
Studies Studies
Factor Reported as Reported as No Total
Outcomes Perceptions
Beneficial Benefit
Massachusetts 1 1 1
Minnesota 1 1 1
North Carolina 1 1 1
New York 4 2 6 6
Oklahoma 1 1 1
South Carolina 1 1 1
Tennessee 3 3 3
Texas 1 1 1
Virginia 1 1 1
Wisconsin 2 1 1 2
Austria 1 1 1
Canada 1 1 1 1 2
Finland 1 1 1 1 2
Israel 2 2 2
South Korea 1 1 1
HIE Type
Directed 5 5 8 2 10
Query 18 2 19 1 20
Multiple 3 1 3 1 4
HIE = health information exchange; HMO = health maintenance organization; vs. = versus
*1 survey study was also an RCT.
Key Question 4. What are the current level of use and primary uses of HIE?
Key Points
• More than three-quarters (76%) of non-Federal acute care hospitals electronically
exchanged laboratory results, radiology reports, clinical care summaries, and/or
medication lists with any outside providers in 2014. This represented an 85 percent
increase since 2008 and a 23 percent increase since 2013. Close to seven in 10 hospitals
(69%) electronically exchanged health information with ambulatory providers outside of
their organization, representing a 92 percent increase since 2008 and a 21 percent
increase since 2013.
• A variety of HIE models are employed across settings. Hospitals and ambulatory care
providers both provide and use data; while laboratory services provide data and
community clinics use data. At least 50 percent of these organizations are reaching an
advanced stage of use of core functionalities; many supporting health care reform
initiatives and advanced analytics.
• Use varies by type of health care professional, with higher use by nurses and clerks, when
compared with physicians. Patient engagement remains low.
• Use is increasing in ambulatory care practices, with a 2013 estimate of 38 percent of
practices using HIE. Characteristics of higher HIE use being larger practice size, practice
26
owned by a health system (vs. physician owned), and multispecialty (vs. single specialty)
practice.
• HIE use in long-term care settings is low (<1%), with the consistent pattern of nonprofits
enjoying wider use than for-profit entities. Less than four in ten residential care facilities
that use EHRs also exchange health information.
• Results of regional and statewide studies that evaluate HIE use in inpatient, outpatient,
community clinic, or EDs suggest that HIE is used for few patients; the extent of HIE use
is low. Results of international/multi-national studies suggest the same finding.
• HIE use was in its infancy in the 2000s but has been steadily increasing since then.
– A recently released 2015 report from the ONC suggests that the United States is
making great progress in exchanging health information.
– HIE is particularly useful in the ED and in the ambulatory setting to alert
providers to inpatient or ED events recently experienced by patients.
• Patients also seem willing to consent to data exchange, as long as the benefits of doing so
are clear to them.
Detailed Synthesis
We identified 58 studies that described the levels of use and primary uses of HIE (Tables 4-
7). Several methods were used by investigators to answer questions about HIE use, including
surveys (25 studies),25,26,73,74,76-96 analyses of HIE audit-logs (13 studies),40,45,54,97-106
retrospective database analyses (9 studies),107-115 and mixed methods (7 studies).116-122 Two
studies used focus group methods,123,124 one study used time-motion methods,61 and another used
geo-coding.125
Over one-half of the studies (30 of 58) analyzed HIE implementations over a regional or
statewide area,45,54,76,77,83-86,88,90,92,96-106,112,118-120,123-126 while an additional 15 evaluated HIE use
nationally.25,26,78-81,87,91,93,107-111,113 Of those that evaluated use regionally or over a statewide area,
10 studies evaluated HIE implementations in the State of New York,45,76,77,96-98,102,106,112,125 five
in Texas,54,101,103-105 five in Tennessee,40,86,99,118,119 two in Indiana,88,92 and two in Minnesota.85,90
Five studies evaluated HIE in a single State (Massachusetts,123 North Carolina,100 Wisconsin,84
Northeastern Ohio,120 and Louisiana124).
Two studies evaluated HIE use across integrated delivery systems. One exchanged data
between the Department of Veterans Affairs (VA), the Department of Defense (DoD), and non-
Federal care organizations,116 and the other between the VA and Kaiser Permanente.82 Seven
studies evaluated HIE use outside of the United States61,89,94,114,115,121,122 and two in multiple
countries including the United States.95,117
The majority of studies evaluated HIE use across inpatient and ambulatory care settings.
Seven studies were limited to evaluations of HIE use in hospitals,76,88,96,107,108,111,117 three of these
used data from the American Hospital Association (AHA).107,108,111 Four studies evaluated HIE
use that involved exchange of data with nursing homes or residential care facilities; two using
data from the National Nursing Home Survey and the National Survey of Residential Care
Facilities,93,113 the other two using data from New York State.77,112 Three studies focused on
evaluating HIE use in the ED; all of these exchanged data regionally.40,99,100 Two studies focused
on evaluating HIE use in office settings using data from the National Ambulatory Medical Care
Survey,91,110 three others used within State data, one from Indiana92 and two from Minnesota.85,90
27
The majority of studies assessed overall use of the HIE, while two assessed the use of HIE
for repeated imaging in the ED,40,102 and two evaluated HIE for prevention or tracking of
infections.83,88
Twenty-seven studies included data collected in 2010 or more recently;25,26,73,74,77,83,88,90-
98,102,106,108-113,120,124,125
the majority of studies used data collected in 2009 or earlier. Fifteen
studies used a query-based HIE;40,54,86,97-99,101-105,118-120,125 the other studies either did not specify,
or multiple HIE implementations were included.
Twenty-nine of the studies were rated as being at low risk of bias;25,26,40,54,76-
81,83,86,88,91,93,94,100,101,103-105,107-111,113,121,125
nine at moderate risk of bias;84,85,90,92,95,96,102,112,122 six at
61,87,89,114,117,120
high risk of bias; and fourteen were not rated due to the type of study design (data
from audit-logs or qualitative studies).45,73,74,82,97-99,106,115,116,118,119,123,124
Level of Use and Primary Uses: Type of HIE

The majority of the studies used a variety of types of HIE, and did not describe these in
detail. Data describing the type of HIE, according to the classification system promulgated by the
Office of the National Coordinator (direct, query-based, or consumer-mediated) were limited to
studies wherein a specific HIE was evaluated. Of these, query-based HIE systems were noted for
evaluations of the MidSouth e-Health Alliance (MSeHA),40,86,99,118,119 the Central Texas HIE (I-
Care),54,101,103-105 the Health Care Efficiency and Affordability Law for New Yorkers Capital
Grant Program (HEAL-NY),97,98,102,125 and the Northeast Ohio Public Health Care System.120
Level of Use and Primary Uses: Health Care Settings and Systems
This summary of HIE use by health care setting and systems (Key Question 4b) has been
combined with the summary by IT system characteristics (Key Question 4c), and data sources
(Key Question 4d) to provide the summary below. Little meaningful information was found on
the use of HIE by provider type (also Key Question 4b) so, when available, this information is
also incorporated into this section.
Participation in HIE, Types of Data Exchanged, Characteristics of

Successfully Participating Organizations (United States–Wide Surveys)
Six studies used survey methods to investigate the frequency of data exchange and types of
data exchanged across regional health information organizations (RHIOs) across the United
States (Table 4).25,78-81,87 Across these studies, between 138 and 207 organizations met the
definition of a RHIO; while between 20 and 81 RHIOs provided data. These data, collected from
2006 through 2012, suggest that entities most commonly providing data are hospitals (83%),
followed by ambulatory settings (60%); and that the entities most commonly receiving data were
ambulatory settings (95%), followed by hospitals (83%), public health departments (50%), and
payers (44%).81 Using survey data collected in 2007, Hessler, et al. focused on the exchange
between RHIO and State and local public health departments, and found that of 138 public health
agencies, 50 (36%) had no RHIO in their jurisdiction; 16 (12%) had no relationship with a
RHIO, and 26 (40%) were exchanging information. Twelve of 20 RHIOs were exchanging
information; seven of these (35%) with public health entities.87 The types of data most frequently
exchanged were laboratory test results (84% to 90%),78,81,87 inpatient data (70%), medication
histories (70%), and outpatient data (60%).78,81 In 2008 and 2009, of 75 operational RHIOs,
covering 14 percent of U.S. hospitals and 3 percent of ambulatory practices, only 13 supported
the criteria for meaningful use criteria of the Health Information Technology for Economic and
28
Clinical Health Act (3% of hospitals and <1% of ambulatory practices),79 while by 2012, there
had been a 61 percent increase in the number of operational RHIOs, from 75 to 119.25
Two additional surveys were conducted by the eHealth Initiative 73,74 One-hundred, ninety-
nine of 315 identified HIE organizations completed the 2013 annual survey. These HIE entities
were a mix of community-based, State-based, and health care delivery organizations. Results
indicate there is no single dominant model of HIE. Ninety organizations use a ‘Direct’ standards-
based protocol for securely exchanging data, mostly for transitions in care. Patient opt out was
the most common consent model, although patient engagement remains low amongst
organizations exchanging data. Eighty-four organizations had reached an advanced stage of
operation or innovation; most took 2 years to become operational. Among organizations that
responded in both 2011 and 2013, 27 more had reached stages 5 (operating), 6 (sustaining), or 7
(innovating) on the eHealth Initiative’s maturity scale, in 2013. Hospitals and ambulatory care
providers are the stakeholders most commonly providing/viewing data; independent laboratories
also commonly provide data. Community and public health clinics commonly view data. HIE
organizations are focusing on functionalities to support health care reform initiatives and
advanced analytics.
The number of HIE organizations identified and that responded in 2014 was lower than in
2013, with 126 of 267 identified responding in 2014.74 Again, there was a mix of community-
based, State-based, and health care delivery organization-based HIE entities responding. Data
were provided by hospitals, ambulatory care providers, laboratories, and community/public
health clinics. Data were accessed by ambulatory care providers, hospitals, community/public
health clinics, and behavioral or mental health providers. Findings suggest an 11 percent increase
over 2013 in the proportion of organizations that have reached stage 6 (operating) or higher (106
organizations). Uses of HIE included support for an accountable care organization to improve
patient outcomes, for a patient centered medical home, for a State Innovation Model, and for a
bundled payment initiative. Results suggest data exchange is reaching a point of stability and
acceptance, and that organizations are settling on a set of core services offerings.26
Nine studies investigated HIE use retrospectively, using U.S.-wide survey data collected for
other purposes, with an information technology add-on.26,91,107-111,113 Four of these used data
from the AHA,26,107,108,111 two from the National Ambulatory Medical Care Survey,
(NAMCS),91,110 and one each from the Commonwealth Fund Health Policy Surveys,109 the
National Nursing Home Survey/National Survey of Residential Care Facilities,111 and, another
from the National Survey of Residential Care Facilities.93
These studies investigated overall participation in HIE use. Results suggest that HIE use by
hospitals has risen from 11 percent (2009)78 to between 30 percent and 58 percent more
recently.108,109,111 Results from the recently released ONC brief suggest that more than three-
quarters (76%) of non-Federal acute care hospitals electronically exchanged laboratory results,
radiology reports, clinical care summaries, and/or medication lists with any outside providers in
2014. This represents an 85 percent increase since 2008 and a 23 percent increase since 2013.
Close to seven in 10 hospitals (69%) electronically exchanged health information with
ambulatory providers outside of their organization, representing a 92 percent increase since 2008
and a 21 percent increase since 2013.26 Characteristics associated with higher use are nonprofit
status, presence of an EHR system, larger market share, and larger practices.107-109,111 Results
from the NAMCS (2011) suggest that the majority of office-based physicians reported being able
to both send and receive data; 64 percent of these exchanges were through an EHR vendor and
28 percent through a hospital system. Activities included viewing laboratory results and
29
incorporating these into the EHR, and exchanging clinical summaries with patients. Primary care
providers were more likely to use HIE than specialists.91 Results from the NAMCS (2013)
suggest that 39 percent of office-based physicians reported having HIE capability with other
providers or hospitals. Characteristics of higher HIE use were larger practice size, practice
owned by a health-system (vs. physician owned), and multispecialty (vs. single specialty)
practice.110 Data from the National Nursing Home Survey (2004) and the National Survey of
Residential Care Facilities Survey, both from the Centers for Disease Control and Prevention,
indicate that HIE use in these settings is low, with the consistent pattern of nonprofits enjoying
wider use than for-profit entities.113 Finally, recent data from the National Survey of Residential
Care Facilities suggest that 23 percent of residential care communities that use EHRs also
exchanged health information. Nearly 25 percent could exchange with pharmacies and 17
percent with physicians.93
30
Table 4. Level of use and primary uses of HIE: participation in HIE, types of data exchanged, and characteristics of successfully
participating organizations (United States–wide studies)
Geographic Risk of Outcome(s)
Study Setting HIE Type Study Type Results
Location Bias Assessed
Adler-Milstein, U.S.-wide RHIOs Varies Cross- Low Participation in RHIO -Most common entities providing and receiving data:
et al., 200881 sectional 83% of hospitals; 67%-95% of ambulatory settings;
survey Types of data 50% of public health departments; 44% of payers.
exchanged. -Types of data exchanged: Test results: 60%-90%;
Inpatient data: 70%; Medication histories: 70%;
Outpatient data: 60%; Images: 56%.
Adler-Milstein, U.S.-wide RHIOs Varies Cross- Low Types of data -Types of data exchanged: Test results: 84%;
Bates, and sectional exchanged. Inpatient data: 70%; Medication histories: 66%;
Jha, 200978 survey Outpatient data: 64%.
Adler-Milstein, U.S.-wide RHIOs Varies Cross- Low Characteristics of -Likelihood of being operational associated with
Landefeld, and sectional successful exchanging narrow set of data and involving broad
Jha, 201080 survey participation. group of stakeholders
Adler-Milstein, U.S.-wide RHIOs Varies Cross- Low Number of -75 operational RHIOs, covering 14% of U.S.
Bates, and Jha sectional operational RHIOs hospitals and 3% of ambulatory practices.
201179 survey supporting stage 1 -13 RHIOs support stage 1 meaningful use
meaningful use; (covering 3% of hospitals and 0.9% of ambulatory
number financially practices).
viable.
Adler-Milstein, U.S.-wide RHIOs Varies Cross- Low Participation in RHIO. -61% increase from 2011 (75 to 119 RHIOs).
Bates, and sectional -Types of data exchanged: Test results: 82%;
Jha, 201325 survey Types of data Summary records: 79%; Discharge records: 66%;
exchanged. Clinical summaries: 61%
-Predominant organization was nonprofit.
Characteristic of
successful
organization.
Hessler, et al., U.S.-wide RHIOs Varies Cross- High Participation in RHIO. -RHIOs:
200987 sectional -12/20 (60%) are exchanging information
survey -7/20 (35%) with Public Health
-Type of data exchanged most frequently: Test
results: 86%.
-Public health agencies:
-50 (36%) have no RHIO in jurisdiction.
-16 (12%) have no relationship with RHIO.
-26 (40%) are exchanging information.
31
participating organizations (United States–wide studies) (continued)
eHealth U.S.-wide All Varies Cross- Not rated Participation in HIE. -84 organizations had reached 'advanced' stage of
Initiative, sectional due to study operation, sustainability, or innovation.
201373 survey design Stage of maturity. -27 more had reached stages 5 (operating), 6
(sustaining), or 7 (innovating) on the eHealth
Key findings. Initiative's HIE maturity scale in 2013 than in 2011.
-Hospitals and ambulatory care providers most
commonly providing/viewing data, followed by
laboratories and community public health clinics.
-Most took 2 years to become operational.
Key findings:
1) Exchanges are focusing on functionalities to
support health reform and advance analytics.
2) Patient engagement remains low amongst
organizations exchanging data.
Swain, et al., U.S.-wide Non- Varies Retrospective Low HIE use between More than three-quarters (76%) of non-Federal
201526 Federal database hospitals and acute care hospitals electronically exchanged
acute care analysis of hospitals; laboratory results, radiology reports, clinical care
hospitals AHA data HIE use between summaries, and/or medication lists with any outside
and hospitals and outside providers. This represents an 85% increase since
outside providers; 2008 and a 23% increase since 2013. Close to
providers Types of data seven in ten hospitals (69%) electronically
exchanged (Labs, exchanged health information with ambulatory
radiology, meds, providers outside of their organization, representing
clinical care a 92% increase since 2008 and a 21% increase
summaries) since 2013.
32
eHealth U.S.-wide All Varies Cross- Not rated Participation HIE. Provides data: 112 hospitals, 100 ambulatory care
Initiative, sectional due to study providers, 56 laboratories, 52 community/public
201474 survey design Stage of maturity. health clinics.
Accesses data: 111 Ambulatory care providers, 104
Key findings. hospitals, 75 community/public health clinics, 65
behavioral or mental health providers.
Key findings: 106 had reached stage 6 (sustaining)

or higher on the eHealth Initiative's HIE maturity
scale (an increase of 11% over 2013).
64 support an accountable care organization; 52
support a Patient Centered Medical Home; 21
support a State Innovation Model; 12 support a
bundled payment initiative.
Looking to the future
1) Data exchange is reaching a point of stability and
acceptance.
2) Organizations are settling on a set of core service
offerings.
3) As organizations mature, they will offer new and
innovative services (public health has already
leveraged HIE; alert notification services may help
accountable care organizations to track patients).
Adler-Milstein, U.S.-wide Hospitals Varies Cross- Low Participation in HIE. 11% of hospitals engaged in HIE.
DesRoches, sectional Use significantly higher for private/nonprofit status,
and Jha, review of Characteristics of greater market bed share, teaching status, large
2011107 database successful size, presence of cardiac ICU, and presence of EHR
analysis of organizations. system.
AHA data
Adler-Milstein U.S.-wide Hospitals Varies Cross- Low Participation in HIE. 30% of hospitals engaged in HIE.
and Jha, sectional Use significantly higher for private/non-profit status;
2014108 Measurement Characteristics of greater market bed share, in less competitive
of HIE usage successful market.
among U.S. organizations. Varies widely by State.
hospitals
33
Furukawa, et U.S.-wide Hospitals Varies Cross – Low Participation HIE. -In 2012, 58% of hospitals exchanging data, 41%
al., 2013111 sectional increase over 2008, (p<0.01).
survey Types of data -In 2012, 51% of hospitals exchanging with
exchanged. unaffiliated ambulatory providers, 36% with other
hospitals outside their organization.
Characteristics of -In 2012, 52%, 53%, 35% and 33% exchanging
successful images, laboratory tests, care summaries,
organizations. prescription lists with outside providers, respectively
(39%, 51%, 40%, 55% increase, respectively)
-After adjusting for hospital and area characteristics,
hospitals with basic EHR and participation in health
information organizations (HIOs) had highest rates
of exchange activity.
-In 2012, 80% of hospital with EHR and HIO were
exchanging, 71% with HIO but no EHR were
exchanging; 60% with EHR but no HIO were
exchanging.
-All consistent across different providers types and
clinical information types.
-Hospital characteristics associated with lower
exchange rates were rural, for-profit, locations with
greater Medicare part A spending.
34
Patel, et al., U.S.-wide Ambulator Varies Cross – Low Participation in HIE. -31% of offices could share clinical summaries.
201391 y Care sectional -Of these, 76% could both send and receive.
survey Types of data -64% of these exchanges were through an EHR
exchanged. vendor; 28% through a hospital-based system.
-55% e- prescribe, 67% view laboratory results, 42%
Characteristics of incorporate lab results into EHR.
successful -State differences: the capacity to electronically
organizations. exchange clinical summaries with patients varied
from 55% (Minnesota) to 18% (Louisiana).
-Proportion of physicians who exchange clinical
summaries with other providers varied from 61%
(Wisconsin) to 15% (Alabama).
-Adoption of EHR strongest practice characteristic
associated with exchange capacity, p<.001.
-EHR vendors have wide range of capacities for
exchange: 24% to 77%.
-Primary care providers more likely to exchange vs.
specialists.
Furukawa, et U.S.-wide Ambulator Varies Cross – Low Participation in HIE. -39% of office-based physicians reported having
al., 2014110 y care sectional HIE capability with other providers or hospitals.
survey Characteristics of -Characteristics of higher HIE use were larger
successful practice size (vs. solo), practices owned by health-
organizations. systems (vs. physician owned); multispecialty
practices (vs. single specialty).
Audet, Squires, U.S.-wide Ambulator Varies Cross- Low Participation in HIE. -32% of physicians engage in HIE.
and Doty, y care sectional -Use significantly higher for practices that have
2014109 analysis of Characteristics of higher proportion for formal IT support, are part of
database successful an integrated system, larger practices, presence of
organizations. EHR system, and receiving financial incentives.
-Use significantly increased since 2009.
35
Hamann and U.S.-wide Nursing Varies Cross- Low Participation in HIE. For profit vs. nonprofit:
Bezboruah, Homes sectional -Percent residential care facilities using HIE: 0.14%
2013113 analysis of Characteristics of vs. 0.21%; p<0.00.
two survey successful Number of partners in HIE: 0.32% vs. 0.42%;
databases organizations. p=0.02.
-For profits less likely to participate in HIE; OR
0.663, p<0.001.
-Supports hypothesis and proposed framework for
why non-profits are more likely to use health IT.
Caffrey and U.S.-wide National Varies Cross- Low Use of HIE among 23% used computerized systems for exchanging
Park-Lee, Survey of sectional residential care health information with pharmacies; 17% with
201393 Residenti survey communities that use physicians; 20% with other health or long-term care
al Care EHRs providers, such as hospitals and nursing homes.
Facilities
AHA = American Hospital Association; e = electronic; EHR = electronic health record; HIE = health information exchange; HIO = health information organization;
ICU = intensive care units; IT= information technology; NAMCS = National Ambulatory Medical Care Survey; RHIO = regional health information organization
U.S. = United States of America; vs. = versus
36
Transfer of Records Between Integrated Delivery Systems
The VA and DoD use the Virtual Lifetime Electronic Record (VLER) system for eHealth
exchange with the private sector, in the Nationwide Health Information Network (NwHIN) – a
‘network of networks’. This is a federated, pull (query-based) model for transfer of records
between integrated delivery systems, using an opt in consent approach by patients. The NwHIN
allows users to pull in data from other organizations (Table 5). In an early study, Bouhaddou et
al. investigated the transfer of records across three integrated delivery systems in San Diego,
California; the VA, DoD, and Kaiser Permanente Southern California. They found that 264 of
363 of patients (73%) who opted in and provided valid authorization could be correlated across
integrated delivery systems.82 In a recent, much larger study, Byrne et al. enrolled 12 sites. Of
the 64,237 veterans who provided authorization and opted in, less than 0.01 percent opted in and
subsequently opted out. The proportion of data matched between exchange partners ranged from
12 percent to 88 percent. The highest matching rates were accomplished using social security
numbers in the matching algorithm. Data were retrieved for 2,724 unique VA patients with the
exchange partner, and for 1,764 unique VA providers reviewing exchange partner data.116
37
Table 5. Level of use and primary uses of HIE: transfer of records between integrated delivery systems
Geographic HIE Risk of Outcome(s)
Study Setting Study Type Results
Location Type Bias Assessed
Bouhaddou, San Diego, Nationwide VLER Cross- Not rated Transfer of records Of 363 patients who opted in and provided
et al., 201182 California Health sectional study due to between integrated valid authorization, 264 could be correlated
Information of patient study delivery systems. across integrated delivery systems, with
Network records design exchange of records between KP and VA, 2-3
(NwHIN; VA, per week.
DoD, Kaiser
Permanente)
Byrne, et al., U.S. VA, DoD, VLER Cross- Not rated Transfer of records -64,237 veterans provided authorization and
2014116 private sectional study due to between integrated opted in.
sector of patient study delivery systems. -31,080 (48%; range 12%-88%).
records design -Highest matching rates with exchange
partners using social security number in their
algorithm.
-5,524 inbound disclosers to VA from
exchange partners (18/100 matched).
-13,913 outbound disclosures to exchange
partner.
-Data retrieved for 2,724 unique VA patients
with exchange partner.
-1,764 unique VA providers reviewing
exchange partner data.
-9% of veterans for whom there was ≥1
disclosure to VA matched with exchange
partner.
DoD = Department of Defense; HIE = health information exchange; KP = Kaiser Permanente; NwHIN = Nationwide Health Information Network; SSN = social security number;
U.S. = United States of America; VA = Veterans Affairs; VLER = Virtual Lifetime Electronic Record
38
Participation in HIE and Extent of Use: Regional or Statewide Initiatives
Nine studies described the use of HIE in the State of New York. Five of these used audit
logs,45,97,98,102,106 two used surveys,76,77 one used a database of clinical data,112 and one geo-
coding125 (Table 6). Most of the HIE implementations are query-based. The studies of audit logs
indicate frequent queries,97,98 and an increasing proportion of physicians accessing HIE over time
(33% to 43% over 18 months).45 Separately, of 63,305 patients enrolled from three hospitals, an
average of 238 clinical event alerts were provided per day to notify ambulatory care providers of
inpatient or ED admissions for their patients; a total of 42,818 events were detected over a 6-
month timeframe.106 Primary HIE users varied by study. In one study, primary users were non-
clinical staff in the outpatient setting and clinicians in the inpatient setting,97 while in another, 86
percent of sessions were with staff in an ED.102
Abramson et al. conducted three statewide surveys in New York, two in 205 hospitals76 and
the other in 632 nursing homes.77 In each, they investigated participation in HIE and the
exchange of data. In hospitals, their results suggest that between 2009 and 2012 the percent of
respondent hospitals participating in HIE and exchanging data, increased from 23 percent to 79
percent. In 2012, institutions exchanged data more frequently with other hospitals (71%) and
ambulatory care providers (69%), than with long-term care facilities (45%) and home health
agencies (38%).96 Among nursing homes 54 percent participate in HIE, with 31 percent of
providers exchanging information outside the system. HIE use was highest when nursing homes
had an EHR. The types of data exchanged were pharmacy (42%), labs (39%), and hospital data
(39%). The seventh study was a retrospective database analysis of clinical data that described a
geriatric care coordination program that used a Clinical Event Notification system to request
information from nursing homes when patients were seen in the ED.112 The authors suggested
that use of the Clinical Event Notification functionality may have facilitated avoidance of 18
percent of hospital admissions, as these admissions lasted less than 48 hours. As not all studies
described the type of HIE in detail, we were unable to draw any conclusions based on the type of
HIE utilized. Finally, using a novel study design, Onyile et al. estimated the proportion of
patients in the New York Clinical Information Exchange (now Healthix) system by mapping the
most current zip code for each patient to the appropriate U.S. county. They found that 88 percent
of patients in the system live within 30 minutes of New York’s Times Square.125
A series of five studies investigated HIE use in a query-based Central Texas HIE. I-Care is
an HIE implementation comprised of hospital systems, public and private clinics, and
governmental agencies operating federally qualified health centers.54,101,103-105 Four of these
studies were conducted across several facility member sites, with a fifth study across two sites.101
For adult patients seen in the ED, use was low; in 57 percent of patients54 and only 2.3 percent of
encounters.105 In a subset of two sites that did not have an EHR (but that mandated use of the
HIE), the HIE was accessed in 21 percent of the encounters.101 Across these studies, HIE use was
higher for those with a greater number of ED visits and hospitalizations,54,101,105 older age, a
greater number of chronic conditions,101,105 females, and those with fragmented care.101 HIE use
was lower for blacks and Hispanics, visits for alcohol use, injury, poisoning, an unfamiliar
patient, and a busier than average day.105 Similar results were found in the study that focused on
children seen in the ED; use was greater for those less than 1 year old, who had more frequent
encounters in the past, and a greater number of diagnoses. Use was lower if the patient was
unfamiliar, or if the day was busier than average.104 In a companion study that investigated how
use of HIE varies by job type and organization in an indigent care setting, Vest et al. found that
39
the most frequent users were those whose positions were administrative, followed by social
services, physicians, nurses, public health professionals, and pharmacy professionals. The
hospital was the workplace for 50 percent of users, followed by adult ED, ambulatory care,
public health agency, mental health agency, and children’s ED. Most clinical access took place
in the ED and in public/mental health agencies. In the majority of use sessions, users accessed
the system in a minimal fashion; almost all use was administrative.103
Of the five studies conducted in the MSeHA, based in Memphis, Tennessee, three used audit-
logs,40,99,118 one was a cross-sectional survey,86 and one used mixed methods.119 MSeHA is an
HIE implementation that facilitates data exchange across EDs and community-based ambulatory
clinics. In 2007, across these studies, HIE use was low, being used for 12.5 percent of the study
population.40 In another, HIE was viewed in the ED for between 3 percent and 10 percent of
visits.99 In a third, HIE was used for only 15 percent of return ED visits and 19 percent of return
clinic visits; yet users reported the HIE provided additional information about histories and
prevented repeat tests or procedures.118 In the separate cross-sectional survey of 151 users, 43
percent reported using HIE less than 1 hour per week, 39 percent between 1 and 4 hours, and 18
percent, greater than 4 hours per week.86 In a separate study of workflow, nurses accessed HIE
when prompted by patients about a recent hospitalization, while providers accessed HIE for
reasons beyond simply identifying a recent hospitalization. HIE access occurred at various points
of care. Workflow patterns evolved over time, due to revisions in access policies and staffing
changes.119 Across these studies, use was higher when the HIE was accessed by nurses and
clerks versus physicians.99,118
Separately, Dixon et al. conducted an online survey of 63 infection preventionists in six
states with HIE, to gauge the awareness and engagement of these preventionists in using HIE for
public health surveillance. One-half of their respondents were unaware of their organization’s
involvement in HIE, and only 10 percent reported their organizations used the HIE.83
Nine additional studies describe HIE use at the State-level, two studies each from Indiana
and Minnesota, and one each from Wisconsin, North Carolina, Massachusetts, Northeastern
Ohio, and Louisiana.84,85,88,90,92,100,120,123,124 These studies used data from 2005123 through 2013.90
Methods of data collection included surveys,84,85,88,90,92,120 interviews,85,124 focus groups,123,124
and audit-logs.100,120 Each study makes a useful contribution to the HIE literature.
In an Indiana study of a coordinated antibiotic-resistance infection tracking, alerting, and
prevention system, of the several thousand patients for whom email alerts were sent,
approximately one-quarter were identified as having had documentation in a different hospital
system of a previous infection with methicillin-resistant staphylococcus aureus or vancomycin-
resistant enterococcus. Capture of this type of laboratory data was found useful.88 Other Indiana
investigators found real-time alerting helpful in prompting followup,92 as did investigators in
Louisiana.124 Patients were generally accepting of data sharing, as long as patient benefit was
evident.124 In a study of small practices (<20 physicians) in Minnesota, results revealed that no
practice was fully involved in a regional HIE and that HIE was not part of most practices’ short-
term strategic plans.85 In a study more recently conducted in Minnesota, intended to monitor
progress toward meeting the legislative requirement that all health care providers have an
interoperable EHR by January 2015, investigators found that over one-half of respondents
exchanged data with affiliated or unaffiliated hospitals.90 The Tripathi et al. study was unique in
that researchers conducted focus groups with patients who lived in three communities that
piloted the Massachusetts HIE. All three communities agreed to share all EHR data except text
notes, consult letters, and scanned reports. Consumer opt in was the preferred consent method, as
40
it is in VLER. Strategies identified to drive consumer opt in included educating patients and
providers about the enhanced convenience and lower costs of HIE.123 Lobach et al. investigated
the impact of the HIE on sentinel events for Medicaid patients in Durham County, North
Carolina. In an analysis of almost 12,000 patients enrolled, they found that 19 percent
experienced a sentinel event over a 6-month period. They concluded that the HIE was useful in
population health management using HIE.100 In a description of HIE implementations in
Wisconsin, Foldy found that 78 percent (21 of 27) of organizations had HIE projects, some
operational, others planned. Most were surveillance systems, delivering data to central registries,
but a growing number served clinicians and patients.84 Kaelber et al. investigated HIE use in the
Northeast Ohio Public health care system, Care Everywhere. Of the 18 percent (74 of 412) of
physicians who responded to the survey, approximately one-third of ED physicians, one-fifth of
primary care physicians, and one-tenth of specialty care physicians used HIE. Use was highest
when patients were older, with more comorbidities, Medicare/Medicaid insured, or black.120
These results reflect the variation in the implementation and impact of HIE, providing data that
are not necessarily generalizable to other settings. These data suggest that small practices are not
adopting HIE, while larger health systems are. They further suggest that HIE may be useful in
exchanging data in the ED, and for surveillance of infectious diseases, that patients and providers
view HIE favorably, and that patients can and do ”buy-in” to the concept of HIE when the
benefits are evident.
41
Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives
Abramson, et New York Hospitals Varies Cross- Low Participation in HIE. -23% of respondent hospitals participate and
al., 201276 State sectional Exchange of data exchange data.
survey -37% participate but do not exchange data.
-40% do not participate
Abramson, et New York Nursing Varies Cross-sectional Low Participation in HIE. -54% participate in HIE.
al., 201477 State homes survey Exchange of data -OR=2.26 more likely to exchange when have
EHR.
-When EHR used, 60% exchange with providers
within system; 31% exchange with providers
outside system.
-HIE highest for pharmacies (42%), labs (39%),
and hospitals (39%).
Abramson, et New York Hospitals Varies Cross-sectional Moderate Use of HIE (sent or 79% (n=102) of respondents reported actively
al., 201496 State survey received). Type of exchanging any electronic patient-level clinical data
institution with an entity outside their institution in 2012 vs.
information is 60% in 2009
shared with. Institutions exchanged data with:
Hospitals outside system: 71% (n=72)
Ambulatory providers outside system: 69% (n=70)
Long term care facilities: 45% (n=46)
Home health agencies: 38% (n=39)
Most commonly exchanged data were radiology

reports, followed by laboratory results, medication
lists and clinical histories.
Kern, et al., Hudson Hospitals MedAllies Cross-sectional Not rated Extent of use. Percent of MDs using portal: 33% months 1-6 vs.
201245 Valley, New and Portal study of audit 42% months 7-12 vs. 43% months 13-18.
York Laboratories logs -Mean days logged-in per month by MD: 8 (SD: 6).
42
Table 6. Level of use and primary uses of HIE: participation in HIE and extent of use, by regional or statewide initiatives (continued)
Campion, et Binghamton, RHIO (2 Southern Cross-sectional Not rated Extent of use. -202,365 auto queries; 54% to hospitals, 46% to
al., 201398 New York hospitals and Tier audit logs clinics.
13 ambulatory HealthLink -145,668 unique patient encounters.
clinics) RHIO; -81, 687 consented patients.
Query -41% of patients had at least one supported
encounter.
Campion, et New York 3 RHIOs Query Cross-sectional Not rated Extent of use. -System access occurred in 60% to 82% of
al., 201397 State (hospital and audit log practice sites registered to use system,
outpatient) depending on community.
-In communities A and B, users were non-clinical
staff in outpatient settings; in community C,
users were inpatient clinicians.
-Proportions of patients whose data were
accessed varied between 5%-60%.
-Most frequently accessed data were patient
summaries, followed by laboratory tests and
imaging data.
Vest, et al., Rochester, RHIO (hospital Query Case-control Low Extent of use. -Each source organization sent average of 971
2013102 New York and outpatient) study of audit- (range: 6 to 8,002) documents to 49 (3 to 106)
and claims log files Patient and other organizations.
from health provider -User organizations accessed average of 49 (1
plans characteristics to 8,444) documents from 6 (1 to 17) source
associated with use organizations.
of an HIE system to -Overall number of radiology reports retrieved in
access radiology outpatient setting was 17 times greater than
report. number of reports retrieved in the ED and
inpatient settings combined (23,201 outpatient
vs. 1,333 ED and 313 inpatient).
-86,152 user sessions with associated claims
files represented the activity of 1,119 different
users representing 145 different workplace
locations; 86% of sessions were with staff; 4%
were with physicians.
43
Moore, et al., New York RHIO; New Not stated Cross-sectional Not rated Extent of use to Over 6 months:
2012106 York Clinical audit log alert ambulatory -42,818 events detected, on average 238 events
Information providers to patient per day.
Exchange events (patients -≥1 event: 6,913 patients.
(NYCLIX; admitted to or -1 event: 1,879 patients.
outpatient). discharged from the -≥10 events: 623 patients
hospital or ED). -Mean number of events in inpatients who had
an event: 7.7 events.
-Mean number of events in all patients: 0.7
events.
Gutteridge, et New York RHIO (ED) Healthix Cross-sectional Moderate Extent of use for -5,722 patients enrolled.
al., 2014112 database clinical event -497 unique notifications sent for 206 patients.
analysis Notification. -219 of 497 (44%) for ED visits.
-121 of 497 (55%) during normal business
hours.
-Hospital admissions resulted from 45% of ED
visits; 18% of these lasted <48 hours,
suggesting they were avoidable
Onyile, et al., New York New York Query Cross-sectional Low Mapped most -12 visits/ 100 patients within 30 miles;
2013125 Clinical analysis of zip current zip code for -0.4 visits/ 100 patients at 100 miles;
Information code data each unique patient -88% of patients live within 30 miles of Times
Exchange to the appropriate Square.
(NYCLIX) U.S. county;
calculated distance
from each zip code
to Times Square.
44
Vest, 200954 Texas Central Texas Query Retrospective Low Association -All levels of HIE information access were
HIE (I-Care). cohort study of between HIE use associated increased expected ED visits and
audit logs and resource use. ambulatory care sensitive hospitalizations, vs.
no information accessed.
Factors that predict -HIE used more for those that used the system
HIE use. more, or were sicker.
-HIE not accessed for 43% of individuals
-Ultimately, these results imply that HIE
information access did not transform care in the
ways many would expect.
After adjusting for confounding factors the
following factors increased the odds of HIE
information access:
OR 1.03 for increasing age.
OR 1.13 for increasing number of chronic
conditions.
OR 1.63 for at least one prior year clinic visit.
OR 1.96 for an ED visit in prior year.
OR 2.02 for being hospitalized in 2004.
Vest, et al., Texas Central Texas: Query Case-control Low Extent of use for System was accessed for 15,586 of 179,445
2011104 I-Care (EDs at study of audit indigent children: encounters (~9%);
11 facilities log files association Basic HIE access:
participating in between OR ~1.5 for over 1 vs. under 1 year old.
HIE) basic/novel HIE use OR ~1.5 for primary care visits in last 12
and resource months.
use/patient OR ~1.5-2 for ED visits in last 12 months.
characteristics. OR ~1.3 for hospitalized.
OR ~1.05 for #diagnoses.
Novel usage=more OR ~0.46 if unfamiliar with patient.
screens. OR ~0.65 if busier than average.
Novel HIE access:
OR ~1.3 for over 1 vs. under 1 year old.
OR ~2 for primary care visits in last 12 months.
OR not significant for ED visits in last 12 months.
OR ~1.15 for hospitalized.
OR ~1.05 for #diagnoses.
OR ~0.19 if unfamiliar with patient.
OR NS if busier than average.
45
Vest, et al., Texas Central Texas: Query Case-control Low Extent of use for -No access of system for 97.7% of encounters.
2011105 I-Care study of audit indigent adults: -Users accessed the I-Care system for 2.3% of
(EDs at 10 log files association the 271,305 encounters.
facilities between basic HIE -Basic HIE usage (42,527) 41% of instances.
participating in use and resource -Sample was predominately Hispanic, younger,
HIE) use/patient and a higher proportion of charity care
characteristics. recipients.
After adjustment:
OR ~0.76 to 0.89 (lower HIE access) for African
American and Hispanics.
HIE access higher for unknown or charity care.
OR 4.7 vs. 2.6 for unknown payer.
OR ~1.25 to 1.5 (higher access) for more ED
visits, hospitalizations.
HIE access lower for alcohol use, injury,
poisoning, unfamiliar patient, busier than
average day.
Vest and Texas Central Texas: Query Case-control Low Extent of use; -297 users, 113 unique job titles, collapsed into
Jasperson, I-Care study of audit- administration (59% of users), social services
2012103 (hospital and log files HIE use by job (~15% of users), physician (~12% of users),
outpatient type, workplace. nurse (~6% of users), public health (~6% of
users), and pharmacy (~1% of users).
Usage patterns. -Workplaces: ambulatory care (~9% of users),
ED (~18% of users), children’s ED (3% of
users), hospital (53% of users), public health
agency (8% of users), or mental health agency
(8% of users).
-In more than 6 out of 10 sessions, users
accessed the system in a minimal fashion.
-Average pattern length was 2.89 screens
(range 1-83 screens); 66% of all user sessions
had a pattern length of only two screens.
-Use was 94% administrative, roughly evenly
distributed across workplaces but for dominance
of hospital accesses (~38%).
-Most clinical access took place in ED and
public/mental health.
46
Vest, et al., Texas Central Texas: Query Case-control Low Extent of use. -HIE accessed for 21% of encounters.
2012101 I-Care study of audit- -7,101 encounter-based, 1,227 retrospective.
(outpatient-2 log files Association In adjusted model, access associated with:
urban safety between HIE use OR 1.12 for female.
net clinics) and patient OR 1.16 for > 40 years.
characteristics OR 1.19 of has chronic diseases.
OR 1.13 if had ED visit in last 3 months.
OR 1.33 if hospitalized in last 4 months.
OR 1.52 if received fragmented care.
Johnson, et Tennessee MidSouth e- Query Multiple site Not rated Extent of use in ED. HIE viewed in 3% of all visits and 10% of visits
al., 200899 Health Alliance case studies of where patient had visit to another site in past 30
(5 EDs) audit-log files Percent of users days.
and qualitative who logged in.
feedback Percent of total users who logged on ranged
from 0 in one site where the high was 12% to
75% by unit clerks in a site that had high use by
other professions.
Bailey, et al., Tennessee MidSouth e- Query Retrospective Low Extent of use. HIE use was low, at 12.5% of study population.
201340 Health Alliance cohort study of Repeat ED visits in
log data which HIE was
accessed vs. repeat
visits in which HIE
was not used for
lumbar or thoracic
imaging.
Gadd, et al., Tennessee MidSouth e- Query Cross-sectional Low Extent of use. -151/162 users (93%)
201186 Health Alliance survey Average usage per week:
<1 hour: =65 (43%)
Between 1 and 4 hours: 58 (39%)
≥4 hours: 27 (18%)
47
Johnson, et Tennessee MidSouth e- Query Multiple site Not rated Extent of use. -Access increased from 4% to 7% of patient
al., 2011118 Health Alliance case studies, encounters over 24 months, ranged from 1% to
(12 EDs and 9 audit logs, Type of data 16 % across sites.
safety net Comment accessed. -15% for return ED visits and 19% for return
clinics) cards, clinic visits.
Feedback in Provider log on -HIE access higher where nurses and clerks
system, rates. involved and lowest where MD only accessed.
Interviews, -Patient opt out rates were 1-3%.
Observations, Participant opt out -Primary user reported consequence of HIE:
ED claims rates. provided additional history (29%); prevented
repeat test or procedure (20%).
Unertl, Tennessee MidSouth e- Query Multiple site Not rated Workflow patterns, Cross organizational patterns; 2 workflow
Johnson, and Health Alliance case studies, by job description. models identified
Lorenzi, (6 EDs and 8 direct 1. Nurse workflow: prompted by patient reporting
2012119 ambulatory observation at recent hospitalization event during intake, HIE
clinics) 14 sites, access by nurse or assistant, printed discharge
informal summary, added to chart
interviews at 2. Physician workflow: HIE accessed by provider
sites, 9 semi (doctor or nurse practitioner) for greater reasons
structured beyond hospitalization; HIE access occurred at
telephone various points of care; HIE review of more
interviews information including history
2009 -Other observations: clerks tracked biopsy
results; workflow patterns evolved over time, due
to factors such as access policies or staffing
changes; residents logged into other EMR due
to lack of HIE access.
-Reasons to access HIE: visit to another
hospital; issues of patient trust; communication
challenges; referrals.
Dixon, Jones, 6 states HIE Varies Cross-sectional Low Extent of use. -10% of infection preventionists reported their
and Grannis, survey Awareness and organizations were formally engaged in HIE.
201383 engagement of -49% were unaware of organizational
infection involvement in HIE.
preventionists in -<5% reporting via secure email, web-based
HIE for public entry, through EHR, or through HIE.
health surveillance. -72% in organizations with EHR
-20% involved in implementation of EHR
48
Kho, et al., Indiana Indiana Not stated Retrospective Low Extent of use. In 3 years:
201388 network for cohort study Coordinated -12,748 email alerts sent on 6,270 unique
Patient Care. with companion antibiotic-resistant patients.
5 hospital survey infection tracking, -23% (MRSA) and 22% (VRE) had previous
systems (17 real-time alerting, history identified at a different hospital system.
hospitals). and prevention -Of 10 infection preventionists surveyed, most
recommended to add automated capture of
laboratory data.
Anand, et al., Indiana Primary care Indiana Cross-sectional Moderate Extent of use. -35% found information helpful vs. 20% not
201292 physician HIE survey Effect of real-time helpful.
offices. alerting from ED, on -24% made followup call to patient vs. 4% sent
physician action attached letter
Fontaine, et Minnesota 9 primary care Not stated Cross-sectional Moderate Extent of use. No practice was fully involved in a regional HIE.
al., 201085 practices with surveys & HIE was not part of most practices’ short-term
fewer than 20 interviews strategic plans.
physicians.
Soderberg Minnesota 1,623 clinics Varies Cross-sectional Moderate Extent of use. -54% exchange data with affiliated hospitals.
and survey To monitor -36% with unaffiliated hospitals.
Laventure, progress toward -Common challenges for HIE: limited capacity of
201390 meeting the others to exchange, lack of technical support or
legislative expertise, competing priorities, cost and privacy
requirement that all concerns.
health care
providers have an
interoperable EHR
by January 2015.
49
Foldy, 200784 Wisconsin HIE Not Cross-sectional Moderate Extent of use. -21 of 27 organizations had HIE.
organizations specified; survey Description of -21 organizations sponsored 16 (76%)
varies projects, stages, operational and 11 (52%) planned HIE
users, organizations projects. Most were surveillance
organizational systems, but a growing proportion served
home, governance, clinicians and patients.
scope. -Most advanced HIE project had 40% of
respondents in implementation and 40% in
operation phases.
-44% delivered data only to central registries,
50% delivered to providers and registries.
-63% based in government organizations.
Lobach, et al., North Carolina RHIO Northern Retrospective Low Extent of use. -Of 11,899 continuously enrolled patients from a
2007100 Piedmont cohort study Frequency and single county over a six-month period, 2,285
Community types of sentinel unique patients (19%) experienced 7,226
Care events. sentinel health events.
Network Frequency of types of events:
(outpatient) -43 hospital admissions for asthma.
-76 hospital admissions for diabetes.
-2,546 low-severity ED visits.
-1,728 ≥2 missed appointments in 60 days.
Tripathi, et Massachusetts Massachusetts Not stated Multiple site Not rated Type of patient Discussion of experience/lessons learned:
al., 2009123 eHealth case studies, consent; 1. Decision on consent: opt in chosen due to
Collaborative consumer focus Types of data to State law stricter than Federal HIPAA law; use of
groups share. centralized data repository; and consumer
feedback.
2. All 3 communities agreed on what to share -
all EHR data except text notes, consult letters
and scanned reports.
3. Consumer focus groups identified themes to
drive HIE/opt in: promote convenience and
costs, promote with providers, State benefits up
front, confront risks, use professional marketing.
4. Consumer opt in across 2 smaller
communities were 88% and 92%.
50
Herwehe, et Louisiana Louisiana Not stated Cross-sectional Not rated Extent of use. In the 2 year period 2/1/2009 to 1/31/2011:
al., 2012124 Public Health focus groups, Counts of real-time -488 registrations of 345 unique patients with
Information interviews, alerts and HIV identified.
Exchange message logs responses. -Clinicians responded to 73% of alerts and
documented actions on note that was shared
Perceptions of with public health.
patients. -Results include statement that 'no negative
feedback has been received from providers' with
no detail.
-Summary of patient interviews found general

acceptance of data sharing as long as there was
patient benefit and a preference for care in the
health care verses the public health system.
-Challenges: concerns about data ownership

and ethics and disparate data systems, but
these are reported as challenges they were able
to address.
Kaelber, et Ohio Northeast Query Cross-sectional High Extent of use. Usage of HIE:
al., 2013120 Ohio Public surveys and -Overall: 1.3%.
Health Care audit logs Characteristics of -ED: 3.6%.
System (10 patients. -Primary care: 2%.
hospitals and Specialty care: 0.5%.
affiliated Perceptions of -Usage highest among patients who were older,
practices using users. with more co-morbid illness, Medicare/Medicaid
Care insured, and black.
Everywhere) -Self-reported impact was more efficient care
(93%), time savings (85%), prevented
admissions (15%), decreased tests ordered
(84%), decreased imaging ordered (74%), and
improved care in other ways (82%)
ED = emergency department; EHR = electronic health record; EMR = electronic medical records; HIE = health information exchange; HIPAA = Health Insurance Portability and
Accountability Act; MD = medical doctor; MRSA = methicillin-resistant Staphylococcus aureus; NS = not significant; NYCLIX = New York Clinical Information Exchange;
OR = odds ratio; RHIO = regional health information organization; U.S. = United States of America; VRE = vancomycin-resistant enterococci; vs. = versus
51
Extent of Use, Types of Information Exchanged, and Adoption in
International or Multinational Settings
Six studies that evaluate the use of HIE in non-U.S. settings met our inclusion criteria, one in
Australia,114 one in South Korea,89 one in Scotland,122 one in England,121 two in Finland61,115
(Table 7). Three multi-country studies,94,95,117 two that included data from the United States,95,117
comprise the last three studies in this group. Lee et al. found that the data most commonly
transmitted differed by setting. From the hospital it was working diagnosis; from the clinic, it
was clinical findings. The most useful data were laboratory or imaging data.89 Silvester and Carr
found that commitment and interest in adoption increased over time.114 Mäenpää et al. also found
a steady increase in uses over time by physicians, nurses and administrative staff. 115 Maass et al.
conducted a unique time-motion study of HIE-facilitated care of 20 diabetic patients, and found
that of 20 visits, four involved use of HIE, with one facilitating a faster treatment decision and
three providing access to the most recent test results.61 Investigating use in the National Health
System in Scotland122 and England,121 Pagliari and Greenhalgh, respectively, both found use to
be relatively low, although Pagliari’s study is now older (2004). Finally, Jha et al. assessed HIE
adoption by physicians and hospitals in six developed countries (United States, United Kingdom,
Canada, Germany, the Netherlands, Australia, and New Zealand), and reported varying results,
but they did find generally low use due to a variety of identified barriers that prevented fuller
adoption. In the United States, fewer than 12 percent of organizations were exchanging data on
less than 1 percent of involved populations.117 In a more recent study conducted in Australia,
Canada, France, Germany, the Netherlands, New Zealand, Norway, Switzerland, the United
Kingdom, and the United States, Schoen found that the percent of primary care physicians
reporting HIE capabilities ranged from a low of 14 percent in Canada to a high of 55 percent in
New Zealand; use in the United States was reported to be 31 percent.95 In a study that included
the 27 European Union countries plus Croatia, Iceland, Norway, and Turkey, Codagnone used a
factor analysis to create a composite metric that ranged between 0 and 4 to measure the extent of
exchange of health information.94 The metric suggested low to moderate use, with an average
score across the 31 countries of 1.88. These early reports suggest that HIE in developed countries
was in the initial stages of use in the early years of the 21st century, and is increasing slowly over
time.
52
Table 7. Level of use and primary uses of HIE: extent of use, types of information exchanged, and adoption in international or
multinational settings
Lee, et al., Seoul, Korea Hospital and 35 Not Before-after High Types of Most commonly transmitted information differed by
201289 clinics specified surveys information setting:
exchanged. -From hospital was working diagnosis: 99% vs. 71%
for clinic, p<0.0001.
-From clinic it was clinical findings: 80%, but this did
not differ from hospital.
-Most useful was laboratory or imaging in both
settings but it was more frequently rated as useful by
hospitals (88% and 7% of cases p<0.0001)
Silvester, et Brisbane, RHIO Not Before-after High Extent of use. -Mean events uploaded for each patient record
al., 2009114 Australia specified database during 12 months: 9.7
analysis of -Increased HIE use by nurses.
clinical -Number of patients registered increased from 474
information (July 2007) to 1,320 (June 2008).
-Increased commitment to use.
-Interest to adopt by others.
Maass, et al., Finland RHIO Not Cross- High Extent of use. Of 20 visits, 4 involved use of information system,
200861 specified sectional with 1 allowing faster treatment decision and 3
survey of HIE- providing access to latest test results.
facilitated care
of 20 diabetic
patients
53
Table 7. Level of use and primary uses of HIE: extent of use, types of information exchanged, and adoption in international or multinational
settings (continued)
Mäenpää, et Finland RHIO Not Retrospective Not rated Extent of use. - HIE utilization rates increased annually in all 10
al., 2012115 specified cohort of audit federations of Municipalities.
logs -Viewing of reference information increased steadily
in each professional group over the 5-year study
period.
-No associations detected between use of HIE and
test ordering outcomes.
Frequency of laboratory test and imaging increased.
The higher the numbers of emergency visits and

appointments, the higher the numbers of emergency
referrals to specialized care, viewed
references, and HIE usage among the groups of
different health care professionals.
Pagliari, et Scotland Primary and Varies Cross- Moderate 6 electronic Access:
al., 2004122 secondary care sectional deliverables: To referral system (47%), results reporting (37%),
survey and 1) outpatient outpatient booking (3%)
database booking;
review 2) referrals; Use:
3) results Results reporting (36%), referral (18%); clinic email
reporting; (9%); outpatient booking (2%)
4) discharge
correspondence Hospital wards able to send e-discharges: 10%;
5) clinic letters; Wards generating and sending e-discharges: 7%;
6) clinic email
Surveys - of responding practices:
Use of Lab results (93%); referrals (58%);
discharges (42%); outpatient booking (16%).
90% reported daily or weekly use.
Clinicians most common users of reporting/ referrals;

Administrative /clerical staff most common users of
discharge/ booking.
54
Greenhalgh, England Primary care Varies Cross- Low Use of the SCR accessed in 4% of all encounters;
et al., 2010121 out-of-hours sectional summary care SCR accessed in 21% of encounters where an SCR
and walk-in database record (SCR) was available;
centers review and When available, clinicians accessed SCR 0% to 84%
ethnographic of time; main determinants of success were clinician
field notes characteristics (not specified);
Jha, et al., U.S., Physicians and Varies Cross- High HIE adoption in Australia: Early pilots, but no major investment. Lack
2008117 U.K., hospitals sectional, developed of unified patient ID an issue.
Canada, mixed countries. Canada: Province-wide efforts, particularly Alberta;
Germany, methods national—early development of ‘Health Infoway’ but
the literature little info exchanged.
Netherlands, review, Germany: Most computers with records not
Australia, surveys and connected; Germans have smart cards, but only
New Zealand interviews administrative data now.
The Netherlands: National ‘SwithPoint’ pilot with 20%
of population, plan full implementation in 2008.
New Zealand: Planning stage, have unified patient
ID, focus of discharge, laboratory and pathology
reports to general practitioners.
U.K.: National Programme, but mostly small amount
of data exchanged in more minor programs.
U.S.: RHIOs, but <12% of organizations exchanging
data and <1% of population involved.
55
Schoen, et Australia, Primary care Varies Cross- Moderate Ability to Percent of primary care physicians reporting HIE
al., 201295 Canada, sectional electronically capabilities:
France, survey exchange Australia: 27%
Germany, patient Canada: 14%
the summaries and France: 39%
Netherlands, test results with Germany: 22%
New doctors outside The Netherlands: 49%
Zealand, their practice New Zealand: 55%
Norway, Norway: 45%
Switzerland, Switzerland: 49%
U.K., and United Kingdom: 38%
U.S. U.S.: 31%
In the U.S. capacity for electronic exchange of

patient information was concentrated in larger
practices and those in integrated health systems
(50% of physicians reported HIE vs. 23% of
physicians not part of integrated practices p<0.05)
Codagnone, 27 countries Varies Varies Cross- Low Factor analysis On a scale between 0 to 4, Denmark score the
et al., 201494 in the sectional to reveal a highest (3.04), while the EU27 plus 4 scored 1.88.
European surveys and composite
Union plus interviews measure of HIE
Croatia, use
Iceland,
Norway and
Turkey
HIE = health information exchange; ID = identification; RHIO = regional health information organization; U.K. = United Kingdom; U.S. = United States of America; vs. = versus
56
Key Question 5. How does the usability of HIE impact effectiveness or
harms for individuals and organizations?
Key Question 6. What facilitators and barriers impact use of HIE?
Key Points
• The 22 studies of usability did not relate usability to effectiveness or harm.
• The evidence was insufficient to compare usability by type of function (query-based or
pull vs. directed or pushed exchange) or by type of architecture (centralized or not).
• The most frequent users rated usability higher than infrequent users.
• Sites with proxy users (e.g., nurses, registrars) in the workflow reported the highest HIE
use.
• The three most commonly cited barriers to HIE use were: lack of critical mass using
exchanges (8 studies); inefficient workflow (10 studies); and poorly designed interface
and update features (7 studies).
• Several facilitators showed promise in promoting electronic health data exchange:
obtaining more complete patient information (6 studies); thoughtful implementation and
workflow (12 studies); and well-designed user interface and data presentation (7 studies).
Detailed Synthesis
We identified nine multiple site case studies,82,99,116,118,119,127-130 11 cross-sectional
studies,58,62,86,94,131-137 and two before-after studies (Table 8).138,139 Because these studies do not
include a comparison with a non-HIE organizational site, risk of bias is not reported but is
described when the details provided sufficient detail. No studies provided results on harm. All
but five of the studies described experience with exchanging health information in the United
States.62,94,133,134,139
57
Table 8. Summary of evidence addressing usability, barriers, and facilitators to use
HIE Description
Author, Year
Type of HIE Evaluation Data Results
Study Design
Patient Consent Process
Bouhaddou, et al., Nationwide Health Informatics Network Patient identifier and Of 363 patients who opted in and provided valid authorization, 264 could be
201182 (NHIN) via CONNECT gateway allows demographic data, correlated; exchange of records between KP and VA 2-3 per week. Older
Multiple site case users to pull in data from other rates of consent patients were more likely to consent for HIE.
studies of patient organizations. The VA and DoD used
records, consent; the VLER systems for eHealth
usage. exchange with private sector. Transfer
of records between integrated delivery
systems; National query-based.
Consent was opt in for the VA and
Kaiser and opt out for DoD.
Byrne, et al., 2014116 HIE between VA, DoD, non-Federal Veterans’ -Used opt in model for patients and 81% of veterans agreed that each
Multiple site case care organizations. The NHIN. The VA authorization patient has a choice
studies. Quantitative and DoD used the VLER systems for preferences, system -Matching of patients varied from 12-88% dependent on whether the
data on Veteran eHealth exchange with private sector. dashboard. 73 exchange partner used social security number
participation and Federated pull (query-based) model provider interviews, 50 -None of the veterans interviewed were aware if their providers were using
provider usage, Transfer of records between integrated veteran interviews and HIE, the user-interfaces at the sites face the provider not the patient
interviews with both. delivery systems; National query- documents from -Providers increased usage after training on VLER system
based. meetings -Providers noted barriers of missing data, additional sign-on and need for
Consent was opt in for the VA and better integration with workflow
Kaiser and opt out for DoD.
Campion, et al., HealtheLink, Rochester New York Online survey 80% used push HIE and 53% used pull HIE. A greater proportion of MDs
201258 RHIO. Direct exchange (push) of local responses from 112 of reported using push HIE always or most of the time (68%) vs. pull HIE
Cross-sectional survey lab and radiology results; query-based 584 invited physicians (19%), (p=0.001). MDs more satisfied with push HIE than pull HIE (p<0.05).
of physician (pull) searching for lab and radiology (19% response rate).
satisfaction with push results across greater Buffalo and Only 99 completed
vs. pull HIE Rochester area survey.
Codagnone, et al., Varies as this was an international Survey of 9196 From focus group sessions, authors reported on usability that HIE remains
201494 survey general practitioners at the “transactional” level and doesn’t yet support information sharing
Cross-sectional survey who used computers across healthcare tiers. There were quite a few general practitioners not yet
and interviews of in 31 European using HIE. Additionally, concern about interoperability, lack of system
general practitioners countries. 2 Focus resilience, lack of data standards and concern about security were barriers
using eHealth that group sessions. to adoption and use.
included HIE.
58
Table 8. Summary of evidence addressing usability, barriers, and facilitators to use (continued)
HIE Description
Author, Year
Study Design
Finnell and Overhage, Indiana Network for Patient Care Online survey Over a six month study period, requests for patient data via HIE increased
2010131 (INPC). Community-wide EMR and responses from 58 of from 15% to 26% per patient contact. The majority of medics surveyed felt
Cross-sectional survey active surveillance of reportable 180 invited medics the HIE information was an important for delivering quality patient care,
of EMS providers and conditions, real-time electronic lab (32% response rate), particularly for patients who can’t communicate their health history. Medics
analysis of use of HIE reporting. Query-based with a Database analysis of who didn’t use HIE cited network difficulties that delayed receiving the INPC
centralized model. Consent was opt out use of INPC per abstract.
for both providers and patients. contact.
Gadd, et al., 201186 MSeHA in Memphis Tennessee. Email survey -3 usability factors were positively predictive of system usage: overall
Cross-sectional survey Consolidated data from multiple responses from 165 of reactions (p<0 0.01), learning (p<0.05), and system functionality (p<0.01)
of HIE use and hospital emergency departments and 237 health care -Users commented that HIE needs more tech support and could use more
usability community-based ambulatory clinics. professionals (70% types of data
Query-based exchange with a response rate).
decentralized system architecture with
secure vaults managed by each
organization. Consent was opt out.
Hincapie, et al., AMIE based on MA-Share created for Focus group meetings Benefits included identification of "doctor shopping", avoiding duplicate
2011132 the NHIN that is a federated query- of 29 physicians on testing, and increased efficiency for gathering information; disadvantage
Cross-sectional, focus based exchange model. Medication HIE quality of care, was limited availability of data.
groups of physicians history, lab test results, and discharge workflow, cost
summaries.
Hypponen , et. al, Varied depending on type of regional Survey included 1693 Users of three local EHR systems preferred electronic HIE to paper to a
2014133 health informational exchange system. physician respondents larger extent than users of other EHR systems. Experiences with an
Cross-sectional survey Type 1: master patient index required aged less than 65 integrated RHIE system (type 3) were more positive than those with other
of Finnish physicians separate login to centralized database. years. 1079 types or RHIE systems. Users of Type 1 reported lengthy log-in process
on HIE success Type 2: web distribution model. Limited specialized care; 614 and information took too long to receive. Recommended that HIE
group of referring physicians could see primary care organizations address interoperability and interface issues, technical and
hospital info. Type 3: regional virtual data standards when designing system. Data format at one institution
model. Clinician used an integrated should be compatible with format of other institutions. Authors also
system that includes all inpatient and commented that those who had access to all information via their own HER
outpatient information. Clinician has may not have realized that they were using HIE.
access to electronic patient record at
other institution. Consent was opt in for
Type 3.
59
HIE Description
Author, Year
Study Design
Johnson, et al., 200899 MSeHA in Memphis, Tennessee. Audit logs, -MSeHA was used for 3% of all visits
Multiple site case Consolidated data from multiple demographics of -The site with the highest usage had registrars looking up HIE data when
studies. hospital emergency departments and users, feedback from patient arrived at the ED
Quantitative analysis community-based ambulatory clinics. users -The site that mostly serves pediatric patients used MSeHA the least vs.
of audit-log files; Query-based exchange with a other sites
qualitative analysis of decentralized system architecture with
feedback of system. secure vaults managed by each
organization. Consent was opt out.
Johnson, et al., MSeHA in Memphis Tennessee. Audit logs, HIE access was higher where nurses and clerks involved and lowest where
2011118 Consolidated data from multiple feedback in system MD only access, patient opt out rates were 1-3%.
Multiple site case hospital emergency departments and (12% of all patient
studies. Quantitative community-based ambulatory clinics. visits with HIE),
analysis of audit data; Query-based exchange with a interviews,
qualitative: semi- decentralized system architecture with observations
structured interviews secure vaults managed by each ED claims
and direct organization. Consent was opt out.
observations.
Kierkegaard, Kaushal 3 RHIO sites with query-based 2 day site visits, onsite -MDs had low tolerance for search failures
and Vest, 2014127 exchanges in New York: 2 federated and telephone -Where clerks were not trained or supported, fewer patients consented
Multi-site case study. models, 1 centralized model. interviews with HIE -MDs often delegated the HIE task
Qualitative, interviews Automated delivery of imaging and lab users and non-users, -Login process perceived as a burden and system was slow.
with users and results to provider EHRs for two observations of
nonusers of HIE. exchanges, automated CCD (one workflow
system). The one system that didn’t
have automated delivery included
secure messaging and event
(admission) notification
60
HIE Description
Author, Year
Study Design
Machan, TILAK, health@net in Tyrol region of Interview with 4 -Overall satisfaction positive for 66.4%, with 83.7% agreeing to receiving all
Ammenwerth, and Austria. Transmission of discharge providers followed by reports electronically, 82.7% reporting less work for filing and archiving, and
Schabetsberger, letters and clinical findings from cross-sectional survey 78.8% agreeing it led to improved quality of care
200662 hospitals to general practitioners. of 104 of 242 (43%) -Barriers were reported, e.g., reports not meeting physician's needs
Cross-sectional. Direct exchange via email that was providers on HIE use -One facilitator is automatic filing of HIE information in patient EHR
Qualitative semi- automatically integrated to physicians’
structured, problem- computer system.
centric interviews
followed by cross-
sectional survey on
usage.
Massy-Westropp, et Exchange in Adelaide, South Australia Satisfaction survey Those who had embraced the use of the integration tools were significantly
al., 2005134 linking a public teaching hospital, ED responses from 55 of more likely to rate Integration higher than those who were not using it as
Cross-sectional and aged home-based care community 132 nurses, clinicians often (p<0.001).
satisfaction survey services organization. When admitted and allied health staff,
and 2 staff focus to the hospital, the patient was added 2 focus group In the discussion they estimated a 20% savings in staff time.
group sessions to a daily inpatient list received by the sessions with staff
home-based providers who could log
into secure website to run live reports
of matched inpatients.
McCullough, et al., 2 states: California, Minnesota. 24 interviews with Identified barriers: Lack of well-functioning area-level exchange,
2014135 California: Collaborate HIE system, a clinicians, challenge achieving a critical mass of users, need strong relationships with
Cross-sectional. Key Query-based exchange from three administrators, and exchange partners, incompatible Health IT used, data ownership and
informant interviews hospitals, 90 providers, and office staff users provider liability concerns about who sees the data, can’t find data on
with stakeholders at laboratories. patients.
practices and health Minnesota: CentraHealth exchange Identified benefits: Improved productivity at initial visit, improved
centers between Federally Qualified health completeness of records, avoidance of duplicative services of patient
Centers and hospitals. This system financial risk Improved nonvisit consults
was in implementation at time of study.
61
HIE Description
Author, Year
Study Design
Messer, et al., 2012138 North Carolina HIV information Interviews and -Qualitative and quantitative approaches provided several “lessons learned”
Before-after study of cooperative regional health information assessment with -It is important to establish clear understanding of privacy and data sharing
organizational organization (CHIC RHIO). 1 large 39 stakeholders; pre among stakeholders
readiness to change, academic med center and 5 AIDS and post survey of 29 -Initial concerns about confidentiality diminished over time as trust was built
needs assessment service organizations. Used providers' satisfaction -Respondents noted it is important to manage expectations upfront
interviews and pre- CAREWare from HRSA. Query-based with HIE, relationships -Clinic staff must use 2 systems the EHR and CAREWare which takes
post quantitative exchange where each participating with other providers, effort and increases errors
survey of HIV provider organization managed its own barriers. -There was an unmet need for training for report generation
users database.
Myers, et al., 2012128 5 exchanges that were part of the Interviews and Web- -Mean composite for ease of use was high (3.9 of 5.0) and no difference by
Multiple site case Information Technology Networks of based survey with role
studies. Quantitative: Care Initiative that included Bronx- case managers, -Mean composite for usefulness was also high (4.0 of 5.0) and no
emailed survey to Lebanon Hospital Center, Duke providers and differences by role
current and intended university; hospitals, the city of nonclinicians on -Qualitative: adoption of the HIE and perceptions of its use and usefulness
users; qualitative: Paterson, Louisiana State University usefulness and ease varied by occupational role of the patient-care team. Also noticed that case
interviews with current Health Care Services Division, New of use. workers outside the clinic used the HIE routinely. Those within clinics used
HIE users during site York Presbyterian Hospital, St. Mary 62 of 102 responded HIE sporadically.
visits Medical Center Foundation. (62%)
Query-based.
Nohr, et al., 2001139 Four types were described: (1) Survey respondents: Several organizations have since started workflow analysis to identify
Before-after Danish common database; (2) Electronic Data Expected benefits in former hidden procedures and for determining user requirements. One of
study that included Interchange via structured messages: 1998 (n=102); the barriers was that most professionals used to the free-text nature of
survey and interviews copies of data are transferred between Experiences in paper records and were now forced into structured format. One of the
on HIE expectation vs systems; (3) middleware: software benefits in 1999 barriers was lack of knowledge about integration principles which left the
experience. between application and database; (4) (n=57); Expected vendors to provide solutions.
internet technology: data barriers in 1998
communicated via browser. (n=101); Experiences One of the facilitators of success was a bottom-up approach with users
in barriers in 99 involved during implementation. It is also helpful if the training go beyond
(n=99). Group basic use and provide information on becoming experts in using HIE.
interviews per site. Finally, the organizations were unprepared technically to have a system
running 24/7. They suggested having back up plans, e.g., mirrored
databases.
62
HIE Description
Author, Year
Study Design
Ozkaynak and 3 ED sites accessing the EDLinking 210 hours direct -The ED providers only used the HIE for 5% of visits
Brennan, 2013129 system in Madison, Wisconsin. observations, varied -It was used primarily for patients in chronic pain to detect drug-seeking
Multiple site case Clinicians can choose to use (or not across shifts, in 5 behavior. This information was then used as support to confirm or confront
studies. Direct use) the exchange. rounds, informal patients who may be abusing the system.
observation, informal conversations to
interviews during followup on
observation, formal observations, plus 13
semi-structured open ended HIE
interviews with HIE interviews.
users.
Rudin, et al., 2011136 Massachusetts eHealth Collaborative. Interviews of 15 -Motivators were belief in improved quality of care, time savings, and
Cross-sectional. All nontext portions of medical record. clinician users, 2 HIE reduced need to answer questions.
Twenty interviews with Could link directly from the EHR to staff, and 3 -Motivation was moderated by missing data, workflow issues, and usability
clinician users, HIE existing exchange. Query-based administrators issues (too many clicks required to get to information).
staff and exchange. Consent was opt in. -Missing data was attributed contributing providers not "locking their notes"
administrators on their EHR.
Thorn, Carter, and HIE name not explicitly stated but may Individual Barrier themes
Bailey, 2014130 be MidSouth eHealth Alliance unstructured 1. Trouble accessing system, acuity of patient or history not available, team
Multiple site case (MSeHA). Query-based exchange. interviews with 15 ED members' inability to access.
studies. Interviews Consent was opt out. physicians 2. HIE use affected decisions only sometimes, for specific cases (e.g. drug
with ED physicians seekers).
using HIE 3. Access challenges, separate login, variability in data being pertinent,
absence of data types or data on specific patients, user design flaws, and
lack of technical support.
4. Barriers to usage also included continued practice of defensive medicine,
desire for autonomy, changing the culture, belief that HIE does not alter
decisions, health system competition, and reduced revenue, workflow
disruption.
63
HIE Description
Author, Year
Study Design
Unertl, Johnson, and MSeHA in Memphis, Tennessee Observation (180 -HIE workflow was modeled for each ED site and clinic
Lorenzi, 2012119 Consolidated data from multiple hours) in 6 ED and 8 -2 models emerged: physician-based and nurse-based
Multiple site case hospital emergency departments and ambulatory clinics,
studies. Ethnographic community-based ambulatory clinics. informal interviews
study, direct Decentralized, query-based exchange. during observation
observation, informal Consent was opt out. and 9 formal semi-
interviews during structured interviews
observation, formal with physicians,
semi-structured nurses and IT
interviews with HIE management
users. Moderate risk
of bias
Yeager, et al., 2014137 LaHIE Interviews with 16 Five themes were identified related to usability.
Cross-sectional. Hybrid, centralized and federated HIE healthcare 1. Physicians found separate HIE logins required recalling separate
Qualitative analysis of in Louisiana that includes DIRECT representatives from passwords and delayed receiving information. Suggested having staff
16 interviews with messaging between providers. organizations access HIE prior to visit and bring into patient chart.
healthcare Providers can share CCDs, lab results, interested in joining 2. Training is needed to get a critical mass of providers to contribute.
stakeholders and electrocardiogram results. LaHIE but not yet 3. Quality of data in HIE is limited if some only provide discrete data.
enrolled (n=4), not 4. Physicians expressed concern about liability if the HIE data isn’t
interested in joining integrated into the patient chart.
(n=4), or already
enrolled (n=8)
AMIE = Arizona Medical Information Exchange; CCD = Continuity of Care Documents; CHIC = Carolina HIV Information Cooperative; DoD = The Department of Defense;
e = electronic; ED = emergency department; e.g. = for example; EHR = electronic health record; HIE = health information exchange; HRSA = Health Resources and Services
Administration; IT = information technology;; KP = Kaiser Permanente; LaHIE= Louisiana health information exchange; MD = medical doctor; MSeHA = Mid-South eHealth
Alliance; NHIN = The Nationwide Health Information Network; RHIE= regional health information exchange; RHIO = regional health information organization; TILAK = Tiroler
Landeskrankenanstalten; VA = The Department of Veterans Affairs; VLER = Virtual Lifetime Electronic Record
64
HIE Usability
Usability was defined in the 1998 International Standards Organization 9241-11 standard as
“the extent to which a product can be used by specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a specified context of use.” We found five surveys on
HIE usability and most defined usability as it relates to function and/or measured satisfaction
with exchanging health information.58,62,86,133,138 One multiple site case study reported usability
as composite measures of: ease of use and usefulness, described below for current and intended
users of five HIE systems.128 The composite score for perceived ease of use (which included
level of agreement for 10 statements on use) averaged 3.9 on a 5.0 scale where 5.0 was “strongly
agree.”128 For example users were asked to provide level of agreement for, “Learning to operate
‘the HIE’ was easy for me.” Similarly, the same respondents averaged 4.0 of 5.0 on the
perceived usefulness composite score, which was also based on responses to 10 statements. The
survey sample included 24 case managers, 21 medical providers, and 17 nonclinician staff
members and perceptions about usability did not vary by role. This emailed survey achieved a 62
percent (62 of 102) response rate and the inter-scale agreement reliability; Cronbach Alpha
ranged from 0.57 to 0.93.
Usability features were also examined in relation to actual use in one cross-sectional study of
health care professionals electronically exchanging health data through the MSeHA.86 Health
professionals were emailed the survey and responded to questions about actual use and usability
features that included questions from the Questionnaire for User Interface Satisfaction (QUIS)
7.0 instrument in six areas: overall reactions, screen, terminology and system information, effort
required to learn the system, system capabilities, and system functionality. Multivariate analyses
revealed that average weekly use of the MSeHA was associated with higher scale scores in:
overall reactions (OR 1.50, p<0.01), learning (OR 1.32, p<0.05), and system functionality (OR
1.34, p<0.01). The reported psychometrics for the survey questionnaire (inter-scale agreement
reliability on the QUIS scales: Cronbach’s Alpha ranging from 0.74 to 0.91) and response rate
(165 of 237, 70%) were good, reducing concern about bias and increasing ability to generalize.
HIE Satisfaction
Satisfaction with HIE, a measure of usability, was examined in one cross-sectional study134
and one before-after study 138 One additional cross-sectional study that stratified satisfaction by
types of HIE is described later.58 Using a pre-post survey study design (n=29), physicians at one
clinic and five AIDS service organizations in North Carolina reported increased satisfaction after
the Carolina HIV Information Cooperative (CHIC) RHIO was implemented.138 Participants
reported improved satisfaction with ease of data exchanged and improved patient care after using
CAREWare software. The respondents also perceived that CAREWare was a good use of
resources. They also reported improved relations with HIV care partners after implementing the
RHIO. By contrast, before implementation, the providers had high expectations for how
exchanging information would affect their work and reported some unmet expectations
afterward.
In a second study on satisfaction of HIE users in Adelaide, South Australia,134 users who
embraced the use of the data exchange integration tools were significantly more likely to rate
integration higher than those who were not using it as often (p<0.001). This result echoes a more
recent study that found frequent users are more pleased with the usability of an HIE system than
infrequent users.86 The response rate for the Massy-Westropp study was 24 percent (55 of 132).
65
While both satisfaction studies134,138 provide descriptive evidence from surveys that users were
satisfied with usability, neither provided sufficient details in the methods sections to eliminate
bias or a comparison that would enable generalization.
Usability of HIE by Type

We also examined whether certain functionality (direct exchange or push vs. query-based
exchange) was more usable. Directed exchange is provider-to-provider electronic exchange of
patient information to coordinate care.32 In this type of exchange, the data are electronically sent
to the recipient’s EHR or clinical inbox.58 In query-based systems, the user accesses an exchange
system, queries for information (e.g., ED, hospital admissions, or discharges) on a particular
patient and pulls data from multiple health care organizations.58 This is important particularly for
unplanned care (e.g., patient comes into the ED).13 We also attempted to evaluate usability by
type of architecture (e.g., whether the query-based system used a centralized or federated model).
However, few publications provided this level of technical detail to make a comparison.
Additionally, the authors used a variety of terms and descriptions which made it difficult to
classify usability by architecture. When the authors provided detail on architecture, it was
included in Table 8.
Only one cross-sectional study evaluated clinician satisfaction with exchanging health
information using query-based (pull) or direct exchange (push).58 In this comparison study,
clinicians had access to “pushed” health data (laboratory and radiology) through certified EHRs;
physicians who ordered tests could designate other physicians to receive the test results. The
physicians in this study could also query (pull), using a secure web portal, for test results, patient
demographics and transcribed reports provided by physicians, hospitals, laboratories and
radiology centers across the greater Buffalo and Rochester areas of New York. More providers
reported using electronically pushed data exchange (80%) than pulled exchange of health
information (53%). A greater proportion of physicians reported using pushed data exchange
always or most of the time (68%) compared with pulled exchange (19%, p=0.001). The
physicians were more satisfied when data were pushed than pulled (p<0.05).
In summary, we found insufficient data to compare usability by type or architecture of the
electronic data exchange.
Facilitators and Barriers Impacting HIE Use

We identified many barriers and facilitators to electronic health data exchange in the
literature. Evaluations of the MSeHA provide the most complete evidence on barriers and
facilitators of use86,99,118,119,130 but other studies echoed similar barriers.62,82,94,116,127-129,131-
133,135,136,139
Barriers and facilitators were assessed with qualitative approaches in these studies
which were difficult to assess for risk of bias and generalizability. In this section, the barriers
mentioned most often are presented in partnership with affiliated facilitators (Table 9).
66
Table 9. Barriers and facilitators of actual HIE use grouped by theme
Barriers Facilitators
Lack of Critical Mass Bouhaddau, et al., More Complete Patient Bouhaddou, et al.,
• Patients concerned about 201182 Information 201182
privacy and security Byrne, et al., • Consider opt in vs. opt out Byrne, et al., 2014116
• Poor matching or patients 2014116 • Obtain consent at Campion, et al.,
• Providers stop using query- Hincapie, et al., registration 201258
based system when can’t find 2011132 • Educate patients on HIE Kierkagaard,
patients Kierkegaard, • Make HIE visible to patients Kaushal, and Vest,
• Incomplete patient Kaushal, and Vest, (turn screen so they can 2014127
information 2014127 see it during visit). Messer, et al.,
• Patients outside of the HIE McCullough, et al., • Consider when to push and 2012138
catchment area 2014135 when to pull data Johnson, et al.,
Ozkaynak and 2011118
Brennan 2012129
Rudin, et al.,
2011136
Thorn, Carter, and
Bailey, 2014130
Inefficient Workflow Byrne, et al., Thoughtful implementation Byrne, et al., 2014116

• Separate login to portal – too 2014116 and workflow Gadd, et al., 201486
many clicks. Hypönnen, et al,. • Identify former hidden Hincapie, et al.,
• Unmet expectations 2013133 workflow 2011132
• Policy that prohibits proxy Johnson, et al., • Provide training for Kierkegaard,
users 2011118 providers and proxy users Kaushal, and Vest,
• Need for more technical Kierkegaard, • Manage expectations of 2014127
support Kaushal, and Vest, new HIE Johnson, et al.,
• Need for culture change 2014127 • Develop workflow for 200899
about practice Machan, providers and proxy users. Johnson, et al.,
Ammenwerth, and • Have providers and proxy- 2011118
Schabetsberger, users involved in design of Messer, et al.,
200662 interface 2012138
Messer, et al., • Implement a case Rudin, et al., 2011136
2012138 management approach for Thorn, Carter, and
Myers, et al., HIE use Bailey, 2014130
2012128 • Have champion HIE users Ozkaynak and
Nohr, et al., Brennan 2012129
• Have sufficient technical
2001139 Nohr, et al., 2001139
support
Rudin, et al., Unertl, Johnson, and
2011136 Lorenzi, 2012119
Thorn, Carter, and
Bailey, 2014130
67
Table 9. Barriers and facilitators of actual HIE use grouped by theme (continued)
Poorly-designed Interface and Hypönnen, et al., Well-designed Interface and Bryne, et al., 2014116
Update Features 2013133 Data Presentation Hypönnen, et al,.
• Too much information and Codagnene and • Monitor quality of data 2013133
slow response Lupiañez- against standards Campion, et al.,
• Duplicate Information Villanueva, 201494 • Provide clear notifications of 201258
• Reports in exchange Kierkegaard, HIE Kierkegaard,
workflow may not meet Kaushal, and Vest • Send brief report first Kaushal, and Vest,
needs of the provider 2014127 • Automatic integration with 2014127
• Competing use with existing Myers, et al., existing provider systems Machan,
patient portal with complete 2012128 • Include providers and proxy Ammenwerth, and
information Thorn, Carter, and users in design of interface Schabetsberger,
• Lack of notes to set context Bailey, 2014130 200662
in patient information Machan, Thorn, Carter, and
• HIE not updated in real time Ammenwerth, and Bailey, 2014130
Schabetsberger, Myers, et al., 2012128
200662
Rudin, et al.,
2011136
HIE = Health information exchange; vs. = versus
Addressing Lack of Critical Mass

Concern was expressed in several studies about the need for a critical mass of users and
populated patient information.82,116,127,129,130,132,135,136 Underlying reasons for lack of critical mass
can include several reasons (e.g., the providers aren’t electronically exchanging the data or
patients have not consented). Patients concerned with privacy and security may not understand
the benefits and/or may not consent to have their data shared with other providers. Even when
they do consent, they may not be properly matched to existing data.132 Also, match rate can vary
by population and setting; for example, the match rate for providers practicing in a homeless
center was lower, but the match rate for ED physicians was higher.132 Some contributing
providers reported legal concerns for sharing patient data and may choose to not participate. The
end result was that providers searching for patient information may grow frustrated at taking the
time to search and stop using the system.
To increase the critical mass, several approaches have been suggested. These include
addressing concern about privacy, careful consideration about the consent process, and a process
for educating patients.58,82,116,118,127,134,138 To address patient and provider concern about privacy,
create clear understanding about privacy and data sharing among all stakeholders (providers,
patients, nonclinician partners) prior to implementation.138 In planning for electronic health
exchange, several authors noted the importance of deciding whether to have opt out or opt in
consent process for patients.58,82,99,116,118,119,127,129,136 Of veterans interviewed, 90 percent were
positive about the VLER HIE system. At the same time, 81 percent felt each person should have
a choice to opt in and the default should not be automatic participation.116 Opting in protocols
seem to yield a high patient participation rate (93% to 97%).58,127,136 When age is considered,
older patients opt in more often than younger patients.82 The percentage of consented patients
can be increased with a workflow that includes front staff members being trained to educate and
consent patients as they first arrive.127 Additionally, patient awareness of provider use of the HIE
may increase patients perception of the benefits of electronically exchanged data. Patients in the
VA reported being unaware that providers were using the VLER system to access information
68
outside of the VA.116 The authors noted that the user interfaces of the VLER are not visible by
patients because the display faces the providers. We identified one organization that used an opt
out protocol (MSeHA).118,119 Patients had the option to opt out at every encounter. The opt out
rate was 1 to 3 percent,118 which is slightly better than programs with an opt in protocol that lose
3 to 7 percent of patients who do not consent.58,136
Addressing Inefficient Workflow in Electronically Exchanging Data

Often the workflow was inefficient to providers attempting to exchange health
information.62,116,118,127,128,130,136,138 Users complained that additional logins and policies against
proxy users increased the time the provider needed to access the patient information.
Sites with proxy users (registrars, nurses, clerks, and other physicians) who accessed the
system and then provided the information to the attending physician had the highest access
rates.99,118 Proxy use was described as a way to save provider time or address needs of limited
users without privileges.130,132,136 Additionally, some organizations made it difficult to get
privileges to access exchanged data so those with privileges were called upon to look up
information for those without.130
An ethnographic qualitative study of the MSeHA identified two role-based workflow
models: physician-based and nurse-based.119 These investigators completed 180 observation
hours of six EDs and eight ambulatory clinics using the MSeHA exchange system, informational
interviews during observation, and nine semi-structured interviews. In the nurse-based model, if
a patient mentioned a recent hospital visit, the triage nurse or medical assistant would search for
data primarily looking for summary documents related to recent hospital visits, such as a
discharge summary, but rarely searched for other medical history. The nurse then printed off the
information for use by the provider. In the provider-based model, physicians and nurse
practitioners searched for electronically exchanged information for more reasons than hospital
visits. These providers browsed online medical history for purposes of decisionmaking. Finally,
another study of the MSeHA reported that use dropped significantly after a new policy
prohibited registrars from searching the system at the start of a visit.118 Initially registrars would
print off a summary sheet of available data. Providers then queried the system, based on the
summary sheet. When a new policy came in place prohibiting registrars and nursing team
members from accessing the system for security reasons, use dropped significantly.
During implementation several other strategies were mentioned related to changing current
workflow: providing training and enough technical support to support the new workflow,86,116
addressing needed culture change,130 and having champion users.99,127 One physician expressed
in an interview that exchanging data is a change from practice. Physicians “get bogged down
[with exchanging health information] and just want to see patients”.130 Introducing new
technology requires addressing the need for change and the resistance that may exist. These
studies also encouraged sites to manage expectations upfront138 and have a pilot implementation
prior to launch so users aren’t disappointed.118,132
Addressing Poorly Designed Interface and Update Features

Several design features of the HIE created barriers to use.62,116,127,128,136 While HIE users
understood why textual notes were not exchanged for confidentiality reasons, this lack of context
made the information less valuable.136 While some users wanted more information, other users
wanted shorter reports to avoid having to scroll up and down, click on many pages or go to
another task. Some complained that the exchange contained too much information that was not
69
filtered enough to be meaningful for providers.127,128 They reported that reading a paper report
was much faster than reviewing the exchanged information.128 This finding was echoed by
another study that recommended the main findings should be sent first in a brief report.62 The
design features could be addressed better at the implementation phase by including more
providers during the design phase.127 Another facilitator is to continually monitor the quality and
usability of the exchanged data to meet standards and the needs of the users.116 Similarly, as
more patient data and more types of data were exchanged, users reported that their system
response slowed suggesting the need to continually review (and reduce) what was being
exchanged.116
Some users expressed concern with how quickly the patient information was updated and
found it more efficient to go directly to the partnering clinic or hospital for information than to
rely on current information in the exchange.128 Systems that automatically integrate with the
providers’ EHRs may reduce this concern and also reduce need for users to have to login into
multiple systems.62,130
Key Question 7. What facilitators and barriers impact implementation of

HIE?
Key Question 8. What factors influence sustainability of HIE?
Key Points
• There was a sizable body of research that attempts to identify and categorize the
facilitators and barriers to implementation and factors that affect the sustainability of HIE
(52 studies).
• This literature identified several categories of characteristics of HIE activities and
organizations (internal factors) that affect implementation
o The most commonly identified facilitators were general organizational
characteristics such as leadership while the most frequently cited barriers were
disincentives such as lack of financial viability.
• The research cited policy and external environment influences as affecting
implementation less frequently than internal factors.
o Laws and mandates that require or support organizations engaging in HIE were
the most frequently reported external facilitator for implementation.
• The most frequently cited negative influence on sustainability was competition that
limited the necessary collaboration among organizations required to support HIE.
• Two key positive influences on sustainability were desire for the expected outcomes from
HIE and the selection of HIE functions most likely to have financial benefits.
Detailed Synthesis
Both implementation and sustainability are organizational level measures of approaches to
change. While the experiences, attitudes, and priorities of individuals may be important,
ultimately the decisions to adopt and continue to support HIE activities are made by
organizations not individuals. For this reason this section focuses on organizational level
characteristics and factors that affect organizations’ decisions and actions.
70
Implementation involves identifying new practices or technologies; making the decision to
incorporate them into workflow and processes; and taking the actions necessary to prepare for
and then initiate adoption of change. Sustainability is essentially the ongoing maintenance of
what was implemented, but also includes the idea that the practice or technology that was
implemented must evolve to continue to meet the changing needs of the organization.
Approaches to understanding implementation and sustainability are rooted in consideration of
the fit between an organization and the practice or technology as well as the external and internal
factors that either facilitate or act as barriers to the change. In the case of HIE, health care
organizations must consider first whether, and then how, to participate in HIE (implementation).
Once HIE is established the focus shifts to how to maintain, improve, and grow the systems
(sustainability).
We identified 52 studies that addressed implementation and/or sustainability (Appendix F).
Fifty of the included studies were published in the past 8 years (2006 to 2014). Eight studies
assessed HIE activities in countries other than the United States, 10 were based on U.S. national
surveys or data, 10 covered multiple sites in the United States, but the most common were 24
studies that covered single State or regional HIE organizations and their efforts. Six of the
studies were about HIE in New York, with five about statewide efforts or several RHIOs and one
about New York City. Three were about HIE in California, but each study was about a difference
regional HIE organization. No other State or metropolitan region was the subject of more than
two studies.
Most of the studies were cross-sectional designs that collected data via surveys and
interviews and relied on qualitative data analysis. More specifically 26 of the studies were cross-
sectional,79,84,85,87,94,100,108,124,140-157 17 were multiple site case studies that compared experiences
across different organizations or sites,82,116,122,123,158-170 two compared outcomes before and after
HIE,114,138 three were retrospective cohorts,43,44,48 two were prospective cohort studies,171,172 and
two were time series.173,174 Almost half (23 of 52) of the studies used data from multiple sources,
while the most common sole sources were interviews (10 studies) and surveys (9 studies). Other
sources of data included databases (4 studies), audit logs (3 studies), and one each that used
documents, organizational assessments, and a literature analysis.
Given the focus of Key Questions 7 and 8 and the sources of data it is not surprising that
most of the analyses where qualitative (25 studies), including narrative summaries and the
identification of themes. Twelve studies used quantitative analyses such as descriptive statistics,
while seven employed more complex multivariate analyses. Eight studies combined qualitative
and quantitative analyses (mixed methods).
Variety in study design, data sources, and analytic methods make assessing the quality across
the 52 studies that address these Key Questions problematic. Quality assessment is frequently
tied to risk of bias and the criteria are related to how the groups are constructed in cohort studies
and how quantitative analyses are used to make these comparisons. While there are criteria for
quality in other types of studies, these are used less frequently and there is not yet widespread
agreement on the criteria, what is necessary to meet them, or what constitutes the difference
between levels of quality. We can say that most of the studies in this section either attempted to
include all sites or participants or included large samples of the population, increasing the
likelihood that they are representative of the target populations. Also as we excluded purely
descriptive studies, the qualitative analyses tended to follow established procedures (e.g.,
involvement of multiple researchers in coding) although in several cases the description of
methods was limited.
71
One or more facilitator or barrier to implementation was identified in 42 studies while 17
studies reported factors related to sustainability. Some studies addressed by implementation and
sustainability. We grouped the facilitators into eight categories and the barriers into seven
categories created based on our interpretation of their similarities. These are described in the text
below. In Tables 10 and 11 the specific factors included in each category are listed below the
category in the first column and the studies that report this factor related to implementation or
sustainability are cited in the second and third column respectively.
Implementation
Facilitators
Seven of the eight categories of facilitators for implementation identified in the literature
(below) are predominately “internal” factors, concerned with the characteristics of the HIE or its
components, while only one category, external policy, addresses the environment for the HIE.
General Structural Characteristics

These include leadership,43,144,164,174 prior experience with or readiness for IT projects,138,158
preexisting membership in a network,155 or trust and solidarity among practices participating in
HIE. One evaluation of HIE efforts concluded that, “having IT initiatives underway prior to
receiving… funding contributed substantially to the states’ readiness and subsequent
implementation progress.”158
HIE Specific Structures

This category includes findings from seven studies and specific factors were goverance,43
and participatory approaches that included efforts to encourage user engagement and stakeholder
buy-in.48,122,124,150,159 Examples include findings that involving users in development was key to
implementation150 and that a participatory process and shared decisionmaking permitted the HIE
to address different values held by participants related to balancing individual rights and public
health.124
Orientation Shift in HIE Organizations

This is a category that could also be called mission or change in ideology. Two studies found
that implementation depended on a shift from competition to collaboration,154 or from ownership
of data to continuity of care that included realizing the value of external information.170 Another
important shift is from treating HIE activities as a pilot test to integrating them into a robust
system integrated in workflow.163 This research highlighted experiences that staying in the pilot
phase for too long was detrimental to full implementation and increased use.
Design Characteristics
Cited as a facilitator for implementation in six studies. Studies found that a design that
reflects an understanding of work flow,150 and designs with smaller scale or more limited scope
were more likely to be implemented.169,173 The architecture and adaptability of information
systems were cited as important design characteristics by two studies161,169 with one researcher
explaining, “Our findings suggest that communities embarking on HIE initiatives would do well
to examine how particular HIE technical architectures map to their objectives, local context,
existing relationships, sustainability plans, and vision of both present and possible future
72
needs.”161 An additional study found that successful HIE organizations used some existing
standards rather than waiting for more universal standards that are under development.159
Key Functions
This is a category of functions that may seem obvious but that are essential. Four studies
reported that HIE systems needed to be set up so that use became part of care routines, so that the
burden and time required of staff was minimized and so that useful data was provided.85,114,116,169
One study concluded: “Implementation outcomes…were shaped substantially by the degree of
attention dedicated to reworking procedures and practices so that HIE usage becomes routine.”169
Another study highlighted that addressing issues related to providing better quality data and
integration into workflow allowed successful system-wide deployment.116 However, the capacity
for advanced use (HIE that provides new tools or information) may be an important facilitator as
HIE evolves. One study cited the example of HIE providing the foundation for development of a
system that alerted providers to important patient events leading to both improvements in quality
of care and contributing to organization goals such as medical home certification.143
Implementation Support
The need for an organization to provide resources to support the implementation of HIE was
cited in the results of four studies. Specific types of support cited included technical assistance
and training infrastructure,114,167 the ability to do extensive testing for data quality,154 and a
comprehensive strategy for HIE activities and their implementation.168
Expected Outcomes
Two studies reported that specific expected outcomes were key to implementation. These
included public awareness of the HIE148 and link to a community need.146 A third study
highlighted the importance of establishing tangible intermediate goals in order to keep
participants engaged and foster ongoing support.159
External Policy
Federal and State laws and mandates,85,140,159 as well as grants,158 were identified as
facilitators in five studies when they promoted, required, or funded HIE director or foundational
components such as EHRs. One study of 31 countries in Europe documented that HIE activities
were more widespread in countries with national healthcare systems verses countries with social
insurance systems.94
Table 10. Facilitators to implementation and sustainability of HIE

Number of Studies Reporting Number of Studies
Facilitator an Implementation Reporting Sustainability
Facilitator Positive Influences
General structure/organization* 8 1
Leadership 443,144,164,174
Prior IT initiatives or IT readiness 2138,158
Network membership 1155
Trust and solidarity 1167
Able to innovate and react quickly 1163
73
Table 10. Facilitators to implementation and sustainability of HIE (continued)
HIE-specific structure* 7 3
Participatory approach/user 548,122,124,150,159
engagement/stakeholder buy-in
Governance 143
HIE lead by Health Information Organization 1171
Community needs assessment 1171
Marketing to patients 1123
Control over technology 1163
Orientation shift* 4
From competitive to collaboration 1154
From ownership of data to continuity of care 1170
To valuing contribution of external information 1170
From pilot to robust system quickly 1163
Design characteristics* 6 3
Information system architecture/adaptability 2161,169
Smaller scale/limited scope 2169,173
Reflect understand of services and work flow 1150
Use of some existing standards while waiting for 1159
single standards in long term future
Select function likely to have financial benefit 2147,160
Key functions* 5 1
Make use routine/minimize burden and 485,114,116,169
time/provide useful data
Advance use (decision support; medical home 1143 1147
functions)
Implementation support* 4
Comprehensive strategy 1168
Extensive testing for data quality assurance 1154
Technical assistance/training/change 2114,167
management
Expected outcomes* 3 3
Public awareness 1148
Link to community need (public health use) 1146
Tangible intermediate goals 1159
Savings exceed costs 144
Quality of care 1153
74
Table 10. Facilitators to implementation and sustainability of HIE (continued)
External policy* 5 1
Laws and mandates 385,140,159 1140
Federal and State grants 1158
Type of Healthcare System (National, Social 194
Insurance, transition)
HIE = health information exchange; IT= information technology
*Bold indicates overall category of facilitator.
Barriers
Barriers to HIE implementation cited in the research are not simply the inverse of the
facilitators. While there is some overlap in the categories, the barriers cited include more
external, environmental factors. The seven categories of barriers are included in Table 11.
External Policy
This is the one category of barriers that corresponds most directly to a category of
facilitators. While Federal and State laws and funding and grants were seen as facilitators for
HIE implementation, changes in Federal policy,164 the fragmented nature of funding (e.g., in
public health HIE may be funded for some activities and not others),157 and the uncertainty and
the timelines for funding were seen as barriers.143,174 One study identified the disconnect between
State or Federal government goals and local realities as a significant barrier to HIE
development.166
Disincentives
This is a broad category and the largest, including 20 studies. Four studies reported that
competition for patients and the difficulty making the business case for HIE are important
barriers,108,142,151,155 and five additional studies more specifically cited the costs of HIE and the
lack of financial viability.85,108,141,158,167 In states with mature HIE implementations, where
presumably the infrastructure was in place, participants cited costs and a lack of understanding of
the value proposition as the major barrier to participation.141 Three studies identified the fact that
the organizations that invest in HIE are not always the ones that benefit (e.g., hospitals invest in
HIE but do not necessarily realize the savings when duplicate tests or admissions are
avoided).155,158,160 One study cited a trend to set up HIE that supported more administrative tasks
over clinical tasks as a barrier.94 Two additional studies cited insufficient resources.84,87 In
addition to financial and resource concerns, five studies identified concerns about data misuse,
ability to protect privacy, and ethical issues related to sharing data.124,142,148,160,165
Structural Characteristics
This is a category of barriers that includes some parallels in the facilitators—leadership can
promote HIE, but lack of leadership or effective communication from management can be
important barriers according to two studies.85,174 While being in a network might facilitate HIE,
one study concluded that hospitals that are part of larger systems are less likely to participate in
HIE, perhaps because patients stay in the system and there is less need for external data.149
Another identified barrier is the mismatch between the geographic coverage of the HIE and the
75
service areas for patients, as would be the case for a hospital with a service area that crosses
State lines and a State-based HIE.148 Diversity and complexity within and across HIE systems
were also cited as barriers. One study concluded that the extent of differences made sharing and
applying lessons learned from one experience to another difficult166 while another stated that
many types of stakeholders and data result in levels of complexity that can impede
implementation.165
Technology
The second most frequently cited (13 studies) category of barriers to implementation were
issues related to technology. More specifically these barriers related to the technological
environment. Two studies cited the lack of standards or differences in standards across
organizations in the terms and definitions used in the data as well as the format of data
sources.87,172 Similarly three studies reported that interoperability across systems was an
issue,85,142,151 while three more studies specifically mentioned difficulties related to EHR
interfaces that made exchange difficult or resulted in inappropriate or inaccuracy matching and
merging.143,154,167 Lack of system resilience, including operating speed and reliability was
identified in a study of HIE activities in 31 European countries94 while a study in the United
States cited lack of information system capacity, particularly in smaller organizations. The
authors of the study in European countries concluded, “we can pinpoint some clear bottlenecks
in terms of ‘electronically embedded’ system inter-connection with other healthcare players,
technical inter-operability, system resilience, and security.[…].Limited adoption of Health
Information Exchange (HIE) is surely also a consequence of such bottlenecks.”94 One study was
less circumspect in citing problems with vendors and reporting that, “the most significant
barriers … were largely due to a long and arduous process of collaborating with commercial
entities involved in technology design and delivery.”48
Lack of Necessary Components

This was presented as a barrier in five studies. Four studies reported that participants or
providers were not sufficiently engaged in implementation of the HIE or were not aware of its
value.84,141,154,158 One study emphasized that physician engagement was important by pointing
out that physicians are the primary source of care data and suggested that for this reason their
engagement is the primary determinant of HIE success.154 One study focused on the challenges
in securing data sharing agreements as a barrier to implementation.143
Fit
This is short hand for the correspondence between an innovation and the potential adopting
organizations. Lack of fit is a barrier that may not be apparent when the innovation is assessed
out of context. Two studies found that HIE implementation was deterred when organizations or
departments were unable or unwilling to integrate HIE into work processes.152,167 Another
instance where lack of fit is problematic is when expectations are not met. Two studies reported
that expectation for the data in terms of timeliness and completeness were barriers to
implementation.100,145 One additional study underlined the fact that timelines were not realistic,
particularly in cases where the technology was to be integrated into quality improvement
activites.143
76
User Interface and Functionality
Eight studies cited specific user interface and functionality problems as barriers to
implementation. These included lacking the technology and human resources needed to adapt the
organization’s software and processes for HIE,141 and the need for training and expertise.142,174
Two studies reported that user problems as fundamental as forgotten logons145 and the technical
performance of network connections hindered implementation.116 One study reported corrupt
data as a barrier to HIE,172 while another reported the lack of tests that identify that the ability to
match patients across systems were a barrier to development.82 One study of an advanced
application of a system to generate alerts based on HIE data stalled when the providers to notify
about a patient’s events could not be identified.100
Table 11. Barriers to implementation and sustainability of HIE

Number of Studies Number of Studies
Barrier Reporting Implementation Reporting Sustainability
Barriers Negative Influences
External policy* 3 1
Laws and regulations 1162
Changes in external (Federal, State) policy 1164
Funding uncertainty and timelines 2143,174
Disincentives* 15 4
Competition/difficult business case 4108,142,151,155 4148,149,156,173
Costs/financial viability 579,85,141,158,167
Organizations that invests does not benefit 3155,158,160
Resources (funding and time) 284,87
Concerns about data misuse, privacy, or ethics 4124,142,148,160
Structure* 4 3
Geographic coverage mismatch with service areas 1148 1156
Lack of leadership and management communication 285,174
Larger hospital systems (less need for external 1149
exchange)
Focus on long term care 1171
Governance/trust 2153,156
Technology* 9 1
Lack or differences in standards 287,172 1162
EHR interface 3143,154,167
Interoperability across systems 385,142,151
Problems with vendors 148
Lack of necessary components* 5 1

Participant/provider engagement, awareness of value 484,141,154,158 1146
Securing data sharing agreements 1143
77
Table 11. Barriers to implementation and sustainability of HIE (continued)
Number of Studies Number of Studies
Barrier Reporting Implementation Reporting Sustainability
Barriers Negative Influences
Fit* 5
Inability or willingness to integrate into work 2152,167
processes
Lack of enough time for development and integration 1143
into Quality Improvement
Failure to meet expectations that data needs will be 2100,145
timely, complete and meet expectations.
User interface and functionality* 8
Tech and HR resources to adapt software and 1141
processes
Need for training and expertise 2142,174
Corrupt data 1172
User interface and technical performance 2116,145
Ability to match patients 182
Difficulty identifying provider to get alerts generated 1100
from HIE
EHR = electronic health record; HIE = health information exchange; HR = human resources; IT = information technology
*Bold indicates overall category of barrier.
Subgroup Differences
During our review we attempted to abstract data from the included studies that would allow
us to determine if the barriers and facilitators to implementation varied by type of HIE, health
care settings, and systems or IT system characteristics. Most publications did not include this
information so we were not able to consistently identify any differences.
We also considered that implementation might change over time as HIE becomes more
common and as new HIE efforts could benefit from the experience of early adapters. At this time
we do not see any notable changes, but this may be to the relatively short time period (less than a
decade) covered by the included studies. While the hardware and software that make HIE
possible have changed significantly in less than a decade, organizational change and clinical
practice patterns have historically changed more slowly.
Sustainability
In making a distinction and summarizing the factors identified in the 17 studies that
considered sustainability separately, we placed studies according to what the researchers/authors
reported as their focus and we accepted their definitions and/or measures.44,108,123,140,146-
149,153,156,159,160,162,163,166,171,173
As HIE and health IT mature, a definition of successful
sustainability may be developed and the evidence could them be reanalyzed incorporating such a
definition.
The factors that have been found to influence the sustainability of HIE fit into the categories
created to summarize the facilitators and barrier for implementation, and in some cases it can be
difficult to make a distinction. This is in part because sustainability is still a future goal rather for
all but the organizations that were very early adopters of HIE.
78
We presented the sustainability factors under the most appropriate category on Tables 10 and
11, but added rows for specific factors when they differ from those identified in studies of
implementation.
Ten included studies identified factors that are positive influences on sustainability. These
included having an HIE implementation led by a health information organization as opposed to a
health care organization171 and having leadership and technology that allowed the HIE
organization to innovate and react quickly to changes in the market and environment.163
Sustainability was also linked to marketing the HIE to patients,123 to how an HIE system
incorporated a community needs assessment,171 and if it selected functions likely to financially
benefit the participants.147,160 One study suggested that HIE implementations with advanced
functions such as providing decision support are more sustainable147 while another pointed out
that these functions should add value related to either Stage 2 meaningful use or reform priorities
in order to support sustainability.159 Achieving important expected outcomes such as improved
quality of care153 and realizing savings that exceed the costs of the HIE system are
understandably important44 and one study described how most of the HIE organizations it
examined are developing subscription fee structures to provide ongoing financial support.159 One
study reported that laws and mandates could promote sustainability as well as implementation of
HIE. 140
However, laws and mandates, particularly changes in these were also one of the reported
negative influences on sustainability.162,166 Four studies found that competition and a difficult
business case for HIE were challenges to sustainability.148,149,156,173 Four structural characteristics
of HIE were also identified. These included the mismatch between the HIE geographic coverage
and where patients receive services,156 issues related to governance and trust among the HIE
collaborators,153,156 and one study found that HIE that focused on long-term care organizations
were less likely to be sustainable.171 Lack of standards was the only factor directly related to the
technology for HIE reported among the negative influences and it was reported in only one
study.162 Lack of sufficient engagement of participants and providers was also reported in one
study.146
While there was less evidence related to sustainability to report in this review than for
implementation, the studies to date suggest it is the more complex of two very complex and
related topics. One researcher suggested this complexity when making the assessment that this
issue for HIE sustainability are sociological not technological.156 Another suggested
sustainability may become less a matter of availability of funds and more one of trust and
responsible stewardship.123 Combined, this result seems to be that sustainability of HIE activities
is further in the future than many originally thought. As one observer noted “recent history
suggests that achieving the kind of ubiquitous use among providers or other users that can drive a
financial value proposition takes time—and likely more time than HIOs have modeled in their
sustainability plans.”163
79
Discussion
Key Findings
• We found no studies of health information exchange (HIE) that reported the impact on
clinical outcomes or that identified harms.
• The majority of the included studies reported that HIE improved resource use by
reducing lab tests, imaging, or hospital admissions and improved quality of care, but the
strength of evidence was low for all outcomes.
• Studies found that HIE was used by between 30 and 58 percent of hospitals and, 38
percent of office-based physicians in 2012, while use remains low in long-term care
settings.
• Within organizations, studies that looked at the number of users or the number of visits in
which the HIE is used found generally very low rates of use.
• Studies did not link usability of HIE to effectiveness but they did link it to use.
• The most commonly cited barriers to HIE use were incomplete patient information,
inefficient workflow, and poorly designed interface and update features.
• Eight categories of factors facilitated HIE: seven cateogires that are internal
characteristics while external factors were less frequently cited and we combined these
into one category.
• Barriers identified in research on HIE implementation focused more on the external
environment (7 categories). Disincentives was the largest category of barriers.
• Factors that influenced sustainability were similiar to the barriers and facilitiators of
implementation. The most frequently cited negative influence was competition and the
lack of a business case for HIE.
Key findings are summarized in Table 12.
Table12. Summary of evidence

Number of Included
Primary Limitations of the
Topic Studies Main Findings
Evidence
Type
Effectiveness 34; Low-quality evidence Studies were from a small number
20 Retrospective cohort somewhat supports the of the functioning HIE
3 Randomized controlled value of HIE for reducing implementations, with similarity to
trial duplicative laboratory and unstudied ones unknown, possibly
2 Cross-sectional radiology test ordering, limiting generalizability.
2 Case series lowering ED costs, reducing
8 Survey (1 survey study hospital admissions (less so Studies looked at limited
was an RCT) for readmissions), improving outcomes compared with the
public health reporting, intended scope of the impact of
increasing ambulatory HIE.
quality of care, and
improving disability claims
processing. No evidence of
harms was reported.
80
Table12. Summary of evidence (continued)
Number of Included
Primary Limitations of the
Topic Studies Main Findings
Evidence
Type
Use 58; Proportion of hospitals and While there are relatively high
25 Surveys ambulatory care practices quality national and regional
13 Audit Logs that have adopted HIE is surveys and reports that are
9 Retrospective database increasing. tracking the expansion of HIE
7 Mixed methods among health care organizations,
2 Focus Groups Currently, proportion of there is not a corresponding
1 Time-motion clinicians using HIE and comprehensive effort to track
1 Geo-Coding proportion of patients or changes in rates of use within
episodes associated with organizations.
HIE use are generally low.
Usability and 22; 3 most commonly cited Studies of usability did not relate it
other factors 9 Multiple site case studies barriers to HIE use were: to effectiveness and do not permit
affecting use 11 Cross-sectional incomplete patient comparisons across settings or
2 Before-after information (8 studies); type of HIE.
inefficient workflow (6
studies); poorly designed Studies had limitations such as
interface and update incomplete reporting on sampling,
features (6 studies). low response rates or selection of
a narrow setting or patient
population which minimize
applicability.
Implementation 52; Most facilitators of Studies do not allow comparison
and 26 Cross-sectional implementation are of the impact of different barrier
sustainability 17 Multiple site case studies characteristics of the HIE or and facilitators.
2 Before-after the internal organizational
3 Retrospective cohorts environment. Many barriers The definition and appropriate
2 Prospective cohorts to implementation are measure of sustainability are not
2 Time series external, environmental yet clear.
factors.
Factors related to
sustainability overlap with
those identified for
implementation.
ED = emergency department; HIE = health information exchange; RCT = randomized controlled trial
Strength of Evidence
Assessing the overall strength of the evidence for this review was complex, given (1) the
very broad scope of the review; (2) the large variety of effects and outcomes examined by
investigators; (3) the diverse types of evidence and study designs; (4) the differing units of
analysis and intervention (from episodes of care, to individual clinicians or patients, to hospitals
or clinics, to health systems, to regional or statewide efforts); (5) the multiple contexts of care,
from acute care in emergency department (ED) visits to public health reporting and analysis; (6)
the variety of technical implementations, even within the broad categories of query-based and
directed HIE; and (7) the likelihood of reporting bias, expected to be in the direction of positive
findings, with likely under-reporting of failed or ineffective HIE. In view of these challenges, we
elected to explicitly and systematically assess the risk of bias and strength of evidence only for
studies addressing the effectiveness and harms of HIE, our Key Questions 1, 2, and 3.
81
These limitations notwithstanding, a collection of low-quality evidence somewhat supports
the value of HIE for reducing duplicative laboratory and radiology test ordering, lowering ED
costs, reducing hospital admissions (less so for readmissions), improving public health reporting,
increasing ambulatory quality of care, and improving disability claims processing. The evidence
is low-quality because of the retrospective nature of the studies and the limited questions that
they ask. It is unlikely that additional studies of the kind included in this review will alter the
overall conclusion that HIE can reduce laboratory and imaging tests associated with episodes of
care without broadening their scope and using more rigorous designs. Though the preponderance
of evidence supports positive effects in terms of reduced resource use and improved quality of
care, it is entirely possible that focused studies with stronger study designs and more
comprehensive assessment of utilization or clinical outcomes might reach a different conclusion.
With respect to cost, we did not identify any studies that employed systematic and
comprehensive economic analysis. Although some of the studies we included projected or
estimated cost savings based on measured changes in utilization or perceptions of clinicians,
there were no studies that explicitly measured costs and assessed economic impact in a
comprehensive fashion. It is fair to say, then, that there was insufficient evidence to reach
conclusions on the economic impact of HIE.
As stated previously, we found no studies explicitly addressing patient-specific clinical
outcomes such as morbidity, mortality, or functional status and hence the body of evidence is
insufficient to determine whether HIE has an impact on patient outcomes.
Findings in Relationship to What Is Already Known

The findings of this review add to the substantial, albeit methodologically challenging,
evidence base relating to health information technology (IT) generally and HIE in particular. A
series of comprehensive and systematic reviews of health IT have been published over the last
decade, including three from a single Evidence-based Practice Center (EPC)5,6,8and one from the
Office of the National Coordinator7 confronted similar challenges in the diversity and breadth of
settings, interventions, and outcomes. Overall, these reviews found that the preponderance of
studies of health IT reported generally positive or “mixed-positive” effects, but with caveats
about the likelihood of publication bias, methodological limitations of the studies, and the
concentration of studies coming from a relatively small number of institutions.
The present systematic review of HIE can be compared with two other systematic reviews of
HIE: one by Rudin et al.32 and another by Rahurkar et al.33 The three systematic reviews used
generally similar approaches, with similar definitions of HIE and focus on studies of HIE impact,
excluding system descriptions and simple case studies. The three reviews differ, however, in
their scope and inclusiveness.
The review by Rahurkar et al. was most narrow in scope, addressing only the impact of HIE
on “health outcomes,” in which the authors included utilization and cost measures. They
searched two databases, Scopus and MEDLINE (along with reference mining), included non-
U.S. studies, and excluded systematic reviews, qualitative studies, and studies of exchange of
administrative and financial information.33
The review by Rudin et al. was broader.32 In addition to health and utilization outcomes of
HIE, they considered studies of patient and provider attitudes, barriers and facilitators to HIE
use, and financial sustainability. These authors searched three databases, MEDLINE, Web of
Science, and the Cochrane databases (along with reference mining), and they excluded studies of
82
public health settings (included by Rahurkar et al.), administrative and financial information
exchange, non-U.S. studies, and studies of usability.
Our review was the most broad in scope of the three, and the most inclusive in the search for
evidence. In addition to patient and population health outcomes, economic, utilization process
outcomes, and barriers and facilitators to implementation and use, our review also included
studies concerned with use and usability of HIE. We also explicitly searched for reports of harms
of HIE (although none were found). Our review was also more comprehensive in the search for
evidence, searching MEDLINE, PsychInfo, CINAHL, the Cochrane databases, Database of
Abstracts of Reviews of Effects, and the National Health Sciences Economic Evaluation
Database, as well as reference mining. We also did trial scans of the Business Premier and the
Institute of Electrical and Electronics Engineers (IEEE) Xplore Digital Library; databases for any
potential relevant evidence. In addition, we included non-U.S. studies, and studies that reported
on public health and surveillance uses as well as exchange of administrative and financial
information.
The three reviews are based on comparable but not identical evidence bases. The present
review includes a total of 136 studies. The review by Rudin et al. included 85 studies, 55 of
which were also included in our review, and the review by Rahurkar et al. included 27 studies,
18 of which were also included in our review. We examined the references of both of these
reviews and included any that met our inclusion criteria.
The overall result is that we examined a more diverse and more inclusive collection of
evidence, especially with respect to usability and use as well as assessing public health settings,
but came to largely similar conclusions. Rahurkar et al. performed a multivariable analysis that
found that study design was the only characteristic associated with finding a beneficial effect,
with the most rigorous studies being less likely to report benefits of HIE.
The problem of overlap across systematic reviews is an important one and has recently been
addressed in the methods guides of the Cochrane Collaboration175-177and the Agency for
Healthcare Research and Quality EPC program.178 When large numbers of systematic reviews
are conducted, there is inevitable overlap when two reviews are based on the same body of
evidence. Additional reviews on a subject do not indicate more evidence on the question, only
more thorough (when independent) examination of the same evidence.
A notable point to be made about the comparison between these reviews is that three review
groups have now independently searched for and assessed the evidence on the effectiveness of
HIE and are in agreement on the main conclusions. This raises the level of confidence in the
conclusions in that the three reviews represent independent replication of one another’s work,
albeit with the same rather significant limitations in the body of evidence on which the
conclusions are based.
Applicability
Are the effects reported on in this review, limited as they are, likely to be observed when
applied under “real world” conditions in health systems, hospitals, and clinics in the United
States? The greatest confidence in the applicability of these findings comes from the breadth of
settings – geographic, organizational, and technical – from which they are derived. That is to say,
for the most part, it can be expected that: (1) near-term resource utilization in the form of
laboratory and imaging test ordering is likely to be reduced when effective HIE capabilities are
deployed, while the effect on other utilization and quality indicators is harder to predict; (2) use
of HIE will be highly dependent on the context of use, perceived value of the information to the
83
patient care task, and the degree of integration into clinical workflows, including potential
delegation by clinicians to other members of the health care team depending on the setting; and
(3) hospital and health system implementation and participation in HIE will be driven by the
perceived value and return on investment, alignment with organizational goals, internal capacity
to address technical challenges, and the presence of local and national external financial,
regulatory, and policy constraints.
On the other hand, there are limitations to the applicability of the findings (beyond
limitations to the internal validity already mentioned) having to do with three main concerns: (1)
concentration of evidence from a relatively small number of sources; (2) use of internally
developed and refined health IT systems compared with local instances of commercial systems;
and (3) the exceptionally broad variety of systems, contexts, and purposes of HIE reported in the
studies included in this review.
First, the concern that the bulk of the evidence about health IT impact arises out of a
relatively small number of centers has been raised before.5 These centers have been referred to as
“health IT leaders,” which are typically large academic medical centers with internally
developed health IT systems, implemented incrementally, and refined over a long period of time.
The nature of the health IT systems is in each case unique (being locally developed), and more
importantly it is difficult to separate the effects of the health IT from the confounding influences
of the health system itself. Whether findings from these systems can be generalized to the very
different context of health system and hospital implementations of commercially developed
systems over shorter periods of time with less internal development and implementation
infrastructure has been called into question.5 This “health IT leader” effect appears to be reduced
in more recent updates to the 2006 systematic review by Chaudhry et al. but the issue remains
important.6,8 In the present review of HIE the concentration of evidence phenomenon is also
present, with large numbers of published studies emanating from relatively few areas, this time
regional implementation programs rather than academic health centers, such as Texas, New
York, and the MidSouth e-Health Alliance.
Second, separate from the “health IT leader” concern, which has to do with the
organizational capacity, resources, and mission of these centers, is the issue of internally
developed systems compared with commercially developed systems. Though no implementation
is truly “off the shelf” because of customization of local instances of commercial systems, the
overall model of health IT purchase and installation is quite different from that of incremental
internal development, implementation, and refinement, such as one sees in systems such as the
Veterans Affairs or the aforementioned “health IT leader” systems. Related to this concern is a
finding from other aspects of health IT,176 namely clinical decision support, that systems
evaluated by their developers tend to achieve more positive outcomes from their evaluation than
external evaluators. This phenomenon must be assessed with HIE as well.
Third and most important in terms of limiting the applicability of these findings about HIE to
real-world use is the exceptionally wide variety of systems, purposes, contexts of use, and
outcomes examined. To address the Key Questions of this systematic review, highly diverse
evidence has been combined to answer general questions about the overall effectiveness of HIE
for various outcomes. However, to predict whether specific implementations of HIE in specific
health care contexts will have favorable impacts on specific desired outcomes is not possible
from this review and in most cases would not be possible from comparison with individual
studies because (a) it is unlikely that studies with low risk of bias have been published for most
84
such specific questions, and (b) in almost all cases these are complex interventions which are
incompletely specified, with insufficient detail to draw strong meaningful inferences.179
Limitations of the Evidence Base

The very significant limitations of the evidence base, that is, the individual studies included
in this review, have been raised in previous systematic reviews of health IT5,6,8 and of HIE.32
Although increasing in number, the relative proportion of well-conducted studies with rigorous
designs remains small, and we know from experience in other domains, such as hormone
replacement therapy, that even a very large number of well-conducted observational studies may
be found to have misled us when results of rigorous experiments become available.180 In view of
this fact, one must continue to proceed with caution when interpreting and applying the results of
observational studies, even well-conducted ones.
Beyond this, there are three primary concerns about the limitations of the available evidence
on the impact of HIE (and health IT in general): (1) suitability of study design; (2) execution of
the studies; and (3) complexity of the interventions with implications for interpretation and for
generalizability.
First, the evidence in this area addresses a wide variety of questions covering diverse
domains beyond medical science from computer science, human factors, sociology, organization
and management and other disciplines. This broad array of questions calls for an equally diverse
range of study designs. Studies of usability and use require usability engineering methods,
studies of individual behavior call for methods from anthropology and behavioral sciences,
studies of organizational change warrant methods drawn from management and systems science,
while studies of population effects call for the methods of epidemiologists. As Sackett and
Wennberg noted, “the question being asked determines the appropriate research architecture,
strategy, and tactics to be used—not tradition, authority, experts, paradigms, or schools of
thought.”181 A significant limitation of this literature, with its breadth of research questions, is
the limited toolbox often drawn upon to answer them.
The second main area of limitation is in execution of the studies. Even when strong study
designs are chosen, their execution may be lacking, whether in sampling strategies, measurement
methods, or analytic approaches. The unit of analysis problem is but one example. Interventions
carried out at the level of the health system, hospital, or clinic may be analyzed at the level of the
patient or episode, without controlling for variation at these multiple levels. Incomplete
measurement is another: for example where ED test ordering is measured in isolation, ignoring
the possiblility that the same test might later be ordered in another setting such as urgent care,
primary care, or in hospital.
The third main area has to do with the complexity of interventions, where the HIE or other
health IT system itself is necessarily only part of a more complex intervention. The complexity
of interventions to change the behavior of clinicians or others in the health systems studied
requires more thorough specification, both in order to adjust for confounders and in order to
make sense out of how to apply interventions elsewhere. Others have documented the
inadequacy of specification of the details of complex interventions and called for a more
systematic and thorough reporting.179,182
Future Research Needs

Given the limited conclusions that can be reached after review of so much published
literature on the effects, use, sustainability, and barriers to implementation and use of HIE, what
85
are the implications for future research? Recognizing that HIE, like health IT in general, will
almost certainly undergo increasingly widespread implementation in the future, the first aim of
researchers should be to shift the emphasis from whether HIE systems should be implemented to
specifically how they should be implemented. The quesion to be answered is not “Does HIE have
positive effects?“ but rather “How can HIE be implemented in order to result in the greatest
benefit for patients, clinicians, and health systems with the least cost and harm?”
The second aim of researchers on HIE should be to develop greater focus and clarity about
the level at which interventions are operating and the types and levels at which outcomes are
measured. The outcomes of interest and the factors influencing them may be quite different at
different levels of analysis, from specific systems or functionalities of HIE; to individual
patients, providers, or episodes of care; to health care units such as the ED, primary care practice,
or hospital ward; to institutions such as hospitals; to aggregates such as health systems; or
broader regional multi-organization entities or regions. Combining or confusing these levels of
intervention and levels of analysis only increase the challenges for those who conduct the
research and for those who wish to interpret and apply it.
To help achieve an improved focus and clarity, a more formal analytic framework and a more
descriptive taxonomy are needed. An example of such a framework that could be usefully
applied in this area is Rasmussen’s socio-technical hierarchy, which specifies the multiple levels
of a complex sociotechnical system that must be considered together to understand system
behavior change.183 Examples of its application include Vicente’s analysis of the forces acting at
multiple levels (Figure 3) to reduce hazards arising from patient controlled analgesia devices184
and Leveson’s Systems—Theoretic Accident Modeling and Processes (STAMP) model for
understanding system performance and safety.185
86
Figure 3. Rasmussen sociotechnical analysis framework* 184
*© Joint Commission Resources: Joint Commission Journal on Quality and Patient Safety. Volume 29: Issue 11. p.599, (2003).
Reprinted with permission.
Recognizing that one cannot understand a system by separately analyzing its parts, Rasmussen developed this analytical
framework that encompasses the full range of dimensions that must be considered together to make sense of socio-technical
behavior.
Similarly, a formal taxonomy for implementation of complex interventions has been

proposed which would enable more complete and useful specification of interventions to allow
better analysis, interpretation, and application.179,187 This taxonomy should be extended specific
to HIE to include clinical, technical, and organizational details of the HIE implementation. The
clinical taxonomy should focus not only on patient outcomes, but also on issues such as health
disparities related to HIE and health system issues that may improve or undermine use of HIE.
The technical taxonomy should include aspects of system architecture, messaging and
terminology standards, and other details. The HIE research community should consider a
standardized reporting instrument for HIE evaluation comparable to the Consolidated Standards
of Reporting Trials (CONSORT) statement for randomized controlled trials (RCTs).188
The third step researchers can take to improve the evidence base for implementation of HIE
is to broaden the methodologic toolbox applied to these questions. As indicated above, the study
approach and architecture must be suited to the question being asked, employing methods from
usability engineering, behavioral sciences, systems engineering, and organizational sciences,
87
depending on the question being addressed. These would include methods used in engineering
and quality improvement, as well as in the study of complex adaptive systems. In epidemiology
it has been proposed that health and health care can be fruitfully studied as complex adaptive
systems, which require “different methods from the usual epidemiological techniques.”189
Examples include infectious disease epidemiology, smoking,189 and obesity.190 Because “(i)
factors at multiple levels, including biological, behavioural and group levels may influence
health and disease, and (ii) … the interrelation among these factors often includes dynamic
feedback and changes over time,” new approaches are needed to complement the classic methods
of clinical trials which are frequently unsuitable for complex interventions in organizational
contexts.
What types of studies should be performed? RCTs are impractical for technologies with
wide-ranging purposes like HIE. Yet, retrospective studies associating HIE versus non-use for
outcomes such as test ordering and hospital admissions are very limited in conclusions that can
be drawn. Research is also challenging because many of the important clinical outcomes that
could benefit from HIE have many other potential contributing and confounding factors relating
to the patient, his or her clinicians, the quality of care delivered, the electronic health record, and
other health IT used, the nature of the health care delivery system, the regulatory environment,
and many more.
Future studies should be prospective, carried out in mature HIE settings, assessing patients
who are likely to benefit from HIE and comparing appropriate outcomes for the use or non-use
of HIE. The prospective collection of data from diverse settings where HIE is used, classified by
the taxonomy advocated above, could allow for prospective cohort studies that could identify
aspects of HIE associated with beneficial outcomes. This will likely require an effort comparable
in scope to national data collection efforts, such as the Patient-Centered Outcomes Research
Institute Clinical Data Research Network initiative.191 Ideally such an undertaking could be
synergistic with these other large-scale efforts.
Evaluation should be a requirement for all HIE implementations, certainly those funded by
grants or other external funding. The challenges of evaluating health IT projects, especially in
community settings, is well-known,30 but all funders must demand this requirement to grow the
evidence base. By the same token, funders must provide adequate resources for such evaluations.
In addition, evaluation should be performed by researchers external to the project to reduce
potential bias from system developers evaluating their own implementations.176
Conclusions
The full impact of HIE on clinical outcomes and potential harms is insufficiently studied,
although evidence provides some support for benefit in reducing use of some specific resources
and achieving improvements in quality of care measures. Use of HIE has increased over time and
is highest in hospitals and lowest in long-term care settings. However, use of HIE within
organizations that offer it is still low. Barriers to HIE use include incomplete patient information,
inefficient workflow, and poorly designed interface and update features, but factors affecting
implementation and sustainability remain unclear. To advance our understanding of HIE, future
studies need to address comprehensive questions, use more rigorous designs, and be part of a
coordinated, systematic approach to studying HIE.
88
References
1. Crossing the Quality Chasm: A New Health 8. Jones SS, Rudin RS, Perry T, et al. Health
System for the 21st Century. Washington, information technology: an updated
DC: National Academies Press; 2001. systematic review with a focus on
meaningful use. Ann Intern Med.
2. Dustin Charles, Meghan Gabriel, Furukawa 2014;160(1):48-54. PMID: 24573664.
M. Adoption of Electronic Health Record
Systems among U.S. Non-federal Acute 9. Bourgeois FC, Olson KL, Mandl KD.
Care Hospitals: 2008-2013. ONC Data Patients treated at multiple acute health care
Brief, no. 16. The Office of the National facilities: quantifying information
Coordinator for Health Information fragmentation. Arch Intern Med.
Technology. Washington, DC. Available at: 2010;170(22):1989-95. PMID: 21149756.
www.healthit.gov/sites/default/files/oncdata 10. Finnell JT, Overhage JM, Grannis S. All
brief16.pdf. Accessed March 2, 2014. health care is not local: an evaluation of the
3. Hsiao C-J, Hing E. Use and Characteristics distribution of emergency department care
of Electronic Health Record Systems delivered in Indiana. AMIA Annu Symp
Among Office-based Physician Practices: Proc. 2011;2011:409-16. PMID: 22195094.
United States, 2001–2013. NCHS Data 11. Clancy C. Keynote Adress AMIA 2007
Brief, No. 143. National Center for Health Anual Symposium. 2007.
Statistics. Hyattsville, MD. Available at:
www.cdc.gov/nchs/data/databriefs/db143.ht 12. What is HIE (Health Information
m. Accessed March 3, 2015. PMID: Exchange)? Department of Health and
24439138. Human Services. Washington, DC.
Available at: www.healthit.gov/providers-
4. Eric Jamoom, Esther Hing. Progress With professionals/health-information-
Electronic Health Record Adoption Among exchange/what-hie. Accessed April 18,
Emergency and Outpatient Departments: 2014.
United States, 2006–2011. NCHS Data
Brief, No. 187. National Center for Health 13. Williams C, Mostashari F, Mertz K, et al.
Statistics. Hyattsville, MD. Available at: From the Office of the National
www.cdc.gov/nchs/data/databriefs/db187.ht Coordinator: the strategy for advancing the
m. Accessed March 3, 2015. PMID: exchange of health information.[Erratum
25714041. appears in Health Aff (Millwood). 2012
Mar;31(3):886]. Health Aff.
5. Chaudhry B, Wang J, Wu S, et al. 2012;31(3):527-36. PMID: 22392663.
Systematic review: impact of health
information technology on quality, 14. Health Information Exchange Roadmap:
efficiency, and costs of medical care. Ann The Landscape and a Path Forward.
Intern Med. 2006;144(10):742-52. PMID: National eHealth Collaborative.
16702590. Washington, DC. Available at:
www.riqi.org/matriarch/documents/NeHC%
6. Goldzweig CL, Towfigh A, Maglione M, et 20Roadmap%20for%20HIE%20-
al. Costs and benefits of health information %20The%20Landscape%20and%20a%20Pa
technology: new trends from the literature. th%20Forward.pdf. Accessed June 17, 2015.
Health Aff (Millwood). 2009;28(2):w282-
93. PMID: 19174390. 15. Dullabh P, Ubri P, Loganathan S, et al.
Evaluation of the State Health Information
7. Buntin MB, Burke MF, Hoaglin MC, et al. Exchange Cooperative Agreement Program:
The benefits of health information State Approaches to Enabling HIE:
technology: a review of the recent literature Typology Brief. Office of the National
shows predominantly positive results. Coordinator for Health Information
Health Aff (Millwood). 2011;30(3):464-71. Technology. 2014. Available at:
PMID: 21383365. www.healthit.gov/sites/default/files/statehiet
ypologybrief.pdf. Accessed April 10, 2015.
89
16. Glasgow A. The nouns and verbs of 25. Adler-Milstein J, Bates DW, Jha AK.
successful HIE. Healthcare IT News. Operational health information exchanges
Available at: www.healthcareitnews.com/ show substantial growth, but long-term
news/nouns-and-verbs-successful-hie. funding remains a concern. Health Aff
Accessed April 19, 2015. (Millwood). 2013;32(8):1486-92. PMID:
23840051.
17. Mary Mosquera. HIE as a verb: ONC want
to move quickly on data exchange. 26. Swain M, Charles D, Patel V, et al. Health
Healthcare IT News. Available at: Information Exchange among U.S. Non-
www.healthcareitnews.com/news/hie-verb- federal Acute Care Hospitals: 2008-2014.
onc-wants-move-quickly-data-exchange. ONC Data Brief No. 24. Washington DC:
Accessed April 19, 2015. The Office of the national Coordinator for
Health Information Technology (ONC).
18. McDonald CJ. The barriers to electronic
2015. Available at: www.healthit.gov/sites/
medical record systems and how to
default/files/data-brief/ONC_DataBrief24_
overcome them. J Am Med Inform Assoc.
HIE_Final.pdf. Accessed April 19, 2015.
1997;4(3):213-21. PMID: 9147340.
27. The Office of the National Coordinator for
19. McDonald C, Overhage J, Barnes M, et al.
The Indiana Network for Patient Care: a
Federal Health IT Strategic Plan 2015-2020.
working local health information
2014. Available at:
infrastructure. Health Aff. 2005;24:1214-20.
www.healthit.gov/sites/default/files/federal-
PMID: 16162565.
healthIT-strategic-plan-2014.pdf. Accessed
20. Frohlich J, Karp S, Smith MD, et al. April 23, 2015.
Retrospective: lessons learned from the
28. The Office of the National Coordinator for
Santa Barbara project and their implications
for health information exchange. Health Aff
Connecting Health Care for the Nation. A
(Millwood). 2007;26(5):w589-91. PMID:
shared nationwide Interoperability
17670777.
Roadmap. Draft Version 1.0. 2015.
21. Kuperman GJ. Health-information Available at: /www.healthit.gov/sites/
exchange: why are we doing it, and what are default/files/nationwide-interoperability-
we doing? J Am Med Inform Assoc. roadmap-draft-version-1.0.pdf. Accessed
2011;18(5):678-82. PMID: 21676940. April 23, 2015.
22. McGowan J, Kuperman G, Olinger L, et al. 29. The Office of the National Coordinator for
Strengthening Health Information Health Information Technology (ONC).
Exchange: Final Report HIE Unintended Report on Health Information Blocking.
Consequences Work Group. Rockville, Washington DC: Department of Health and
MD: Westat. 2012. Available at: Human Services. 2015. Available at:
www.healthit.gov/sites/default/files/hie_uc_ www.healthit.gov/sites/default/files/reports/i
workgroup_final_report.pdf. Accessed nfo_blocking_040915.pdf. Accessed April
January 7, 2014. 17, 2015.
23. Yasnoff WA, Sweeney L, Shortliffe EH. 30. Kern LM, Ancker JS, Abramson E, et al.
Putting health IT on the path to success. Evaluating health information technology in
JAMA. 2013;309(10):989-90. PMID: community-based settings: lessons learned. J
23483170. Am Med Inform Assoc. 2011;18(6):749-53.
PMID: 21807649.
24. Adler-Milstein J, Jha AK. Sharing clinical
data electronically: a critical challenge for 31. Hincapie A, Warholak T. The impact of
fixing the health care system. JAMA. health information exchange on health
2012;307(16):1695-6. PMID: 22535851. outcomes. Appl Clin Inform. 2011;2(4):499-
507. PMID: 23616891.
90
32. Rudin RS, Motala A, Goldzweig CL, et al. 39. Bailey JE, Wan JY, Mabry LM, et al. Does
Usage and effect of health information health information exchange reduce
exchange: a systematic review. Ann Intern unnecessary neuroimaging and improve
Med. 2014;161(11):803-11. PMID: quality of headache care in the emergency
25437408. department? J Gen Intern Med.
2013;28(2):176-83. PMID: 22648609.
33. Rahurkar S, Vest JR, Menachemi N. Despite
the spread of health information exchange, 40. Bailey JE, Pope RA, Elliott EC, et al. Health
there is little evidence of its impact on cost, information exchange reduces repeated
use, and quality of care. Health Aff diagnostic imaging for back pain. Ann
(Millwood). 2015;34(3):477-83. PMID: Emerg Med. 2013;62(1):16-24. PMID:
25732499. 23465552.
34. Yi R, Samarth A, Dearfield C, et al. 41. Ben-Assuli O, Shabtai I, Leshno M. The
Synthesis of Lessons Learned in the First 5 impact of EHR and HIE on reducing
Years of State and Regional Demonstration avoidable admissions: controlling main
Health Information Exchange Projects. differential diagnoses. BMC Med Inform
AHRQ Publication No. 11-0050-EF. Decis Mak. 2013;13:49. PMID: 23594488.
42. Dixon BE, McGowan JJ, Grannis SJ.
Quality and Research. March, 2011.
Electronic laboratory data quality and the
Available at: http://healthit.ahrq.gov/sites/
value of a health information exchange to
default/files/docs/citation/synthesis_of_lesso
support public health reporting processes.
ns_learned.pdf. Accessed April 18, 2014.
35. Pan E BC, Damico D, Crimmins M. Guide PMID: 22195084.
to Evaluating Health Information Exchange
43. Feldman SS, Horan TA. Collaboration in
Projects (Prepared for the Agency for
electronic medical evidence development: a
Healthcare Research and Quality under
case study of the Social Security
Contract No.290200900023-I.). AHRQ
Administration's MEGAHIT System. Int J
Publication No. 14-0015-EF. Rockville,
Med Inf. 2011;80(8):e127-40. PMID:
MD: Agency for Healthcare Research and
21333588.
Quality. September 2014. Available at:
http://healthit.ahrq.gov/sites/default/files/doc 44. Frisse ME, Johnson KB, Nian H, et al. The
s/page/guide-to-evaluating-hie-projects.pdf. financial impact of health information
Accessed April 23, 2015. exchange on emergency department care. J
Am Med Inform Assoc. 2012;19(3):328-33.
36. Methods Guide for Effectiveness and
PMID: 22058169.
Comparative Effectiveness Reviews. AHRQ
Publication Number 10(14)-EHC062-EF. 45. Kern LM, Barrón Y, Dhopeshwarkar RV, et
Rockville, MD: Agency for Healthcare al. Health information exchange and
Research and Quality. January 2014. ambulatory quality of care. Appl Clin
Available at: Inform. 2012;3(2):197-209. PMID:
www.effectivehealthcare.ahrq.gov. 23646072.
Accessed April 18, 2014. PMID: 21433403.
37. Hersh W, Totten A, Gorman P, et al. Health Outcomes assessment of the regional health
Information Exchange PROSPERO information exchange: a five-year follow-up
2014:CRD42014013285 Available at: study. Methods Inf Med. 2011;50(4):308-18.
www.crd.york.ac.uk/PROSPERO/display_re PMID: 21336419.
cord.asp?ID=CRD42014013285#.VTga1fnF
9u1. Accessed April 22, 2014. 47. Magnus M, Herwehe J, Gruber D, et al.
Improved HIV-related outcomes associated
38. Shea BJ, Hamel C, Wells GA, et al. with implementation of a novel public health
AMSTAR is a reliable and valid information exchange. Int J Med Inf.
measurement tool to assess the 2012;81(10):e30-8. PMID: 22883431.
methodological quality of systematic
reviews. J Clin Epidemiol.
2009;62(10):1013-20. PMID: 19230606.
91
48. Nagykaldi ZJ, Yeaman B, Jones M, et al. 57. Altman R, Shapiro JS, Moore T, et al.
HIE-i-Health Information Exchange With Notifications of hospital events to outpatient
Intelligence. J Ambulatory Care Manage. clinicians using health information
2014;37(1):20-31. PMID: 24309392. exchange: a post-implementation survey.
Inform Prim Care. 2012;20(4):249-55.
49. Overhage JM, Grannis S, McDonald CJ. A
PMID: 23890336.
comparison of the completeness and
timeliness of automated electronic 58. Campion TR, Jr., Ancker JS, Edwards AM,
laboratory reporting and spontaneous et al. Push and pull: physician usage of and
reporting of notifiable conditions. Am J satisfaction with health information
Public Health. 2008;98(2):344-50. PMID: exchange. AMIA Annu Symp Proc.
18172157. 2012;2012:77-84. PMID: 23304275.
50. Ross SE, Radcliff TA, Leblanc WG, et al. 59. Chang KC, Overhage JM, Hui SL, et al.
Effects of health information exchange Enhancing laboratory report contents to
adoption on ambulatory testing rates. J Am improve outpatient management of test
Med Inform Assoc. 2013;20(6):1137-42. results. J Am Med Inform Assoc.
PMID: 23698257. 2010;17(1):99-103. PMID: 20064809.
51. Shapiro JS, Johnson SA, Angiollilo J, et al. 60. Kaushal R, Dhopeshwarkar R, Gottlieb L, et
Health Information Exchange Improves al. User experiences with pharmacy benefit
Identification Of Frequent Emergency manager data at the point of care. J Eval
Department Users. Health Aff. Clin Pract. 2010;16(6):1076-80. PMID:
2013;32(12):2193-8. PMID: 24301405. 20666888.
52. Tzeel A, Lawnicki V, Pemble KR. The 61. Maass MC, Asikainen P, Mäenpää T, et al.
Business Case for Payer Support of a Usefulness of a Regional Health Care
Community-Based Health Information Information System in primary care. A case
Exchange: A Humana Pilot Evaluating Its study. Comput Methods Programs Biomed.
Effectiveness in Cost Control for Plan 2008;91(2):175-81. PMID: 18514363.
Members Seeking Emergency Department
62. Machan C, Ammenwerth E, Schabetsberger
Care. Am Health Drug Benefits.
T. Evaluation of the electronic transmission
2011;4(4):207-15. PMID: 25126351.
of medical findings from hospitals to
53. Tzeel A, Lawnicki V, Pemble KR. "Hidden" practitioners by triangulation. Methods Inf
Value: How Indirect Benefits of Health Med. 2006;45(2):225-33. PMID: 16538293.
Information Exchange Further Promote
63. Park H, Lee S-i, Kim Y, et al. Patients'
Sustainability. Am Health Drug Benefits.
perceptions of a health information
2012;5(6):333-40. PMID: 24991331.
exchange: a pilot program in South Korea.
54. Vest JR. Health information exchange and Int J Med Inf. 2013;82(2):98-107. PMID:
healthcare utilization. J Med Syst. 22658777.
2009;33(3):223-31. PMID: 19408456.
64. Vest JR, Miller TR. The association between
55. Vest JR, Kern LM, Silver MD, et al. The health information exchange and measures
potential for community-based health of patient satisfaction. Appl Clin Inform.
information exchange systems to reduce 2011;2(4):447-59. PMID: 23616887.
hospital readmissions. J Am Med Inform
65. Lang E, Afilalo M, Vandal AC, et al. Impact
Assoc. 2014 PMID: 25100447.
of an electronic link between the emergency
56. Vest JR, Kern LM, Campion TR, Jr., et al. department and family physicians: a
Association between use of a health randomized controlled trial. CMAJ.
information exchange system and hospital 2006;174(3):313-8. PMID: 16399880.
admissions. Appl Clin Inform.
66. Afilalo M, Lang E, Léger R, et al. Impact of
2014;5(1):219-31. PMID: 24734135.
a standardized communication system on
continuity of care between family physicians
and the emergency department. CJEM.
2007;9(2):79-86. PMID: 17391577.
92
67. Willis JM, Edwards R, Anstrom KJ, et al. 76. Abramson EL, McGinnis S, Edwards A, et
Decision support for evidence-based al. Electronic health record adoption and
pharmacotherapy detects adherence health information exchange among
problems but does not impact medication hospitals in New York State. J Eval Clin
use. Stud Health Technol Inform. Pract. 2012;18(6):1156-62. PMID:
2013;183:116-25. PMID: 23388267. 21914089.
68. Jones SS, Friedberg MW, Schneider EC. 77. Abramson EL, McGinnis S, Moore J, et al.
Health information exchange, Health A statewide assessment of electronic health
Information Technology use, and hospital record adoption and health information
readmission rates. AMIA Annu Symp Proc. exchange among nursing homes. Health
2011;2011:644-53. PMID: 22195120. Serv Res. 2014;49(1 Pt 2):361-72. PMID:
24359612.
69. Lammers EJ, Adler-Milstein J, Kocher KE.
Does health information exchange reduce 78. Adler-Milstein J, Bates DW, Jha AK. U.S.
redundant imaging? Evidence from Regional health information organizations:
emergency departments. Med Care. progress and challenges. Health Aff.
2014;52(3):227-34. PMID: 24374414. 2009;28(2):483-92. PMID: 19276008.
70. Carr CM, Gilman CS, Krywko DM, et al. 79. Adler-Milstein J, Bates DW, Jha AK. A
Observational study and estimate of cost survey of health information exchange
savings from use of a health information organizations in the United States:
exchange in an academic emergency implications for meaningful use. Ann Intern
department. J Emerg Med. 2014;46(2):250- Med. 2011;154(10):666-71. PMID:
6. PMID: 24071033. 21576534.
71. Winden TJ, Boland LL, Frey NG, et al. Care 80. Adler-Milstein J, Landefeld J, Jha AK.
everywhere, a point-to-point HIE tool: Characteristics associated with regional
utilization and impact on patient care in the health information organization viability. J
ED. Appl Clin Inform. 2014;5(2):388-401. Am Med Inform Assoc. 2010;17(1):61-5.
PMID: 25024756. PMID: 20064803.
72. Ben-Assuli O, Shabtai I, Leshno M. Using 81. Adler-Milstein J, McAfee AP, Bates DW, et
electronic health record systems to optimize al. The state of regional health information
admission decisions: The Creatinine case organizations: current activities and
study. Health Informatics J. 2015;21(1):73- financing. Health Aff. 2008;27(1):w60-9.
88. PMID: 24692078. PMID: 18073225.
73. eHealth Initiative. Result from Survey on 82. Bouhaddou O, Bennett J, Cromwell T, et al.
Health Data Exchange 2013. The Challenge The Department of Veterans Affairs,
to Connect. Available at: https://ehi-rails- Department of Defense, and Kaiser
app.s3.amazonaws.com/uploads/article/file/ Permanente Nationwide Health Information
3/eHIResultsFromSurveyonHealthDataExch Network exchange in San Diego: patient
ange2013.pdf. Accessed June 17, 2015. selection, consent, and identity matching.
74. eHealth Initiative. Post HITECH: The
PMID: 22195064.
Landscape of Health Information Exchange.
Available at: 83. Dixon BE, Jones JF, Grannis SJ. Infection
http://visit.medicity.com/rs/aetnainc/images/ preventionists' awareness of and
2014%20eHI%20Data%20Exchange%20Su engagement in health information exchange
rvey%20Key%20Findings.pdf. Accessed to improve public health surveillance. Am J
June 17, 2015. Infect Control. 2013;41(9):787-92. PMID:
23415767.
75. Carr CM, Saef SH, Zhao J, et al. Can data
from a health information exchange be used 84. Foldy S. Inventory of electronic health
to describe patients who visit multiple information exchange in Wisconsin, 2006.
emergency departments within a region? WMJ. 2007;106(3):120-5. PMID:
Acad Emerg Med. 2014;21(5):S141. 17642349.
93
85. Fontaine P, Zink T, Boyle RG, et al. Health 95. Schoen C, Osborn R, Squires D, et al. A
information exchange: participation by survey of primary care doctors in ten
Minnesota primary care practices. Arch countries shows progress in use of health
Intern Med. 2010;170(7):622-9. PMID: information technology, less in other areas.
20386006. Health Aff (Millwood). 2012;31(12):2805-
16. PMID: 23154997.
86. Gadd CS, Ho Y-X, Cala CM, et al. User
perspectives on the usability of a regional 96. Abramson EL, Silver M, Kaushal R.
health information exchange. J Am Med Meaningful use status and participation in
Inform Assoc. 2011;18(5):711-6. PMID: health information exchange among New
21622933. York State hospitals: A longitudinal
assessment. Jt Comm J Qual Patient Saf.
87. Hessler BJ, Soper P, Bondy J, et al.
2014;40(10)
Assessing the relationship between health
information exchanges and public health 97. Campion TR, Jr., Edwards AM, Johnson
agencies. J Public Health Manag Pract. SB, et al. Health information exchange
2009;15(5):416-24. PMID: 19704310. system usage patterns in three communities:
practice sites, users, patients, and data. Int J
88. Kho AN, Doebbeling BN, Cashy JP, et al. A
Med Inf. 2013;82(9):810-20. PMID:
regional informatics platform for
23743323.
coordinated antibiotic-resistant infection
tracking, alerting, and prevention. Clin 98. Campion TR, Jr., Vest JR, Ancker JS, et al.
Infect Dis. 2013;57(2):254-62. PMID: Patient encounters and care transitions in
23575195. one community supported by automated
query-based health information exchange.
89. Lee S-I, Park H, Kim J-W, et al. Physicians'
perceptions and use of a health information
PMID: 24551330.
exchange: a pilot program in South Korea.
Telemed J E Health. 2012;18(8):604-12. 99. Johnson KB, Gadd CS, Aronsky D, et al.
PMID: 22352898. The MidSouth eHealth Alliance: use and
impact in the first year. AMIA Annu Symp
90. Soderberg K, Laventure M. Minnesota
Proc. 2008:333-7. PMID: 18999184.
clinics' adoption, use and exchange of
electronic health information. Minn Med. 100. Lobach DF, Kawamoto K, Anstrom KJ, et
2013;96(9):45-8. PMID: 24494362. al. Proactive population health management
in the context of a regional health
91. Patel V, Swain MJ, King J, et al. Physician
information exchange using standards-based
capability to electronically exchange clinical
decision support. AMIA Annu Symp Proc.
information, 2011. Am J Manag Care.
2007:473-7. PMID: 18693881.
2013;19(10):835-43. PMID: 24304162.
101. Vest JR, Gamm LD, Ohsfeldt RL, et al.
92. Anand V, Sheley ME, Xu S, et al. Real time
Factors associated with health information
alert system: a disease management system
exchange system usage in a safety-net
leveraging health information exchange.
ambulatory care clinic setting. J Med Syst.
Online J Public Health Inform. 2012;4(3)
2012;36(4):2455-61. PMID: 21523428.
PMID: 23569648.
102. Vest JR, Grinspan ZM, Kern LM, et al.
93. Caffrey C, Park-Lee E. Use of electronic
Using a health information exchange system
health records in residential care
for imaging information: patterns and
communities. NCHS data brief.
predictors. AMIA Annu Symp Proc.
2013(128):1-8. PMID: 24152578.
2013;2013:1402-11. PMID: 24551416.
94. Codagnone C, Lupiañez-Villanueva F.
103. Vest JR, Jasperson JS. How are health
Benchmarking Deployment of eHealth
professionals using health information
among General Practitioners (2013) - Final
exchange systems? Measuring usage for
Report. Lukembourg: European
evaluation and system improvement. J Med
Commission.
Syst. 2012;36(5):3195-204. PMID:
22127521.
94
104. Vest JR, Jasperson JS, Zhao H, et al. Use of 114. Silvester BV, Carr SJ. A shared electronic
a health information exchange system in the health record: lessons from the coalface.
emergency care of children. BMC Med Med J Aust. 2009;190(11 Suppl):S113-6.
Inform Decis Mak. 2011;11:78. PMID: PMID: 19485857.
22208182.
105. Vest JR, Zhao H, Jaspserson J, et al. Factors The utilization rate of the regional health
motivating and affecting health information information exchange: how it impacts on
exchange usage. J Am Med Inform Assoc. health care delivery outcomes. J Public
2011;18(2):143-9. PMID: 21262919. Health Manag Pract. 2012;18(3):215-23.
PMID: 22473113.
106. Moore T, Shapiro JS, Doles L, et al. Event
detection: a clinical notification service on a 116. Byrne CM, Mercincavage LM, Bouhaddou
health information exchange platform. O, et al. The Department of Veterans
AMIA Annu Symp Proc. 2012;2012:635-42. Affairs' (VA) implementation of the Virtual
PMID: 23304336. Lifetime Electronic Record (VLER):
Findings and lessons learned from Health
107. Adler-Milstein J, DesRoches CM, Jha AK.
Information Exchange at 12 sites. Int J Med
Health information exchange among US
Inf. 2014;83(8):537-47. PMID: 24845146.
hospitals. Am J Manag Care.
2011;17(11):761-8. PMID: 22084896. 117. Jha AK, Doolan D, Grandt D, et al. The use
of health information technology in seven
108. Adler-Milstein J, Jha AK. Health
nations. Int J Med Inf. 2008;77(12):848-54.
information exchange among U.S. hospitals:
PMID: 18657471.
Who's in, who's out, and why? Healthcare.
2014;2(1):26-32. 118. Johnson KB, Unertl KM, Chen Q, et al.
Health information exchange usage in
109. Audet A-M, Squires D, Doty MM. Where
emergency departments and clinics: the
are we on the diffusion curve? Trends and
who, what, and why. J Am Med Inform
drivers of primary care physicians' use of
Assoc. 2011;18(5):690-7. PMID: 21846788.
health information technology. Health Serv
Res. 2014;49(1 Pt 2):347-60. PMID: 119. Unertl KM, Johnson KB, Lorenzi NM.
24358958. Health information exchange technology on
the front lines of healthcare: workflow
110. Furukawa MF, King J, Patel V, et al.
factors and patterns of use. J Am Med
Despite Substantial Progress In EHR
Inform Assoc. 2012;19(3):392-400. PMID:
Adoption, Health Information Exchange
22003156.
And Patient Engagement Remain Low In
Office Settings. Health Aff. 2014:1-8. 120. Kaelber DC, Waheed R, Einstadter D, et al.
PMID: 25104827. Use and perceived value of health
information exchange: one public healthcare
111. Furukawa MF, Patel V, Charles D, et al.
system's experience. Am J Manag Care.
Hospital Electronic Health Information
2013;19(10 Spec No):Sp337-43. PMID:
Exchange Grew Substantially In 2008-12.
24511888.
Health Aff. 2013;32(8):1346-54. PMID:
23918477. 121. Greenhalgh T, Stramer K, Bratan T, et al.
Adoption and non-adoption of a shared
112. Gutteridge DL, Genes N, Hwang U, et al.
electronic summary record in England: a
Enhancing a Geriatric Emergency
mixed-method case study. BMJ.
Department Care Coordination Intervention
2010;340:c3111. PMID: 20554687.
Using Automated Health Information
Exchange-Based Clinical Event 122. Pagliari C, Gilmour M, Sullivan F.
Notifications. EGEMS (Wash DC). Electronic Clinical Communications
2014;2(3) Implementation (ECCI) in Scotland: a
mixed-methods programme evaluation. J
113. Hamann DJ, Bezboruah KC. Utilization of
Eval Clin Pract. 2004;10(1):11-20. PMID:
technology by long-term care providers:
14731147.
comparisons between for-profit and
nonprofit institutions. J Aging Health.
2013;25(4):535-54. PMID: 23509114.
95
123. Tripathi M, Delano D, Lund B, et al. 132. Hincapie AL, Warholak TL, Murcko AC, et
Engaging patients for health information al. Physicians' opinions of a health
exchange. Health Aff. 2009;28(2):435-43. information exchange. J Am Med Inform
PMID: 19276000. Assoc. 2011;18(1):60-5. PMID: 21106994.
124. Herwehe J, Wilbright W, Abrams A, et al. 133. Hyppönen H, Reponen J, Lääveri T, et al.
Implementation of an innovative, integrated User experiences with different regional
electronic medical record (EMR) and public health information exchange systems in
health information exchange for HIV/AIDS. Finland. Int J Med Inf. 2014;83(1):1-18.
J Am Med Inform Assoc. 2012;19(3):448- PMID: 24200753.
52. PMID: 22037891.
134. Massy-Westropp M, Giles LC, Law D, et al.
125. Onyile A, Vaidya SR, Kuperman G, et al. Connecting hospital and community care:
Geographical distribution of patients visiting the acceptability of a regional data linkage
a health information exchange in New York scheme. Aust Health Rev. 2005;29(1):12-6.
City. J Am Med Inform Assoc. PMID: 15683350.
2013;20(e1):e125-30. PMID: 23104049.
135. McCullough JM, Zimmerman FJ, Bell DS,
126. Bailey JE, Wan J, Pope R, et al. Health et al. Electronic health information exchange
information exchange use reduces avoidable in underserved settings: examining
diagnostic imaging in the emergency initiatives in small physician practices &
evaluation of back pain. J Gen Intern Med. community health centers. BMC Health
2011;26(University of Tennessee Health Serv Res. 2014;14:415. PMID: 25240718.
Science Center, Memphis, United
136. Rudin R, Volk L, Simon S, et al. What
States):S349-S50.
Affects Clinicians' Usage of Health
127. Kierkegaard P, Kaushal R, Vest JR. How Information Exchange? Appl Clin Inform.
could health information exchange better 2011;2(3):250-62. PMID: 22180762.
meet the needs of care practitioners? Appl
137. Yeager VA, Walker D, Cole E, et al. Factors
Clin Inform. 2014;5(4):861-77.
Related to Health Information Exchange
128. Myers JJ, Koester KA, Chakravarty D, et al. Participation and Use. J Med Syst.
Perceptions regarding the ease of use and 2014;38(8) PMID: 24957395.
usefulness of health information exchange
138. Messer LC, Parnell H, Huffaker R, et al. The
systems among medical providers, case
development of a health information
managers and non-clinical staff members
exchange to enhance care and improve
working in HIV care and community
patient outcomes among HIV+ individuals
settings. Int J Med Inf. 2012;81(10):e21-9.
in rural North Carolina. Int J Med Inf.
PMID: 22854159.
2012;81(10):e46-55. PMID: 22898321.
129. Ozkaynak M, Brennan PF. Revisiting
139. Nohr C, Kristensen M, Andersen SK, et al.
sociotechnical systems in a case of
Shared experience in 13 local Danish EPR
unreported use of health information
projects: the Danish EPR Observatory. Stud
exchange system in three hospital
Health Technol Inform. 2001;84(Pt 1):670-
emergency departments. J Eval Clin Pract.
4. PMID: 11604822.
2013;19(2):370-3. PMID: 22420774.
140. Adjerid I, Padman R. Impact of health
130. Thorn SA, Carter MA, Bailey JE.
disclosure laws on health information
Emergency physicians' perspectives on their
exchanges. AMIA Annu Symp Proc.
use of health information exchange. Ann
2011;2011:48-56. PMID: 22195054.
Emerg Med. 2014;63(3):329-37. PMID:
24161840. 141. Dixon B, Miller T, Overhage M. Barriers to
achieving the last mile in health information
131. Finnell JT, Overhage JM. Emergency
exchange: a survey of small hospitals and
medical services: the frontier in health
physician practices. J Healthc Inf Manag.
information exchange. AMIA Annu Symp
2013;27(4):55-8.
Proc. 2010;2010:222-6. PMID: 21346973.
96
142. Dobalian A, Claver ML, Pevnick JM, et al. 151. Overhage JM, Evans L, Marchibroda J.
Organizational challenges in developing one Communities' readiness for health
of the Nationwide Health Information information exchange: the National
Network trial implementation awardees. J Landscape in 2004. J Am Med Inform
Med Syst. 2012;36(2):933-40. PMID: Assoc. 2005;12(2):107-12. PMID:
20703640. 15561785.
143. Fairbrother G, Trudnak T, Christopher R, et 152. Pirnejad H, Bal R, Berg M. Building an
al. Cincinnati Beacon Community Program inter-organizational communication network
Highlights Challenges And Opportunities and challenges for preserving
On The Path To Care Transformation. interoperability. Int J Med Inf.
Health Aff. 2014;33(5):871-7. PMID: 2008;77(12):818-27. PMID: 18579436.
24799586.
153. Rudin RS, Simon SR, Volk LA, et al.
144. Feldman SS, Schooley LB, Bhavsar PG. Understanding the decisions and values of
Health Information Exchange stakeholders in health information
Implementation: Lessons Learned and exchanges: experiences from Massachusetts.
Critical Success Factors From a Case Study. Am J Public Health. 2009;99(5):950-5.
JMIR Med Inform. 2014;2(2):e19. PMID: 19299671.
145. Genes N, Shapiro J, Vaidya S, et al. 154. Saff E, Lanway C, Chenyek A, et al. The
Adoption of health information exchange by Bay Area HIE. A case study in connecting
emergency physicians at three urban stakeholders. J Healthc Inf Manag.
academic medical centers. Appl Clin 2010;24(1):25-30. PMID: 20077922.
Inform. 2011;2(3):263-9. PMID: 23616875.
155. Vest JR. More than just a question of
146. Goldwater J, Jardim J, Khan T, et al. technology: factors related to hospitals'
Emphasizing Public Health Within a Health adoption and implementation of health
Information Exchange: An Evaluation of the information exchange. Int J Med Inf.
District of Columbia’s Health Information 2010;79(12):797-806. PMID: 20889370.
Exchange Program. EGEMS (Wash DC).
156. Vest JR, Campion Jr TR, Kaushal R.
2014;2(3)
Challenges, Alternatives, and Paths to
147. Kern LM, Wilcox A, Shapiro J, et al. Which Sustainability for Health Information
components of health information Exchange Efforts. J Med Syst. 2013;37(6):1-
technology will drive financial value? Am J 8. PMID: 24141531.
Manag Care. 2012;18(8):438-45. PMID:
157. Vest JR, Issel LM. Factors related to public
22928759.
health data sharing between local and state
148. McGowan JJ, Jordan C, Sims T, et al. Rural health departments. Health Serv Res.
RHIOs: common issues in the development 2014;49(1 Pt 2):373-91. PMID: 24359636.
of two state-wide health information
158. Dullabh P, Hovey L. Large Scale Health
networks. AMIA Annu Symp Proc.
Information Exchange: Implementation
2007:528-32. PMID: 18693892.
Experiences from Five States. Stud Health
149. Miller AR, Tucker C. Health information Technol Inform. 2013;192:613-7. PMID:
exchange, system size and information silos. 23920629.
J Health Econ. 2014;33:28-42. PMID:
159. Dullabh P, Ubri P, Hovey L. The State HIE
24246484.
Program Four Years Later: Key Findings on
150. Nykänen P, Karimaa E. Success and failure Grantees’ Experiences from a Six-State
factors in the regional health information Review. Office of the National Coordinator
system design process--results from a for Health Information Technology. 2014.
constructive evaluation study. Methods Inf Available at: www.healthit.gov/sites/
Med. 2006;45(1):85-9. PMID: 16482376. default/files/CaseStudySynthesisGranteeExp
erienceFinal_121014.pdf. Accessed April
22, 2015.
97
160. Grossman JM, Kushner KL, November EA. 169. Steward WT, Koester KA, Collins SP, et al.
Creating sustainable local health information The essential role of reconfiguration
exchanges: can barriers to stakeholder capabilities in the implementation of HIV-
participation be overcome? Res Briefs. related health information exchanges. Int J
2008(2):1-12. PMID: 18496926. Med Inf. 2012;81(10):e10-20. PMID:
22841703.
161. McCarthy DB, Propp K, Cohen A, et al.
Learning from Health Information Exchange 170. Unertl MK, Johnson BK, Gadd SC, et al.
Technical Architecture and Implementation Bridging Organizational Divides in Health
in Seven Beacon Communities. EGEMS Care: An Ecological View of Health
(Wash DC). 2014;2(1):6. Information Exchange. JMIR Med Inform.
2013;1(1):e3.
162. Miller RH. Satisfying patient-consumer
principles for health information exchange: 171. Kern LM, Wilcox AB, Shapiro J, et al.
evidence from California case studies. Community-based health information
Health Aff. 2012;31(3):537-47. PMID: technology alliances: potential predictors of
22392664. early sustainability. Am J Manag Care.
2011;17(4):290-5. PMID: 21615199.
163. Morris G, Afzal S, Bhasker M, et al. Query-
Based Exchange: Key Factors Influencing 172. Schabetsberger T, Ammenwerth E,
Success and Failure. Office of the National Andreatta S, et al. From a paper-based
Coordinator for Health Information transmission of discharge summaries to
Technology. 2012. Available at: electronic communication in health care
www.healthit.gov/sites/default/files/query_b regions. Int J Med Inf. 2006;75(3-4):209-15.
ased_exchange_final.pdf. Accessed April PMID: 16112892.
10, 2015.
173. Kern LM, Barrón Y, Abramson EL, et al.
164. Phillips AB, Wilson RV, Kaushal R, et al. HEAL NY: Promoting interoperable health
Implementing health information exchange information technology in New York State.
for public health reporting: a comparison of Health Aff. 2009;28(2):493-504. PMID:
decision and risk management of three 19276009.
regional health information organizations in
174. Merrill JA, Deegan M, Wilson RV, et al. A
New York state. J Am Med Inform Assoc.
system dynamics evaluation model:
2014;21(e1):e173-7. PMID: 23975626.
implementation of health information
165. Pouloudi A. Information technology for exchange for public health reporting. J Am
collaborative advantage in healthcare Med Inform Assoc. 2013;20(e1):e131-8.
revisited. Information & Management. PMID: 23292910.
1999;35(6):345-56.
175. Becker L, Oxman A. Chapter 22: Overviews
166. Agency for Healthcare Research and of reviews. In: Higgins J, Green S, eds.
Quality. Evolution of State Health Cochrane Handbook for Systematic Reviews
Information Exchange/A Study of Vision, of Interventions Version 5.1.0 (updated
Strategy, and Progress. Available at: March 2011): The Cochrane Collaboration;
http://www.avalerehealth.net/research/docs/ 2011.
State_based_Health_Information_Exchange
176. Roshanov PS, Fernandes N, Wilczynski JM,
_Final_Report.pdf. Accessed November 20,
et al. Features of effective computerised
2014.
clinical decision support systems: meta-
167. Ross SE, Schilling LM, Fernald DH, et al. regression of 162 randomised trials. BMJ.
Health information exchange in small-to- 2013;346:f657. PMID: 23412440.
medium sized family medicine practices:
177. Thomson D, Russell K, Becker L, et al. The
motivators, barriers, and potential
evolution of a new publication type: Steps
facilitators of adoption. Int J Med Inf.
and challenges of producing overviews of
2010;79(2):123-9. PMID: 20061182.
reviews. Research Synthesis Methods.
168. Sicotte C, Paré G. Success in health 2010;1:198-211.
information exchange projects: solving the
implementation puzzle. Soc Sci Med.
2010;70(8):1159-65. PMID: 20137847.
98
178. White C, Ip S, McPheeters M. Using 186. Rasmussen J. Risk Management in a
existing systematic reviews to replace de Dynamic Society: A Modelling Problem.
novo processes in conducting Comparative Safety Science. 1997;27(2/3):183-213.
Effectiveness Reviews. Methods Guide for
187. Michie S, Richardson M, Johnston M, et al.
Comparative Effectiveness Reviews.
The behavior change technique taxonomy
(v1) of 93 hierarchically clustered
Research and Quality; 2009.
techniques: building an international
179. Michie S, Fixsen D, Grimshaw JM, et al. consensus for the reporting of behavior
Specifying and reporting complex behaviour change interventions. Annals of behavioral
change interventions: the need for a medicine : a publication of the Society of
scientific method. Implement Sci. Behavioral Medicine. 2013;46(1):81-95.
2009;4:40. PMID: 19607700. PMID: 23512568.
180. Petitti DB. Hormone replacement therapy 188. Schulz KF, Altman DG, Moher D.
and heart disease prevention: CONSORT 2010 statement: updated
experimentation trumps observation. JAMA. guidelines for reporting parallel group
1998;280(7):650-2. PMID: 9718060. randomized trials. Ann Intern Med.
2010;152(11):726-32. PMID: 20335313.
181. Sackett DL, Wennberg JE. Choosing the
best research design for each question. BMJ. 189. Pearce N, Merletti F. Complexity,
1997;315(7123):1636. PMID: 9448521. simplicity, and epidemiology. Int J
Epidemiol. 2006;35(3):515-9. PMID:
182. Craig P, Dieppe P, Macintyre S, et al.
16415326.
Developing and evaluating complex
interventions: the new Medical Research 190. Galea S, Riddle M, Kaplan GA. Causal
Council guidance. BMJ. 2008;337:a1655. thinking and complex system approaches in
PMID: 18824488. epidemiology. Int J Epidemiol.
2010;39(1):97-106. PMID: 19820105.
183. Rasmussen J. Risk management in a
dynamic society: a modelling problem. 191. Fleurence RL, Curtis LH, Califf RM, et al.
Safety science. 1997;27(2):183-213. Launching PCORnet, a national patient-
centered clinical research network. J Am
184. Vicente KJ. What does it take? A case study
Med Inform Assoc. 2014:amiajnl-2014-
of radical change toward patient safety. Joint
002747. PMID: 24821743.
Commission journal on quality and safety.
2003;29(11):598-609. PMID: 14619352.
185. Leveson N. Engineering a Safer World.
Systems Thinking Applied to Safety.
Cambridge: MIT Press; 2012.
99
Abbreviations and Acronyms
AHA American Hospital Association
AHRQ Agency for Healthcare Research and Quality
CHIC Carolina HIV Information Cooperative
CI confidence interval
CONSORT Consolidated Standards of Reporting Trials
CT computed tomography
DoD Department of Defense
ED emergency department
eGEMS Generating Evidence and Methods to improve patient outcomes
EHR electronic health record
EPC Evidence-based Practice Center
HEAL-NY Health Care Efficiency and Affordability Law for New Yorkers Capital
Grant Program
HIE health information exchange
HITECH Health Information Technology for Economic and Clinical Health
I-Care Central Texas HIE
IEEE Institute of Electrical and Electronics Engineers
IT information technology
K Thousand
MSeHA MidSouth e-Health Alliance
NAMCS National Ambulatory Medical Care Survey
NwHIN Nationwide Health Information Network
ONC The Office of the National Coordinator for Health Information
Technology
OR odds ratio
PICOTS populations, interventions, comparators, outcomes, timing, types of
studies, and setting
QUIS Questionnaire for User Interface Satisfaction
RCT randomized controlled trials
RHIO regional health information organization
STAMP Systems—Theoretic Accident Modeling and Processes
TEP Technical Expert Panel
VA Veterans Affairs
VLER Virtual Lifetime Electronic Record
100
Appendix A. Search Strategies
Database: Ovid MEDLINE® and Ovid OLDMEDLINE® <1990 to

February 2015> Search Strategy
1 (health information adj5 exchang$).mp.
2 hie.mp.
3 exp Medical Records/
4 exp Systems Analysis/
5 exp Medical Informatics/
6 Information Dissemination/
7 3 or 4 or 5 or 6
8 2 and 7
9 1 or 8
10 health information organization$.mp.
11 7 and 10
12 (hio or hios or rhio or rhios).mp.
13 7 and 12
14 ((clinical$ or health$) adj5 (data adj3 exchang$)).mp.
15 7 and 14
16 (patient$ adj2 match$).mp.
17 7 and 16
18 ((query or querie$) adj3 (base or based or bases or basing) adj5 exchang$).mp.
19 7 and 18
20 directed exchang$.mp.
21 7 and 20
22 ((consumer$ or patient$) adj5 mediat$ adj7 exchang$).mp.
23 7 and 22
24 ((health information adj5 tech$) and exchang$).mp.
25 7 and 24
26 (health information adj7 network$).mp.
27 7 and 26
28 ((health information or ((electronic$ or computer$) adj2 (health or medic$ or patient$) adj2
record$) or ehr or emr) adj7 exchang$).mp.
29 7 and 28
30 (exchang$ adj5 network$).mp.
31 7 and 30 (116)
32 (interoperab$ adj7 standard$).mp. (320)
33 7 and 32
34 ((inter or between or across) adj3 (organization$ or systems) adj7 network$).mp.
35 7 and 34
36 9 or 11 or 13 or 15 or 17 or 19 or 21 or 23 or 25 or 27 or 29 or 31 or 33 or 35
37 Medical Record Linkage/
38 exp systems integration/
39 37 and 38
40 exp Cooperative Behavior/
A-1
41 37 and 40
42 exp Medical Informatics Applications/
43 37 and 42
44 10 or 12 or 14 or 16 or 18 or 20 or 22 or 24 or 26 or 28 or 30 or 32
45 43 and 44
46 36 or 39 or 41 or 45
47 6 and 38 and 42
48 6 and 38 and 40
49 4 and 37 and 40
50 4 and 37 and 42
51 6 and 37 and 42
52 6 and 37 and 40
53 4 and 38 and 40
54 46 or 47 or 48 or 49 or 50 or 51 or 52 or 53
55 limit 54 to english language
Database: PsycINFO <1990 to February 2015> Search

Strategy
1 ((healthcare information or health information) adj5 exchang$).mp. [mp=title, abstract,
heading word, table of contents, key concepts, original title, tests & measures]
2 exp medical records/
3 exp information systems/
4 exp Information Dissemination/
5 exp systems analysis/
6 exp information technology/
7 exp computer mediated communication/
8 2 or 3 or 4 or 5 or 6 or 7
9 hie.mp.
10 8 and 9
11 1 or 10
16 8 and 15
24 8 and 23
25 (interoperab$ adj7 standard$).mp.
26 11 or 12 or 14 or 16 or 17 or 18 or 19 or 20 or 21 or 22 or 24 or 25
A-2
Databases: EBM Reviews - Cochrane Database of Systematic
Reviews, Cochrane Central Register of Controlled Trials,
Database of Abstracts of Reviews of Effects, NHS Economic
Evaluation Database <1990 to January 2015>
Search Strategy
1 (health information adj5 exchang$).mp.
2 hie.mp.
3 ((health or medical) adj3 (record or records)).mp. [mp=title, text, subject heading word]
4 ((System or systems) adj3 Analysis).mp. [mp=title, text, subject heading word]
5 ((health$ or medic$) adj5 informatic$).mp. [mp=title, text, subject heading word]
6 ((informat$ or data) adj5 (link$ or disseminat$ or transfer$ or request$ or share$ or
sharing)).mp. [mp=title, text, subject heading word]
7 3 or 4 or 5 or 6
8 2 and 7
9 1 or 8
11 7 and 10
13 7 and 12
15 7 and 14
17 7 and 16
19 7 and 18
21 7 and 20
23 7 and 22
25 7 and 24
27 7 and 26
29 7 and 28
31 7 and 30
32 (interoperab$ adj7 standard$).mp.
33 7 and 32
34 ((inter or between or across) adj3 (organization$ or systems) adj7 network$).mp.
35 7 and 34
36 ((health$ or medic$) adj3 record adj7 (link$ or disseminat$ or transfer$ or request$ or
share$ or sharing)).mp. [mp=title, text, subject heading word]
37 9 or 11 or 13 or 15 or 17 or 19 or 21 or 23 or 25 or 27 or 29 or 31 or 33 or 35 or 36
A-3
Appendix B. Inclusion and Exclusion Criteria
Table B1. Inclusion and exclusion criteria
Include Exclude
Population All KQs: Any individual or group of health care providers, patients, managers, health care institutions, All KQs: Not applicable to a U.S.
or regional organizations. population.
Interventions All KQs: Heath Information Exchange . HIE is defined as the electronic sharing of clinical information All KQs: Hypothetical HIEs, HIE within an
among users such as health care providers, patients, administrators or policy makers across the organization/single setting, independent
boundaries of health care institutions, health data repositories, States and others, typically not within a electronic prescription or referral system,
single organization or among affiliated providers, while protecting the integrity, privacy, and security of a single person accessing multiple
the information. systems, registries, HIE for research,
marketing or administration, non-
electronic transfers.
Comparators KQ 1-3: Time period prior to HIE implementations, geographic or organizational locations without HIE, KQ 1-3: No comparator
situations in which HIE is not available, multiple types of HIE, characteristics of the different settings
and systems in which HIE is used.
KQ 4-8: No comparison required KQ 4-8: None
B-1
Appendix C. List of Included Studies
Abramson EL, McGinnis S, Edwards A, et al. Electronic health record adoption and health
information exchange among hospitals in New York State. J Eval Clin Pract. 2012;18(6):1156-
62. PMID: 21914089.
Abramson EL, McGinnis S, Moore J, et al. A statewide assessment of electronic health record
adoption and health information exchange among nursing homes. Health Serv Res. 2014;49(1 Pt
2):361-72. PMID: 24359612.
Abramson EL, Silver M, Kaushal R. Meaningful use status and participation in health
information exchange among New York State hospitals: A longitudinal assessment. Jt Comm J
Qual Patient Saf. 2014;40(10)
Adjerid I, Padman R. Impact of health disclosure laws on health information exchanges. AMIA
Annu Symp Proc. 2011;2011:48-56. PMID: 22195054.
Adler-Milstein J, Bates DW, Jha AK. U.S. Regional health information organizations: progress
and challenges. Health Aff. 2009;28(2):483-92. PMID: 19276008.
Adler-Milstein J, Bates DW, Jha AK. A survey of health information exchange organizations in
the United States: implications for meaningful use. Ann Intern Med. 2011;154(10):666-71.
PMID: 21576534.
Adler-Milstein J, Bates DW, Jha AK. Operational health information exchanges show substantial
growth, but long-term funding remains a concern. Health Aff (Millwood). 2013;32(8):1486-92.
PMID: 23840051.
Adler-Milstein J, DesRoches CM, Jha AK. Health information exchange among US hospitals.
Am J Manag Care. 2011;17(11):761-8. PMID: 22084896.
Adler-Milstein J, Jha AK. Health information exchange among U.S. hospitals: Who's in, who's
out, and why? Healthcare. 2014;2(1):26-32.
Adler-Milstein J, Landefeld J, Jha AK. Characteristics associated with regional health

information organization viability. J Am Med Inform Assoc. 2010;17(1):61-5. PMID: 20064803.
Adler-Milstein J, McAfee AP, Bates DW, et al. The state of regional health information
organizations: current activities and financing. Health Aff. 2008;27(1):w60-9. PMID: 18073225.
Afilalo M, Lang E, Léger R, et al. Impact of a standardized communication system on continuity

of care between family physicians and the emergency department. CJEM. 2007;9(2):79-86.
PMID: 17391577.
Agency for Healthcare Research and Quality. Evolution of State Health Information Exchange/A
Study of Vision, Strategy, and Progress. Available at:
C-1
http://www.avalerehealth.net/research/docs/State_based_Health_Information_Exchange_Final_R
eport.pdf. Accessed November 20, 2014.
Altman R, Shapiro JS, Moore T, et al. Notifications of hospital events to outpatient clinicians
using health information exchange: a post-implementation survey. Inform Prim Care.
2012;20(4):249-55. PMID: 23890336.
Anand V, Sheley ME, Xu S, et al. Real time alert system: a disease management system
leveraging health information exchange. Online J Public Health Inform. 2012;4(3) PMID:
23569648.
Audet A-M, Squires D, Doty MM. Where are we on the diffusion curve? Trends and drivers of
primary care physicians' use of health information technology. Health Serv Res. 2014;49(1 Pt
2):347-60. PMID: 24358958.
Bailey JE, Pope RA, Elliott EC, et al. Health information exchange reduces repeated diagnostic
imaging for back pain. Ann Emerg Med. 2013;62(1):16-24. PMID: 23465552.
Bailey JE, Wan JY, Mabry LM, et al. Does health information exchange reduce unnecessary
neuroimaging and improve quality of headache care in the emergency department? J Gen Intern
Med. 2013;28(2):176-83. PMID: 22648609.
Ben-Assuli O, Shabtai I, Leshno M. The impact of EHR and HIE on reducing avoidable
admissions: controlling main differential diagnoses. BMC Med Inform Decis Mak. 2013;13:49.
PMID: 23594488.
Ben-Assuli O, Shabtai I, Leshno M. Using electronic health record systems to optimize

admission decisions: The Creatinine case study. Health Informatics J. 2015;21(1):73-88. PMID:
24692078.
Bouhaddou O, Bennett J, Cromwell T, et al. The Department of Veterans Affairs, Department of

Defense, and Kaiser Permanente Nationwide Health Information Network exchange in San
Diego: patient selection, consent, and identity matching. AMIA Annu Symp Proc.
2011;2011:135-43. PMID: 22195064.
Byrne CM, Mercincavage LM, Bouhaddou O, et al. The Department of Veterans Affairs' (VA)
implementation of the Virtual Lifetime Electronic Record (VLER): Findings and lessons learned
from Health Information Exchange at 12 sites. Int J Med Inf. 2014;83(8):537-47. PMID:
24845146.
Caffrey C, Park-Lee E. Use of electronic health records in residential care communities. NCHS
data brief. 2013(128):1-8. PMID: 24152578.
Campion TR, Jr., Ancker JS, Edwards AM, et al. Push and pull: physician usage of and
satisfaction with health information exchange. AMIA Annu Symp Proc. 2012;2012:77-84.
PMID: 23304275.
C-2
Campion TR, Jr., Edwards AM, Johnson SB, et al. Health information exchange system usage
patterns in three communities: practice sites, users, patients, and data. Int J Med Inf.
2013;82(9):810-20. PMID: 23743323.
Campion TR, Jr., Vest JR, Ancker JS, et al. Patient encounters and care transitions in one
community supported by automated query-based health information exchange. AMIA Annu
Symp Proc. 2013;2013:175-84. PMID: 24551330.
Carr CM, Gilman CS, Krywko DM, et al. Observational study and estimate of cost savings from
use of a health information exchange in an academic emergency department. J Emerg Med.
2014;46(2):250-6. PMID: 24071033.
Chang KC, Overhage JM, Hui SL, et al. Enhancing laboratory report contents to improve
outpatient management of test results. J Am Med Inform Assoc. 2010;17(1):99-103. PMID:
20064809.
Codagnone C, Lupiañez-Villanueva F. Benchmarking Deployment of eHealth among General

Practitioners (2013) – Final Report. Luxembourg: European Commission:2014. Available at:
file:///X:/SOM/Informatics/Informat/EPC%20IV%20TO11%20Health%20Info%20Exchange/4
%20Articles,%20Searches,%20ENL/PDFs/Articles/ON%20HOLD/Codagnone%202013.pdf.
Accessed December 16, 2014.
Dixon B, Miller T, Overhage M. Barriers to achieving the last mile in health information
exchange: a survey of small hospitals and physician practices. J Healthc Inf Manag.
2013;27(4):55-8.
Dixon BE, Jones JF, Grannis SJ. Infection preventionists' awareness of and engagement in health
information exchange to improve public health surveillance. Am J Infect Control.
2013;41(9):787-92. PMID: 23415767.
Dixon BE, McGowan JJ, Grannis SJ. Electronic laboratory data quality and the value of a health
information exchange to support public health reporting processes. AMIA Annu Symp Proc.
2011;2011:322-30. PMID: 22195084.
Dobalian A, Claver ML, Pevnick JM, et al. Organizational challenges in developing one of the
Nationwide Health Information Network trial implementation awardees. J Med Syst.
2012;36(2):933-40. PMID: 20703640.
Dullabh P, Hovey L. Large Scale Health Information Exchange: Implementation Experiences

from Five States. Stud Health Technol Inform. 2013;192:613-7. PMID: 23920629.
Dullabh P, Ubri P, Hovey L. The State HIE Program Four Years Later: Key Findings on
Grantees’ Experiences from a Six-State Review. Office of the National Coordinator for Health
Information Technology. 2014. Available at:
http://www.healthit.gov/sites/default/files/CaseStudySynthesisGranteeExperienceFinal_121014.
pdf. Accessed April 22, 2015.
C-3
eHealth Initiative. Result from Survey on Health Data Exchange 2013. The Challenge to
Connect. Available at: http://www.ehidc.org/resource-center/reports/view_document/458-
survey-results-results-from-survey-on-data-exchange-2013-data-exchange. Accessed March 3,
2015.
eHealth Initiative. Post HITECH: The Landscape of Health Information Exchange. Available at:
http://www.ehidc.org/resource-center/publications/view_document/461-reports-2014-ehi-data-
exchange-survey-key-findings. Accessed January 9, 2014.
Fairbrother G, Trudnak T, Christopher R, et al. Cincinnati Beacon Community Program

Highlights Challenges And Opportunities On The Path To Care Transformation. Health Aff.
2014;33(5):871-7. PMID: 24799586.
Feldman SS, Horan TA. Collaboration in electronic medical evidence development: a case study
of the Social Security Administration's MEGAHIT System. Int J Med Inf. 2011;80(8):e127-40.
PMID: 21333588.
Feldman SS, Schooley LB, Bhavsar PG. Health Information Exchange Implementation: Lessons
Learned and Critical Success Factors From a Case Study. JMIR Med Inform. 2014;2(2):e19.
Finnell JT, Overhage JM. Emergency medical services: the frontier in health information
exchange. AMIA Annu Symp Proc. 2010;2010:222-6. PMID: 21346973.
Foldy S. Inventory of electronic health information exchange in Wisconsin, 2006. WMJ.

2007;106(3):120-5. PMID: 17642349.
Fontaine P, Zink T, Boyle RG, et al. Health information exchange: participation by Minnesota
primary care practices. Arch Intern Med. 2010;170(7):622-9. PMID: 20386006.
Frisse ME, Johnson KB, Nian H, et al. The financial impact of health information exchange on
emergency department care. J Am Med Inform Assoc. 2012;19(3):328-33. PMID: 22058169.
Furukawa MF, King J, Patel V, et al. Despite Substantial Progress In EHR Adoption, Health
Information Exchange And Patient Engagement Remain Low In Office Settings. Health Aff.
2014:1-8. PMID: 25104827.
Furukawa MF, Patel V, Charles D, et al. Hospital Electronic Health Information Exchange Grew
Substantially In 2008-12. Health Aff. 2013;32(8):1346-54. PMID: 23918477.
Gadd CS, Ho Y-X, Cala CM, et al. User perspectives on the usability of a regional health
information exchange. J Am Med Inform Assoc. 2011;18(5):711-6. PMID: 21622933.
Genes N, Shapiro J, Vaidya S, et al. Adoption of health information exchange by emergency

physicians at three urban academic medical centers. Appl Clin Inform. 2011;2(3):263-9. PMID:
23616875.
C-4
Goldwater J, Jardim J, Khan T, et al. Emphasizing Public Health Within a Health Information
Exchange: An Evaluation of the District of Columbia’s Health Information Exchange Program.
EGEMS (Wash DC). 2014;2(3)
Greenhalgh T, Stramer K, Bratan T, et al. Adoption and non-adoption of a shared electronic

summary record in England: a mixed-method case study. BMJ. 2010;340:c3111. PMID:
20554687.
Grossman JM, Kushner KL, November EA. Creating sustainable local health information
exchanges: can barriers to stakeholder participation be overcome? Res Briefs. 2008(2):1-12.
PMID: 18496926.
Gutteridge DL, Genes N, Hwang U, et al. Enhancing a Geriatric Emergency Department Care
Coordination Intervention Using Automated Health Information Exchange-Based Clinical Event
Notifications. EGEMS (Wash DC). 2014;2(3)
Hamann DJ, Bezboruah KC. Utilization of technology by long-term care providers: comparisons
between for-profit and nonprofit institutions. J Aging Health. 2013;25(4):535-54. PMID:
23509114.
Herwehe J, Wilbright W, Abrams A, et al. Implementation of an innovative, integrated electronic

medical record (EMR) and public health information exchange for HIV/AIDS. J Am Med Inform
Assoc. 2012;19(3):448-52. PMID: 22037891.
Hessler BJ, Soper P, Bondy J, et al. Assessing the relationship between health information
exchanges and public health agencies. J Public Health Manag Pract. 2009;15(5):416-24. PMID:
19704310.
Hincapie AL, Warholak TL, Murcko AC, et al. Physicians' opinions of a health information
exchange. J Am Med Inform Assoc. 2011;18(1):60-5. PMID: 21106994.
Hyppönen H, Reponen J, Lääveri T, et al. User experiences with different regional health
information exchange systems in Finland. Int J Med Inf. 2014;83(1):1-18. PMID: 24200753.
Jha AK, Doolan D, Grandt D, et al. The use of health information technology in seven nations.
Int J Med Inf. 2008;77(12):848-54. PMID: 18657471.
Johnson KB, Gadd CS, Aronsky D, et al. The MidSouth eHealth Alliance: use and impact in the
first year. AMIA Annu Symp Proc. 2008:333-7. PMID: 18999184.
Johnson KB, Unertl KM, Chen Q, et al. Health information exchange usage in emergency
departments and clinics: the who, what, and why. J Am Med Inform Assoc. 2011;18(5):690-7.
PMID: 21846788.
Jones SS, Friedberg MW, Schneider EC. Health information exchange, Health Information
Technology use, and hospital readmission rates. AMIA Annu Symp Proc. 2011;2011:644-53.
PMID: 22195120.
C-5
Kaelber DC, Waheed R, Einstadter D, et al. Use and perceived value of health information
exchange: one public healthcare system's experience. Am J Manag Care. 2013;19(10 Spec
No):Sp337-43. PMID: 24511888.
Kaushal R, Dhopeshwarkar R, Gottlieb L, et al. User experiences with pharmacy benefit

manager data at the point of care. J Eval Clin Pract. 2010;16(6):1076-80. PMID: 20666888.
Kern LM, Barrón Y, Abramson EL, et al. HEAL NY: Promoting interoperable health
information technology in New York State. Health Aff. 2009;28(2):493-504. PMID: 19276009.
Kern LM, Barrón Y, Dhopeshwarkar RV, et al. Health information exchange and ambulatory
quality of care. Appl Clin Inform. 2012;3(2):197-209. PMID: 23646072.
Kern LM, Wilcox A, Shapiro J, et al. Which components of health information technology will
drive financial value? Am J Manag Care. 2012;18(8):438-45. PMID: 22928759.
Kern LM, Wilcox AB, Shapiro J, et al. Community-based health information technology
alliances: potential predictors of early sustainability. Am J Manag Care. 2011;17(4):290-5.
PMID: 21615199.
Kho AN, Doebbeling BN, Cashy JP, et al. A regional informatics platform for coordinated
antibiotic-resistant infection tracking, alerting, and prevention. Clin Infect Dis. 2013;57(2):254-
62. PMID: 23575195.
Kierkegaard P, Kaushal R, Vest JR. How could health information exchange better meet the
needs of care practitioners? Appl Clin Inform. 2014;5(4):861-77.
Lammers EJ, Adler-Milstein J, Kocher KE. Does health information exchange reduce redundant
imaging? Evidence from emergency departments. Med Care. 2014;52(3):227-34. PMID:
24374414.
Lang E, Afilalo M, Vandal AC, et al. Impact of an electronic link between the emergency
department and family physicians: a randomized controlled trial. CMAJ. 2006;174(3):313-8.
PMID: 16399880.
Lee S-I, Park H, Kim J-W, et al. Physicians' perceptions and use of a health information
exchange: a pilot program in South Korea. Telemed J E Health. 2012;18(8):604-12. PMID:
22352898.
Lobach DF, Kawamoto K, Anstrom KJ, et al. Proactive population health management in the
context of a regional health information exchange using standards-based decision support. AMIA
Annu Symp Proc. 2007:473-7. PMID: 18693881.
Maass MC, Asikainen P, Mäenpää T, et al. Usefulness of a Regional Health Care Information
System in primary care. A case study. Comput Methods Programs Biomed. 2008;91(2):175-81.
PMID: 18514363.
C-6
Machan C, Ammenwerth E, Schabetsberger T. Evaluation of the electronic transmission of
medical findings from hospitals to practitioners by triangulation. Methods Inf Med.
2006;45(2):225-33. PMID: 16538293.
Mäenpää T, Asikainen P, Gissler M, et al. Outcomes assessment of the regional health

information exchange: a five-year follow-up study. Methods Inf Med. 2011;50(4):308-18.
PMID: 21336419.
Mäenpää T, Asikainen P, Gissler M, et al. The utilization rate of the regional health information
exchange: how it impacts on health care delivery outcomes. J Public Health Manag Pract.
2012;18(3):215-23. PMID: 22473113.
Magnus M, Herwehe J, Gruber D, et al. Improved HIV-related outcomes associated with

implementation of a novel public health information exchange. Int J Med Inf. 2012;81(10):e30-8.
PMID: 22883431.
Massy-Westropp M, Giles LC, Law D, et al. Connecting hospital and community care: the
acceptability of a regional data linkage scheme. Aust Health Rev. 2005;29(1):12-6. PMID:
15683350.
McCarthy DB, Propp K, Cohen A, et al. Learning from Health Information Exchange Technical
Architecture and Implementation in Seven Beacon Communities. EGEMS (Wash DC).
2014;2(1):6.
McCullough JM, Zimmerman FJ, Bell DS, et al. Electronic health information exchange in
underserved settings: examining initiatives in small physician practices & community health
centers. BMC Health Serv Res. 2014;14:415. PMID: 25240718.
McGowan JJ, Jordan C, Sims T, et al. Rural RHIOs: common issues in the development of two
state-wide health information networks. AMIA Annu Symp Proc. 2007:528-32. PMID:
18693892.
Merrill JA, Deegan M, Wilson RV, et al. A system dynamics evaluation model: implementation
of health information exchange for public health reporting. J Am Med Inform Assoc.
2013;20(e1):e131-8. PMID: 23292910.
Messer LC, Parnell H, Huffaker R, et al. The development of a health information exchange to
enhance care and improve patient outcomes among HIV+ individuals in rural North Carolina. Int
J Med Inf. 2012;81(10):e46-55. PMID: 22898321.
Miller AR, Tucker C. Health information exchange, system size and information silos. J Health
Econ. 2014;33:28-42. PMID: 24246484.
Miller RH. Satisfying patient-consumer principles for health information exchange: evidence
from California case studies. Health Aff. 2012;31(3):537-47. PMID: 22392664.
Moore T, Shapiro JS, Doles L, et al. Event detection: a clinical notification service on a health
information exchange platform. AMIA Annu Symp Proc. 2012;2012:635-42. PMID: 23304336.
C-7
Morris G, Afzal S, Bhasker M, et al. Query-Based Exchange: Key Factors Influencing Success
and Failure. Office of the National Coordinator for Health Information Technology. 2012
Myers JJ, Koester KA, Chakravarty D, et al. Perceptions regarding the ease of use and usefulness
of health information exchange systems among medical providers, case managers and non-
clinical staff members working in HIV care and community settings. Int J Med Inf.
2012;81(10):e21-9. PMID: 22854159.
Nagykaldi ZJ, Yeaman B, Jones M, et al. HIE-i-Health Information Exchange With Intelligence.
J Ambulatory Care Manage. 2014;37(1):20-31. PMID: 24309392.
Nohr C, Kristensen M, Andersen SK, et al. Shared experience in 13 local Danish EPR projects:
the Danish EPR Observatory. Stud Health Technol Inform. 2001;84(Pt 1):670-4. PMID:
11604822.
Nykänen P, Karimaa E. Success and failure factors in the regional health information system
design process--results from a constructive evaluation study. Methods Inf Med. 2006;45(1):85-9.
PMID: 16482376.
Onyile A, Vaidya SR, Kuperman G, et al. Geographical distribution of patients visiting a health
information exchange in New York City. J Am Med Inform Assoc. 2013;20(e1):e125-30. PMID:
23104049.
Overhage JM, Evans L, Marchibroda J. Communities' readiness for health information exchange:
the National Landscape in 2004. J Am Med Inform Assoc. 2005;12(2):107-12. PMID: 15561785.
Overhage JM, Grannis S, McDonald CJ. A comparison of the completeness and timeliness of
automated electronic laboratory reporting and spontaneous reporting of notifiable conditions.
Am J Public Health. 2008;98(2):344-50. PMID: 18172157.
Ozkaynak M, Brennan PF. Revisiting sociotechnical systems in a case of unreported use of

health information exchange system in three hospital emergency departments. J Eval Clin Pract.
2013;19(2):370-3. PMID: 22420774.
Pagliari C, Gilmour M, Sullivan F. Electronic Clinical Communications Implementation (ECCI)

in Scotland: a mixed-methods programme evaluation. J Eval Clin Pract. 2004;10(1):11-20.
PMID: 14731147.
Park H, Lee S-i, Kim Y, et al. Patients' perceptions of a health information exchange: a pilot
program in South Korea. Int J Med Inf. 2013;82(2):98-107. PMID: 22658777.
Patel V, Swain MJ, King J, et al. Physician capability to electronically exchange clinical
information, 2011. Am J Manag Care. 2013;19(10):835-43. PMID: 24304162.
Phillips AB, Wilson RV, Kaushal R, et al. Implementing health information exchange for public
health reporting: a comparison of decision and risk management of three regional health
information organizations in New York state. J Am Med Inform Assoc. 2014;21(e1):e173-7.
PMID: 23975626.
C-8
Pirnejad H, Bal R, Berg M. Building an inter-organizational communication network and
challenges for preserving interoperability. Int J Med Inf. 2008;77(12):818-27. PMID: 18579436.
Pouloudi A. Information technology for collaborative advantage in healthcare revisited.

Information & Management. 1999;35(6):345-56.
Ross SE, Radcliff TA, Leblanc WG, et al. Effects of health information exchange adoption on
ambulatory testing rates. J Am Med Inform Assoc. 2013;20(6):1137-42. PMID: 23698257.
Ross SE, Schilling LM, Fernald DH, et al. Health information exchange in small-to-medium
sized family medicine practices: motivators, barriers, and potential facilitators of adoption. Int J
Med Inf. 2010;79(2):123-9. PMID: 20061182.
Rudin R, Volk L, Simon S, et al. What Affects Clinicians' Usage of Health Information
Exchange? Appl Clin Inform. 2011;2(3):250-62. PMID: 22180762.
Rudin RS, Simon SR, Volk LA, et al. Understanding the decisions and values of stakeholders in
health information exchanges: experiences from Massachusetts. Am J Public Health.
2009;99(5):950-5. PMID: 19299671.
Saff E, Lanway C, Chenyek A, et al. The Bay Area HIE. A case study in connecting
stakeholders. J Healthc Inf Manag. 2010;24(1):25-30. PMID: 20077922.
Schabetsberger T, Ammenwerth E, Andreatta S, et al. From a paper-based transmission of

discharge summaries to electronic communication in health care regions. Int J Med Inf.
2006;75(3-4):209-15. PMID: 16112892.
Schoen C, Osborn R, Squires D, et al. A survey of primary care doctors in ten countries shows
progress in use of health information technology, less in other areas. Health Aff (Millwood).
2012;31(12):2805-16. PMID: 23154997.
Shapiro JS, Johnson SA, Angiollilo J, et al. Health Information Exchange Improves
Identification Of Frequent Emergency Department Users. Health Aff. 2013;32(12):2193-8.
PMID: 24301405.
Sicotte C, Paré G. Success in health information exchange projects: solving the implementation
puzzle. Soc Sci Med. 2010;70(8):1159-65. PMID: 20137847.
Silvester BV, Carr SJ. A shared electronic health record: lessons from the coalface. Med J Aust.
2009;190(11 Suppl):S113-6. PMID: 19485857.
Soderberg K, Laventure M. Minnesota clinics' adoption, use and exchange of electronic health
information. Minn Med. 2013;96(9):45-8. PMID: 24494362.
Steward WT, Koester KA, Collins SP, et al. The essential role of reconfiguration capabilities in
the implementation of HIV-related health information exchanges. Int J Med Inf.
2012;81(10):e10-20. PMID: 22841703.
C-9
Swain M, Charles D, Patel V, et al. Health Information Exchange among U.S. Non-federal Acute
Care Hospitals: 2008-2014. ONC Data Brief No. 24. Washington DC: The Office of the national
Coordinator for Health Information Technology (ONC). 2015. Available at:
http://www.healthit.gov/sites/default/files/data-brief/ONC_DataBrief24_HIE_Final.pdf.
Thorn SA, Carter MA, Bailey JE. Emergency physicians' perspectives on their use of health
information exchange. Ann Emerg Med. 2014;63(3):329-37. PMID: 24161840.
Tripathi M, Delano D, Lund B, et al. Engaging patients for health information exchange. Health
Aff. 2009;28(2):435-43. PMID: 19276000.
Tzeel A, Lawnicki V, Pemble KR. The Business Case for Payer Support of a Community-Based
Health Information Exchange: A Humana Pilot Evaluating Its Effectiveness in Cost Control for
Plan Members Seeking Emergency Department Care. Am Health Drug Benefits. 2011;4(4):207-
15. PMID: 25126351.
Tzeel A, Lawnicki V, Pemble KR. "Hidden" Value: How Indirect Benefits of Health Information
Exchange Further Promote Sustainability. Am Health Drug Benefits. 2012;5(6):333-40. PMID:
24991331.
Unertl KM, Johnson KB, Lorenzi NM. Health information exchange technology on the front
lines of healthcare: workflow factors and patterns of use. J Am Med Inform Assoc.
2012;19(3):392-400. PMID: 22003156.
Unertl MK, Johnson BK, Gadd SC, et al. Bridging Organizational Divides in Health Care: An
Ecological View of Health Information Exchange. JMIR Med Inform. 2013;1(1):e3.
Vest JR. Health information exchange and healthcare utilization. J Med Syst. 2009;33(3):223-31.
PMID: 19408456.
Vest JR. More than just a question of technology: factors related to hospitals' adoption and
implementation of health information exchange. Int J Med Inf. 2010;79(12):797-806. PMID:
20889370.
Vest JR, Campion Jr TR, Kaushal R. Challenges, Alternatives, and Paths to Sustainability for
Health Information Exchange Efforts. J Med Syst. 2013;37(6):1-8. PMID: 24141531.
Vest JR, Gamm LD, Ohsfeldt RL, et al. Factors associated with health information exchange
system usage in a safety-net ambulatory care clinic setting. J Med Syst. 2012;36(4):2455-61.
PMID: 21523428.
Vest JR, Grinspan ZM, Kern LM, et al. Using a health information exchange system for imaging
information: patterns and predictors. AMIA Annu Symp Proc. 2013;2013:1402-11. PMID:
24551416.
Vest JR, Issel LM. Factors related to public health data sharing between local and state health
departments. Health Serv Res. 2014;49(1 Pt 2):373-91. PMID: 24359636.
C-10
Vest JR, Jasperson JS. How are health professionals using health information exchange systems?
Measuring usage for evaluation and system improvement. J Med Syst. 2012;36(5):3195-204.
PMID: 22127521.
Vest JR, Jasperson JS, Zhao H, et al. Use of a health information exchange system in the
emergency care of children. BMC Med Inform Decis Mak. 2011;11:78. PMID: 22208182.
Vest JR, Kern LM, Campion TR, Jr., et al. Association between use of a health information
exchange system and hospital admissions. Appl Clin Inform. 2014;5(1):219-31. PMID:
24734135.
Vest JR, Kern LM, Silver MD, et al. The potential for community-based health information
exchange systems to reduce hospital readmissions. J Am Med Inform Assoc. 2014 PMID:
25100447.
Vest JR, Miller TR. The association between health information exchange and measures of
patient satisfaction. Appl Clin Inform. 2011;2(4):447-59. PMID: 23616887.
Vest JR, Zhao H, Jaspserson J, et al. Factors motivating and affecting health information
exchange usage. J Am Med Inform Assoc. 2011;18(2):143-9. PMID: 21262919.
Willis JM, Edwards R, Anstrom KJ, et al. Decision support for evidence-based pharmacotherapy
detects adherence problems but does not impact medication use. Stud Health Technol Inform.
2013;183:116-25. PMID: 23388267.
Winden TJ, Boland LL, Frey NG, et al. Care everywhere, a point-to-point HIE tool: utilization
and impact on patient care in the ED. Appl Clin Inform. 2014;5(2):388-401. PMID: 25024756.
Yeager VA, Walker D, Cole E, et al. Factors Related to Health Information Exchange
Participation and Use. J Med Syst. 2014;38(8) PMID: 24957395.
C-11
Appendix D. List of Excluded Studies
Report on community health information exchanges. Medicine on the Net. 2004;10(3):9-9.
Exclusion: Wrong study design
Connecting communities: making inroads to exchange electronic healthcare data at the local level. Qual Lett Healthc
Lead. 2005;17(8):2-10. PMID: 16304880.
Implementation of SNOMED-CT needed to facilitate interoperable exchange of health information. J AHIMA.

2005;76(9):30, 2.
A primer for building RHIOs. Hosp Health Netw. 2006;80(2):49-56. PMID: 16572948.
Health information exchange activities continuing to mature, says survey. Healthc Financ Manage. 2007;61(2):11.
New computer network helps EDs to reduce redundant test orders: observers see significant savings, benefits in
patient safety. ED Manag. 2008;20(12):133-4. PMID: 19086738.
Wisconsin HIE optimizes community care. Communication among ED clinicians and federally qualified health
centers in the Milwaukee area was improved, including real-time access to patient historical-encounter data. Health
Manag Technol. 2009;30(12):28-9. PMID: 20043491.
States with the most health information exchanges. Mod Healthc. 2009;39(27):32. PMID: 19606671.
By the numbers. States with the most health information exchanges. Based on eHealth initiative's directory of health
information exchange initiatives. Mod Healthc. 2010;40(14):34. PMID: 20402215.
Physicians support health information exchange but are concerned about paying monthly fees. AHRQ Research
Activities. 2010(359):14.
Information exchange yields better decisions. ED Manag. 2010;22(9):103-4. PMID: 20853581.

Exclusion: No data relevant to a Key Question
Social Security report details $2 million return on HIE. For the Record (Great Valley Publishing Company, Inc).
2010;22(4):6.
Findings from site visit to community clinic health network in san Diego, CA. Available at:
http://aspe.hhs.gov/sp/reports/2010/chcit2010/SanDiego.html. Accessed November 10, 2014.
States with the most health information exchanges. Based on ehealth initiative's map of health information exchange
activity in the U.S. Mod Healthc. 2011;41(21):32. PMID: 21714447.
By The Numbers: States with the most health information exchanges. Mod Healthc. 2011;41(21):32.
D-1
High-tech approach to medication reconciliation saves time, bolsters safety at hospital in northern Virginia. ED
Manag. 2011;23(10):117-9. PMID: 21972757.
Exclusion: Not HIE
Survey shows health information exchange on the rise. For the Record (Great Valley Publishing Company, Inc).
2011;23(15):5.
States with the most health information exchanges: Based on eHealth initiative's map of health information
exchange activity in the U.S. Mod Healthc. 2012;42(24):34. PMID: 22957359.
What is HIE (Health Information Exchange)? Department of Health and Human Services. Washington, DC.
Available at: http://www.healthit.gov/providers-professionals/health-information-exchange/what-hie. Accessed
April 18, 2014.
Health Information Exchange Roadmap: The Landscape and a Path Forward. National eHealth Collaborative.
Washington, DC. Available at: http://www.nationalehealth.org/hie-roadmap. Accessed April 18, 2014.
Study: More Hospitals Joining Health Information Exchanges. J AHIMA. 2012;83(11):13.

Electronic tools for health information exchange: An evidence-based analysis. Ont Health Technol Assess Ser.
2013;13(11):1-76. PMID: 2419479.
Exclusion: Systematic review not meeting our requirements
Health Information Exchange May Reduce Hospital Admissions. For the Record (Great Valley Publishing
Company, Inc). 2014;26(5):32.
Aas IHM, Geitung JT. Choosing networks for picture archiving and communication systems and teleradiology. J
Telemed Telecare. 2003;9 Suppl 1:S27-9. PMID: 12952712.
Exclusion: Not HIE
Aas IM. The organizational challenge for health care from telemedicine and e-health. Oslo: Work Res Inst. 2007
Exclusion: Not HIE
Adler-Milstein J, DesRoches CM, Furukawa MF, et al. More than half of US hospitals have at least a basic EHR,
but stage 2 criteria remain challenging for most. Health Aff (Millwood). 2014;33(9):1664-71. PMID: 25104826.
Exclusion: Not HIE
Afzal S, Morris G, Palmer S. Health Information Exchange Services in Support of Disaster Preparedness and
Emergency Medical Response: Assessment of Opportunity in California and the Gulf Coast: Office of the National
Coordinator for Health Information Technology; 2014.
Agarwal M, Bourgeois J, Sodhi S, et al. Updating a patient-level ART database covering remote health facilities in
Zomba district, Malawi: Lessons learned. Public Health Action. 2013;3(2):175-9.
Exclusion: Not HIE
Ahern DK, Kreslake JM, Phalen JM. What is eHealth (6): perspectives on the evolution of eHealth research. J Med
Internet Res. 2006;8(1):e4. PMID: 16585029.
D-2
Exclusion: Not HIE
Ahmed S, Bartlett SJ, Ernst P, et al. Effect of a web-based chronic disease management system on asthma control
and health-related quality of life: study protocol for a randomized controlled trial. Trials. 2011;12:260. PMID:
22168530.
Allen A, Des Jardins TR, Heider A, et al. Making it local: Beacon communities use health information technology to
optimize care management. Popul Health Manag. 2014;17(3):149-58. PMID: 24476558.
Allen C, Des Jardins TR, Heider A, et al. Data Governance and Data Sharing Agreements for Community-Wide
Health Information Exchange: Lessons from the Beacon Communities. EGEMS (Wash DC). 2014;2(1)
Allen KA. Parent and Provider Decision-Making for Infants with HIE, Duke University; 2012.
Exclusion: Not HIE
Allender S, Nichols M, Foulkes C, et al. The development of a network for community-based obesity prevention:
the CO-OPS Collaboration. BMC Public Health. 2011;11:132. PMID: 21349185.
Exclusion: Not HIE
Ancker JS, Edwards AM, Miller MC, et al. Consumer perceptions of electronic health information exchange. Am J
Prev Med. 2012;43(1):76-80. PMID: 22704751.
Exclusion: Not HIE
Ancker JS, Miller MC, Patel V, et al. Sociotechnical challenges to developing technologies for patient access to
health information exchange data. J Am Med Inform Assoc. 2014;21(4):664-70. PMID: 24064443.
Ancker JS, Silver M, Miller MC, et al. Consumer experience with and attitudes toward health information
technology: a nationwide survey. J Am Med Inform Assoc. 2013;20(1):152-6. PMID: 22847306.
Exclusion: Not HIE
Anderson JG. Social, ethical and legal barriers to e-health. Int J Med Inf. 2007;76(5-6):480-3. PMID: 17064955.
Exclusion: Not HIE
Andrade SE, Davis RL, Cheetham TC, et al. Medication Exposure in Pregnancy Risk Evaluation Program. Matern
Child Health J. 2012;16(7):1349-54. PMID: 22002179.
Exclusion: Not HIE
Angiollilo J, Fleischman W, Kuperman G, et al. Improving identification of hospital readmissions using a regional
health information exchange. Acad Emerg Med. 2012;19:S50.
Angst CM. Protect my privacy or support the common-good? Ethical questions about electronic health information
exchanges. J Bus Ethics. 2009;90(Suppl 2):169-78.
Angulo C, Crespo P, Maldonado JA, et al. Non-invasive lightweight integration engine for building EHR from
autonomous distributed systems. Int J Med Inform. 2007;76 Suppl 3:S417-24. PMID: 17600763.
Appleby C. NYCLIX: New York HIE life. An expansive HIE network has taken shape in the nation's most densely
populated urban area. Healthc Inform. 2010;27(10):29-31. PMID: 21049716.
D-3
Appleby C. Surfing the HIE. The Santa Cruz information exchange experience offers lessons on what works.
Healthc Inform. 2010;27(6):68-9. PMID: 20593734.
Arar NH, Wen L, McGrath J, et al. Communicating about medications during primary care outpatient visits: the role
of electronic medical records. Inform Prim Care. 2005;13(1):13-22. PMID: 15949171.
Exclusion: Not HIE
Asangansi I, Braa K. The emergence of mobile-supported national health information systems in developing
countries... MEDINFO 2010: Proceedings of the 13th World Congress on Medical Informatics, Part 1. Stud Health
Technol Inform. 2010;160:540-4.
Aschman DJ, Abshire TC, Shapiro AD, et al. A community-based partnership to promote information infrastructure
for bleeding disorders. Am J Prev Med. 2011;41(6 Suppl 4):S332-7. PMID: 22099355.
Ash JS, Guappone KP. Qualitative evaluation of health information exchange efforts. J Biomed Inform. 2007;40(6
Suppl):S33-9. PMID: 17904914.
Ashley L, Jones H, Forman D, et al. Feasibility test of a UK-scalable electronic system for regular collection of
patient-reported outcome measures and linkage with clinical cancer registry data: The electronic Patient-reported
Outcomes from Cancer Survivors (ePOCS) system. BMC Med Inform Decis Mak. 2011;11:66. PMID: 22029686.
Badia CM, Duenas AE, Martinez OM, et al. My health log. Eur J Intern Med. 2011;22:S56-S7.
Bah S, Alharthi H, El Mahalli AA, et al. Annual Survey on the Level and Extent of Usage of Electronic Health
Records in Government-related Hospitals. Perspect Health Inf Manag. 2011;8(4):1-12. PMID: 22016668.
Exclusion: Not HIE
Bailey JE, Wan J, Pope R, et al. Health information exchange use reduces avoidable diagnostic imaging in the
emergency evaluation of back pain. J Gen Intern Med. 2011;26(University of Tennessee Health Science Center,
Memphis, United States):S349-S50.
Bailey JE, Yu X, Ward RD, et al. Effect of health information exchange on hospital admissions for chest pain. J
Investig Med. 2012;60(1):465-6.
Balas A, Al Sanousi A. Interoperable electronic patient records for health care improvement. Stud Health Technol
Inform. 2009;150:19-23. PMID: 19745258.
Balasingham I, Ihlen H, Leister W, et al. Communication of medical images, text, and messages in inter-enterprise
systems: a case study in Norway. IEEE Trans Inf Technol Biomed. 2007;11(1):7-13. PMID: 17249398.
Balka E, Tolar M, Coates S, et al. Socio-technical issues and challenges in implementing safe patient handovers:
Insights from ethnographic case studies. Int J Med Inf. 2013;82(12):e345-e57. PMID: 23218926.
Exclusion: Not HIE
D-4
Ball MJ, Gold J. Banking on health: Personal records and information exchange. J Healthc Inf Manag.
2006;20(2):71-83. PMID: 16669591.
Ballard J, Rosenman M, Weiner M. Harnessing a health information exchange to identify surgical device adverse
events for urogynecologic mesh. AMIA Annu Symp Proc. 2012;2012:1109-18. PMID: 23304387.
Bansal M, Grannis S, Kansky J, et al. Evaluating cost differences among operational teams supporting the Indiana
health information exchange. Value Health. 2009;12(3):A87.
Bansler JP, Havn E. Pilot implementation of health information systems: Issues and challenges. Int J Med Inf.
2010;79(9):637-48. PMID: 20576466.
Bara D, McPhillips-Tangum C, Wild EL, et al. Integrating child health information systems in public health
agencies. J Public Health Manag Pract. 2009;15(6):451-8. PMID: 19823148.
Exclusion: Not HIE
Barbarito F, Pinciroli F, Mason J, et al. Implementing standards for the interoperability among healthcare providers
in the public regionalized Healthcare Information System of the Lombardy Region. J Biomed Inform.
2012;45(4):736-45. PMID: 22285983.
Barrows RC, Jr., Ezzard J. Technical architecture of ONC-approved plans for statewide health information
exchange. AMIA Annu Symp Proc. 2011;2011:88-97. PMID: 22195059.
Basch P. Will interoperable HIT lead to a net gain or to a net loss for physicians? (2/23/2005). Health Aff
(Millwood). 2005;Suppl Web Exclusives:W5-S-1-W5-S-3; author reply W5-S-3-W5-S-6. PMID: 16440450.
Bassi J, Lau F. Measuring value for money: a scoping review on economic evaluation of health information systems.
J Am Med Inform Assoc. 2013;20(4):792-801. PMID: 23416247.
Exclusion: Not HIE
Bates DW, Gawande AA. Improving Safety with Information Technology. N Engl J Med. 2003;348(25):2526-34.
PMID: 12815139.
Exclusion: Not HIE
Beaulieu-Volk D. EHRs' interoperability challenge. HIE expansion aimed at helping providers exchange health
information safely, but not all services created equally. Med Econ. 2014;91(6):50-3. PMID: 25219166.
Beckjord EB, Rechis R, Nutt S, et al. What Do People Affected by Cancer Think About Electronic Health
Information Exchange? Results From the 2010 LIVESTRONG Electronic Health Information Exchange Survey and
the 2008 Health Information National Trends Survey. J Oncol Pract. 2011;7(4):237-41. PMID: 22043188.
Bell DS, Cima L, Seiden DS, et al. Effects of laboratory data exchange in the care of patients with HIV. Int J Med
Inf. 2012;81(10):e74-82. PMID: 22906370.
Exclusion: Not HIE
Ben-Assuli O, Shabtai I, Leshno M. Using electronic health record systems to optimize admission decisions: The
Creatinine case study. Health Informatics J. 2014. PMID: 24692078.
D-5
Exclusion: More recent data available
Ben-Assuli O, Shabtai I, Leshno M, et al. EHR in emergency rooms: Exploring the effect of key information
components on main complaints. J Med Syst. 2014;38(4). PMID: 24687240.
Exclusion: No comparison group
Benford MS, Slack CB. Development of a statewide maternal and child health information network... MATCH.
Comput Nurs. 1989;7(1):9-14. PMID: 2924201.
Bergmann J, Bott OJ, Pretschner DP, et al. An e-consent-based shared EHR system architecture for integrated
healthcare networks. Int J Med Inform. 2007;76(2-3):130-6. PMID: 16971171.
Berry JG, Goldmann DA, Mandl KD, et al. Health information management and perceptions of the quality of care
for children with tracheotomy: a qualitative study. BMC Health Serv Res. 2011;11:117. PMID: 21605385.
Exclusion: Not HIE
Beynon-Davies P, Lloyd-Williams M. When health information systems fail. Top Health Inf Manage.
1999;20(1):66-79. PMID: 10539424.
Biondich PG, Grannis SJ. The Indiana Network for Patient Care: an integrated clinical information system informed
by over thirty years of experience. J Public Health Manag Pract. 2004:S81-6. PMID: 15643364.
Bipartisan Policy Center. Clinician Perspectives on Electronic Health Information Sharing for Transitions of Care
2012. Available at: https://www.acponline.org/running_practice/technology/bpc_clinician_survey_100312.pdf.
Accessed November 10, 2014.
Black CD, Burchill CA, Roos LL. The Population Health Information System: data analysis and software. Med
Care. 1995;33(12 Suppl):DS127-31. PMID: 7500665.
Blaya JA, Shin SS, Yagui MJA, et al. A web-based laboratory information system to improve quality of care of
tuberculosis patients in Peru: Functional requirements, implementation and usage statistics. BMC Med Inform Decis
Mak. 2007;7. PMID: 17963522.
Exclusion: Not HIE
Blobel B. Standards and solutions for architecture based, ontology driven and individualized pervasive health. Stud
Health Technol Inform. 2012;177:147-57. PMID: 22942047.
Exclusion: Not HIE
Bohren BF, Hadzikadic M. Turning medical data into decision-support knowledge. Proc Annu Symp Comput Appl
Med Care. 1994:735-9. PMID: 7950022.
Exclusion: Not HIE
Bonney W. Determinants in the acceptance of Health Level Seven (HL7) version 3 messaging standard. Diss Abstr
Int. 2013;73(12-B(E)).
Exclusion: Not HIE
Boockvar KS, Livote EE, Goldstein N, et al. Electronic health records and adverse drug events after patient transfer.
Qual Saf Health Care. 2010;19(5):e16. PMID: 20724395.
Exclusion: Not HIE
D-6
Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic
review to taxonomy and interventions. BMC Health Serv Res. 2010;10:231. PMID: 20691097.
Exclusion: Not HIE
Bouhaddou O, Bennett J, Teal J, et al. Toward a Virtual Lifetime Electronic Record: the Department of Veterans
Affairs experience with the Nationwide Health Information Network. AMIA Annu Symp Proc. 2012;2012:51-60.
PMID: 23304272.
Bouhaddou O, Cromwell T, Davis M, et al. Translating standards into practice: experience and lessons learned at the
Department of Veterans Affairs. J Biomed Inform. 2012;45(4):813-23. PMID: 22285982.
Exclusion: Not HIE
Bouhaddou O, Warnekar P, Parrish F, et al. Exchange of computable patient data between the Department of
Veterans Affairs (VA) and the Department of Defense (DoD): terminology mediation strategy. J Am Med Inform
Assoc. 2008;15(2):174-83. PMID: 18096911.
Bourn M, Davies CA. A prodigious information systems failure. Top Health Inf Manage. 1996;17(2):34-44. PMID:
10162539.
Bowen R, Carey S, Carter P, et al. HIE management and operational considerations. J AHIMA. 2011;82(5):56-61.
PMID: 21667869.
Brailer DJ. Connection tops collection: peer-to-peer technology lets caregivers access necessary data, upon request,
without using a repository. Health Manag Technol. 2001;22(8):28-9. PMID: 11499130.
Brailer DJ. From Santa Barbara to Washington: a person's and a nation's journey toward portable health information.
Health Aff. 2007;26(5):w581-8. PMID: 17670776.
Branger PJ, van't Hooft A, van der Wouden JC, et al. Shared care for diabetes: supporting communication between
primary and secondary care. Int J Med Inf. 1999;53(2-3):133-42. PMID: 10193883.
Exclusion: Not HIE
Brattheim B, Faxvaag A, Toussaint P. When information sharing is not enough. Stud Health Technol Inform.
2011;169:359-63. PMID: 21893773.
Brelstaff G, Moehrs S, Anedda P, et al. Internet patient records: new techniques. J Med Internet Res. 2001;3(1):E8.
PMID: 11720950.
Exclusion: Not HIE
Brennan CP. Managed care and health information networks. J Health Care Finance. 1995;21(4):1-5. PMID:
7583779.
Brocht DF, Abbott PA, Smith CA, et al. A clinic on wheels. A paradigm shift in the provision of care and the
challenges of information infrastructure. Comput Nurs. 1999;17(3):109-13. PMID: 10341475.
Brokel JM. Capture, exchange and use data, information and knowledge within electronic health records. Iowa
Nurse Reporter. 2007;20(1):1, 25, 7.
D-7
Brokel JM. Regional health information organization (RHIO) to exchange data. Iowa Nurse Reporter. 2007;20(2):4-
5.
Brokel JM. Iowa e-health project: planning for health information exchange with nursing standardized language
with health information technology tools. Iowa Nurse Reporter. 2009;22(3):1.
Brokel JM, Harrison MI. Redesigning care processes using an electronic health record: a system's experience. Jt
Comm J Qual Patient Saf. 2009;35(2):82-92. PMID: 19241728.
Exclusion: Not HIE
Brown CVR, Foulkrod KH, Sadler HT, et al. Autologous blood transfusion during emergency trauma operations.
Arch Surg. 2010;145(7):690-4. PMID: 20644133.
Exclusion: Not HIE
Brown JS, Holmes JH, Shah K, et al. Distributed health data networks: a practical and preferred approach to multi-
institutional evaluations of comparative effectiveness, safety, and quality of care. Med Care. 2010;48(6 Suppl):S45-
51. PMID: 20473204.
Brown ML, Riley GF, Potosky AL, et al. Obtaining long-term disease specific costs of care: application to Medicare
enrollees diagnosed with colorectal cancer. Med Care. 1999;37(12):1249-59. PMID: 10599606.
Exclusion: Not HIE
Buntin MB, Burke MF, Hoaglin MC, et al. The benefits of health information technology: a review of the recent
literature shows predominantly positive results. Health Aff (Millwood). 2011;30(3):464-71. PMID: 21383365.
Burkle T, Schweiger R, Altmann U, et al. Transferring data from one EPR to another: content--syntax--semantic.
Methods Inf Med. 1999;38(4-5):321-5. PMID: 10805022.
Burstin H, Clancy C. Primary care experience: crossing the chasm between theory and practice. J Gen Intern Med.
2004;19(10):1064-5. PMID: 15482561.
Exclusion: Not HIE
Butler B. Health Information Exchange between Jails and Their Communities: A Bridge That Is Needed under
Healthcare Reform. Perspect Health Inf Manag. 2014:1-6. PMID: 24808809.
Caldwell D. Health information exchange. MLO Med Lab Obs. 2012;44(11):46. PMID: 23173526.
Caldwell D. Management Q&A. Health information exchange. MLO Med Lab Obs. 2012;44(11):46.
Callen J, Paoloni R, Li J, et al. Perceptions of the effect of information and communication technology on the
quality of care delivered in emergency departments: a cross-site qualitative study. Ann Emerg Med. 2013;61(2):131-
44. PMID: 23083964.
Callen JL, Braithwaite J, Westbrook JI. Contextual implementation model: a framework for assisting clinical
information system implementations. J Am Med Inform Assoc. 2008;15(2):255-62. PMID: 18096917.
D-8
Exclusion: Not HIE
Carr CM, Krywko DM, Moore HE, et al. The impact of a health information exchange on the management of
patients in an urban academic emergency department: An observational study and cost analysis. Ann Emerg Med.
2012;60(4):S15.
Carr CM, Saef SH, Zhao J, et al. Can data from a health information exchange be used to describe patients who visit
multiple emergency departments within a region? Acad Emerg Med. 2014;21(5):S141.
Exclusion: Not HIE
Carr D, Howells A, Chang M, et al. An integrated approach to stakeholder engagement. Healthc Q. 2009;12 Spec
No Ontario:62-70. PMID: 19458512.
Exclusion: Not HIE
Carr K, Bangalore D, Benin A, et al. Leveraging the benefits of Health Information Technology to support
healthcare delivery model redesign. J Healthc Inf Manag. 2006;20(1):31-41. PMID: 16429957.
Exclusion: Not HIE
Cebul RD, Love TE, Jain AK, et al. Electronic health records and quality of diabetes care. N Engl J Med.
2011;365(9):825-33. PMID: 21879900.
Exclusion: Not HIE
Centers for Disease C, Prevention. State electronic disease surveillance systems --- United States, 2007 and 2010.
MMWR Morb Mortal Wkly Rep. 2011;60(41):1421-3. PMID: 22012115.
Exclusion: Not HIE
Centorrino F, Mark TL, Talamo A, et al. Health and economic burden of metabolic comorbidity among individuals
with bipolar disorder. J Clin Psychopharmacol. 2009;29(6):595-600. PMID: 19910727.
Exclusion: Not HIE
Champagne T. The development of community-based health information exchanges: A comparative assessment of

organizational models. Diss Abstr Int. 2014;75(5-B(E)):No Pagination Specified.
Chan TC, Killeen JP, Castillo EM, et al. San diego safety net health information exchange. Ann Emerg Med.
2011;58(4):S310.
Chang I, Hwang H-G, Hung M-C, et al. Factors affecting cross-hospital exchange of Electronic Medical Records.
Information & Management. 2009;46(2):109-15.
Exclusion: Not HIE
Chau PYK, Hu PJH. Information technology acceptance by individual professionals: A model comparison approach.
Decision Sciences. 2001;32(4):699-718.
Exclusion: Not HIE
Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality,
efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742-52. PMID: 16702590.
Chen C, Garrido T, Chock D, et al. The Kaiser Permanente electronic health record: Transforming and streamlining
modalities of care. Health Aff. 2009;28(2):323-33. PMID: 19275987.
Exclusion: Not HIE
D-9
Chen R, Enberg G, Klein GO. Julius--a template based supplementary electronic health record system. BMC Med
Inform Decis Mak. 2007;7:10. PMID: 17474997.
Cheung K-C, van der Veen W, Bouvy ML, et al. Classification of medication incidents associated with information
technology. J Am Med Inform Assoc. 2014;21(e1):e63-70. PMID: 24064444.
Exclusion: Not HIE
Cimino JJ, Frisse ME, Halamka J, et al. Consumer-mediated health information exchanges: The 2012 ACMI debate.
J Biomed Inform. 2014;48:5-15. PMID: 24561078.
Ciriello JN, Kulatilaka N. Smart health community: the hidden value of health information exchange. Am J Manag
Care. 2010;16(12 Suppl HIT):SP31-6. PMID: 21314218.
Clancy GP, Duffy FD. Going "all in" to transform the Tulsa community's health and health care workforce. Acad
Med. 2013;88(12):1844-8. PMID: 24128637.
Clayton PD, Narus SP, Huff SM, et al. Building a comprehensive clinical information system from components. The
approach at Intermountain Health Care. Methods Inf Med. 2003;42(1):1-7. PMID: 12695790.
Coffman T, Porter JP, Frisse ME. Reducing HIE costs through real-time data feed visualizations. AMIA Annu Symp
Proc. 2008:913. PMID: 18999214.
Exclusion: Not HIE
Coiera E. Building a National Health IT System from the middle out. J Am Med Inform Assoc. 2009;16(3):271-3.
PMID: 19407078.
Collaborative MT. Advanced technologies to lower health care costs and improve quality: Executive summary.
Available at: http://mehi.masstech.org/sites/mehi/files/documents/AdvancedTechnologies2004.pdf. Accessed April
22, 2015.
Collin S, Reeves BC, Hendy J, et al. Implementation of computerised physician order entry (CPOE) and picture
archiving and communication systems (PACS) in the NHS: quantitative before and after study. BMJ.
2008;337(a939):1-8. PMID: 18703655.
Exclusion: Not HIE
Collins SA, Bakken S, Vawdrey DK, et al. Model development for EHR interdisciplinary information exchange of
ICU common goals. Int J Med Inf. 2011;80(8):e141-9. PMID: 20974549.
Exclusion: Not HIE
Conn J. RHIOs make it work. Data-sharing project connects three networks. Mod Healthc. 2006;36(7):22. PMID:
16515062.
Constantinides P, Barrett M. Large-scale ICT innovation, power, and organizational change: The case of a regional
health information network. J Appl Behav Sci. 2006;42(1):76-90.
Corado C, Cashy J, Kho A, et al. Fragmented care among stroke patients at 4 Chicago hospitals. Stroke. 2014;45
Exclusion: Not HIE
D-10
Cormont S, Vandenbussche P-Y, Buemi A, et al. Implementation of a platform dedicated to the biomedical analysis
terminologies management. AMIA Annu Symp Proc. 2011;2011:1418-27. PMID: 22195205.
Exclusion: Not HIE
Corporation RHI. Overlay Regional Health Information Exchange (HIE) Systems: The Sustainable Business Model
for Health Care Information Technology in the United States. Available at:
http://ruralhealthit.com/downloads/Overlay_Regional_Health_Information_Exchange_Systems.pdf. Accessed April
22, 2015.
Costa C, Ferreira C, Bastiao L, et al. Dicoogle - an open source peer-to-peer PACS. J Digit Imaging.
2011;24(5):848-56. PMID: 20981467.
Exclusion: Not HIE
Cresswell K, Sheikh A. The NHS Care Record Service (NHS CRS): recommendations from the literature on
successful implementation and adoption. Inform Prim Care. 2009;17(3):153-60. PMID: 20074427.
Crosson JC, Ohman-Strickland PA, Cohen DJ, et al. Typical electronic health record use in primary care practices
and the quality of diabetes care. Ann Fam Med. 2012;10(3):221-7. PMID: 22585886.
Exclusion: Not HIE
Crounse B. A compelling, sustainable business model for RHIO’s. Available at:

http://blogs.msdn.com/healthblog/archive/2005/10/08/478037.aspx. Accessed April 22, 2015.
Csaba Egyhazy, Raj Mukherji. Interoperability architecture using RM-ODP. Commun ACM. 2004;47(2):93-7.
Cummins MR, Crouch B, Gesteland P, et al. Inefficiencies and vulnerabilities of telephone-based communication
between U. S. poison control centers and emergency departments. Clin Toxicol (Phila). 2013;51(5):435-43. PMID:
23697459.
Exclusion: Not HIE
Cummins MR, Crouch BI, Gesteland P, et al. Electronic information exchange between emergency departments and
poison control centers: a Delphi study. Clin Toxicol (Phila). 2012;50(6):503-13. PMID: 22612793.
da Silva KR, Costa R, Crevelari ES, et al. Glocal clinical registries: pacemaker registry design and implementation
for global and local integration--methodology and case study. PLoS ONE. 2013;8(7):e71090. PMID: 23936257.
Exclusion: Not HIE
da Silva ME, Coeli CM, Ventura M, et al. Informed consent for record linkage: a systematic review. J Med Ethics.
2012;38(10):639-42. PMID: 22403083.
Exclusion: Not HIE
Damberg CL, Raube K, Teleki SS, et al. Taking stock of pay-for-performance: a candid assessment from the front
lines. Health Aff (Millwood). 2009;28(2):517-25. PMID: 19276011.
Exclusion: Not HIE
D'Amore JD, Mandel JC, Kreda DA, et al. Are Meaningful Use Stage 2 certified EHRs ready for interoperability?
Findings from the SMART C-CDA Collaborative. J Am Med Inform Assoc. 2014;21(6):1060-8. PMID: 24970839.
Exclusion: Not HIE
D-11
D'Amore JD, Sittig DF, Ness RB. How the continuity of care document can advance medical research and public
health. Am J Public Health. 2012;102(5):e1-4. PMID: 22420795.
D'Amore JD, Sittig DF, Wright A, et al. The promise of the CCD: challenges and opportunity for quality
improvement and population health. AMIA Annu Symp Proc. 2011;2011:285-94. PMID: 22195080.
Exclusion: Not HIE
Daniel GW, Ewen E, Willey VJ, et al. Efficiency and economic benefits associated with the use of a payer-based
electronic health record in an emergency department among a health insured population. Value Health.
2009;12(3):A14.
Exclusion: Not HIE
Darmon D, Sauvant R, Staccini P, et al. Which functionalities are available in the electronic health record systems
used by French general practitioners? An assessment study of 15 systems. Int J Med Inf. 2014;83(1):37-46. PMID:
24231269.
Exclusion: Not HIE
Daskalakis S, Katharaki M, Mantas J. The use of data envelopment analysis to measure the efficiency and
interoperability of information technology in Greek public healthcare organisations. Journal on Information
Technology in Healthcare. 2008;6(3):188-96.
Exclusion: Not HIE
Davidson SJ, Zwemer FL, Jr., Nathanson LA, et al. Where's the beef? The promise and the reality of clinical
documentation. Acad Emerg Med. 2004;11(11):1127-34. PMID: 15528575.
Exclusion: Not HIE
de Brantes F, Emery DW, Overhage JM, et al. The potential of HIEs as infomediaries. J Healthc Inf Manag.
2007;21(1):69-75. PMID: 17299928.
de la Torre I, Diaz FJ, Anton M, et al. Performance evaluation of a web-based system to exchange Electronic Health
Records using Queueing model (M/M/1). J Med Syst. 2012;36(2):915-24. PMID: 20703642.
Exclusion: Not HIE
Deas TM, Jr., Solomon MR. Health information exchange: foundation for better care. Gastrointest Endosc.
2012;76(1):163-8. PMID: 22726476.
Delano D. Roadmap of a successful local HIE: The Massachusetts eHealth Collaborative provides an instructive
success story. Health Manag Technol. 2011;32(9):20-1. PMID: 21961258.
Demski H, Hildebrand C, Brass A, et al. Improvement of cross-sector communication in the integrated health
environment. Stud Health Technol Inform. 2010;155:95-100. PMID: 20543315.
Department of Health and Human Services. Doctors and hospitals’ use of health IT more than doubles since 2012.
Washington, DC. Available at: http://www.hhs.gov/news/press/2013pres/05/20130522a.html. Accessed April 18,
2014.
Detmer D, Bloomrosen M, Raymond B, et al. Integrated personal health records: transformative tools for consumer-
centric care. BMC Med Inform Decis Mak. 2008;8:45. PMID: 18837999.
D-12
Detmer DE. Engineering information technology for actionable information and better health - balancing social
values through desired outcomes, complementary standards and decision-support. Stud Health Technol Inform.
2010;153:107-18. PMID: 20543241.
Exclusion: Not HIE
Devlies J, De Moor G, De Clercq E, et al. Health data exchange, health data sharing and decentralised clinical data
collections--recommendations from a Belgian expert group. Stud Health Technol Inform. 2008;141:162-212. PMID:
18953136.
Devoe JE, Gold R, Spofford M, et al. Developing a network of community health centers with a common electronic
health record: description of the Safety Net West Practice-based Research Network (SNW-PBRN). J Am Board Fam
Med. 2011;24(5):597-604. PMID: 21900444.
Exclusion: Not HIE
Devriendt E, Wellens N, Vesentini L, et al. BelRAI software for standardized data exchange between geriatric
health care organizations. Eur Geriatr Med. 2012;3:S71.
Dhopeshwarkar RV, Kern LM, O'Donnell HC, et al. Health care consumers' preferences around health information
exchange. Ann Fam Med. 2012;10(5):428-34. PMID: 22966106.
Dierker L. The state connection. State-level efforts in health information exchange. J AHIMA. 2008;79(5):40-3.
PMID: 18512425.
Dimick C. Varying privacy practices that pose barriers to health information exchange are putting HIM concerns in
the national spotlight. J AHIMA. 2007;78(10):29-33.
Dimick C. HISPC privacy and security collaborative hands off three years of work. J AHIMA. 2009;80(5):21-5.
PMID: 19507777.
Dimick C. Open for business: private networks create a marketplace for health information exchange. J AHIMA.
2012;83(5):22-6; quiz 7. PMID: 22670323.
Dimitropoulos L, Patel V, Scheffler SA, et al. Public attitudes toward health information exchange: perceived
benefits and concerns. Am J Manag Care. 2011;17(12 Spec No.):SP111-6. PMID: 22216769.
Exclusion: Not HIE
Dimitropoulos L, Rizk S. A state-based approach to privacy and security for interoperable health information
exchange. Health Aff. 2009;28(2):428-34. PMID: 19275999.
Disanti W, Rajapakse RO, Korelitz BI, et al. Incidence of neoplasms in patients who develop sustained leukopenia
during or after treatment with 6-mercaptopurine for inflammatory bowel disease. Clin Gastroenterol Hepatol.
2006;4(8):1025-9. PMID: 16765651.
Exclusion: Not HIE
Dixon BE. The perceived and real value of health information exchange in public health surveillance. Diss Abstr Int.
2014;75(5-B(E)).
D-13
Dixon BE, Grannis SJ, Revere D. Measuring the impact of a health information exchange intervention on provider-
based notifiable disease reporting using mixed methods: a study protocol. BMC Med Inform Decis Mak.
2013;13:121-. PMID: 24171799.
Dixon BE, Miller T, Overhage JM. Assessing HIE stakeholder readiness for consumer access: lessons learned from
the NHIN trial implementations. J Healthc Inf Manag. 2009;23(3):20-5. PMID: 19663160.
Exclusion: Not HIE
Dixon BE, Vreeman DJ, Grannis SJ. The long road to semantic interoperability in support of public health:
Experiences from two states. J Biomed Inform. 2014;49:3-8. PMID: 24680985.
Doarn CR, Nicogossian A. Policy implications of scholarly publications in health information technology. World
Med Health Policy. 2013;5(2):161-70.
DoBias M. RHIOs facing trouble: survey. Few physicians electronically sharing clinical data. Mod Healthc.
2007;37(50):32. PMID: 18203370.
Dobrev A, Stroetmann T, Veli N. Sources of financing and policy recommendations to Member States and the
European Commission on boosting eHealth investment. 2008. Available at: http://www.financing-
ehealth.eu/downloads/documents/feh_d5_3_final_study_report.pdf. Accessed April 22, 2015.
Doebbeling BN, Chou AF, Tierney WM. Priorities and strategies for the implementation of integrated informatics
and communications technology to improve evidence-based practice. J Gen Intern Med. 2006;21 Suppl 2:S50-7.
PMID: 16637961.
Exclusion: Not HIE
Dolin RH, Wiesenthal AM. National health information network cost and structure. Ann Intern Med.
2006;144(2):145; author reply 7. PMID: 16418420.
Donnelly J, Mussi J, Parisot C, et al. Building an interoperable regional health information network today with IHE
integration profiles. J Healthc Inf Manag. 2006;20(3):29-38. PMID: 16903659.
Dorr D, Bonner LM, Cohen AN, et al. Informatics systems to promote improved care for chronic illness: a literature
review. J Am Med Inform Assoc. 2007;14(2):156-63. PMID: 17213491.
Exclusion: Not HIE
Dowling AF. CHINS-the current state. Information Networks for Community Health. 1997:15-41.
Downing GJ, Zuckerman AE, Coon C, et al. Enhancing the quality and efficiency of newborn screening programs
through the use of health information technology. Semin Perinatol. 2010;34(2):156-62. PMID: 20207265.
Downs SM, van Dyck PC, Rinaldo P, et al. Improving newborn screening laboratory test ordering and result
reporting using health information exchange. J Am Med Inform Assoc. 2010;17(1):13-8. PMID: 20064796.
Duftschmid G, Wrba T, Gall W, et al. The strategic approach of managing healthcare data exchange in Austria.
Methods Inf Med. 2004;43(2):124-32. PMID: 15136861.
D-14
Dullabh P, Adler-Milstein J, Hovey L, et al. Key Challenges to Enabling Health Information Exchange and How
States Can Help. Office of the National Coordinator for Health Information Technology. 2014. Available at:
http://www.healthit.gov/sites/default/files/state_hie_evaluation_stakeholder_discussions.pdf. Accessed April 10,
2015.
Dullabh P, Adler-Milstein J, Nye C, et al. Evaluation of the State Health Information Exchange Cooperative
Agreement Program: Early Findings from a Review of Twenty-Seven States. Developed by NORC for the Office of
the National Coordinator for Health IT (ONC). Bethesda, MD: University of Chicago. 2012. Available at:
http://www.healthit.gov/sites/default/files/pdf/state-health-info-exchange-coop-program-evaluation.pdf. Accessed
December 16, 2014.
Dullabh P, Hovey L, Ubri P, et al. Evaluation of the State Health Information Exchange Cooperative Agreement
Program: Physician Experiences and Perceptions of Health Information Exchange February 2013 University of
Chicago: Office of the National Coordinator for Health Information Technology. 2012. Available at:
http://www.healthit.gov/sites/default/files/providerfocusgroupsynthesis_02_08_13.pdf. Accessed April 10, 2015.
Dullabh P, Hovey L, Ubri P, et al. Evaluation of the State Health Information Exchange Cooperative Agreement
Program: Case Study Synthesis: Experiences from Five States in Enabling HIE. Bethesda, MD: NORC at the
University of Chicago 2013. Availible at: http://healthit.gov/sites/default/files/casestudysynthesisdocument_2-8-
13.pdf. Accessed December 16, 2014.
Dullabh P, Milstein J, Nye C, et al. Evaluation of the State Health Information Exchange Cooperative Agreement
Program: Early Findings from a Review of Twenty-Seven States: January 2012. University of Chicago: Office of
the National Coordinator for Health Information Technology 2012. Available at:
http://www.healthit.gov/sites/default/files/pdf/state-health-info-exchange-coop-program-evaluation.pdf. Accessed
April 10, 2015.
Dullabh P, Moiduddin A, Nye C, et al. The Evolution of the State Health Information Exchange Cooperative
Agreement Program: State Plans to Enable Robust HIE: August 2011. NORC at the University of Chicago: Office
of the National Coordinator for Health Information Technology. 2011. Available at:
http://www.healthit.gov/sites/default/files/pdf/state-health-info-exchange-program-evolution.pdf. Accessed April
10, 2015.
Dullabh P, Ubri P, Loganathan S, et al. Evaluation of the State Health Information Exchange Cooperative
Agreement Program: State Approaches to Enabling HIE: Typology Brief. Office of the National Coordinator for
Health Information Technology. 2014. Available at:
http://www.healthit.gov/sites/default/files/statehietypologybrief.pdf. Accessed April 10, 2015.
Dupuits F. The role of community health information networks in disease management. Disease Management &
Health Outcomes. 2000;8(4):185-95.
Dykes P, Bakken S. National and regional health information infrastructures: making use of information technology
to promote access to evidence. Stud Health Technol Inform. 2004;107(Pt 2):1187-91. PMID: 15361000.
Exclusion: Not HIE
D-15
Eason K, Dent M, Waterson P, et al. Bottom-up and middle-out approaches to electronic patient information
systems: a focus on healthcare pathways. Inform Prim Care. 2012;20(1):51-6. PMID: 23336835.
Exclusion: Not HIE
Edwards A, Hollin I, Barry J, et al. Barriers to cross--institutional health information exchange: a literature review. J
Healthc Inf Manag. 2010;24(3):22-34. PMID: 20677469.
eHealth Initiative. Report on Health Information Exchange: Sustainable HIE in a Changing Landscape. 2011
eHealth Initiative. 2014 Results from Survey on Health Data Exchange. Available at:
http://www.ehidc.org/resource-center/publications/view_document/460-webinar-materials-2014-results-from-
survey-on-health-data-exchange. Accessed January 9, 2014.
Einbinder JS, Bates DW. Leveraging information technology to improve quality and safety. Yearb Med Inform.
2007:22-9. PMID: 17700900.
Exclusion: Not HIE
Ellingsen G, Monteiro E. Big is beautiful: electronic patient records in large Norwegian hospitals 1980s-2001.
Exclusion: Not HIE
Ellingsen G, Monteiro E, Roed K. Integration as interdependent workaround. Int J Med Inf. 2013;82(5):e161-9.
PMID: 23083928.
Exclusion: Not HIE
Elliott E, Bailey JE, Wan JY, et al. Does health information exchange use decrease duplicate imaging in the
emergency evaluation of back pain? J Gen Intern Med. 2011;26((Elliott E.; Bailey J.E.; Wan J.Y.; Pope R.A.;
Waters T.M.) Medicine and Preventive Medicine, University of Tennessee, Health Science Center, Memphis,
United States):S273.
Elnahal SM, Joynt KE, Bristol SJ, et al. Electronic health record functions differ between best and worst hospitals.
Am J Manag Care. 2011;17(4):e121-47. PMID: 21774097.
Exclusion: Not HIE
Ervin NE, Berry MM. Community readiness for a computer-based health information network. J N Y State Nurses
Assoc. 2006;37(1):5-11. PMID: 16929715.
Evans JM, Guthrie B, Pagliari C, et al. Do general practice characteristics influence uptake of an information
technology (IT) innovation in primary care? Inform Prim Care. 2008;16(1):3-8. PMID: 18534072.
Exclusion: Not HIE
Exchange IHI. 2010 Annual Report. 2010. Available at: http://mpcms.blob.core.windows.net/bd985247-f489-435f-

a7b4-49df92ec868e/docs/f42f53db-c797-4620-8dd2-97c9b04ba4f8/ihie-2010-annual-report.pdf. Accessed April 22,
2015.
Exeter DJ, Rodgers S, Sabel CE. "Whose data is it anyway(alpha)" The implications of putting small area-level
health and social data online. Health Policy. 2014;114(1):88-96. PMID: 23932285.
Exclusion: Not HIE
D-16
Eysenbach G. Infodemiology and infoveillance tracking online health information and cyberbehavior for public
health. Am J Prev Med. 2011;40(5 Suppl 2):S154-8. PMID: 21521589.
Exclusion: Not HIE
Fehrenbach SN, Kelly JC, Vu C. Integration of child health information systems: current state and local health
department efforts. J Public Health Manag Pract. 2004;Suppl:S30-5. PMID: 15643356.
Exclusion: Not HIE
Feldman SS, Horan MTA. Using the Nationwide Health Information Network to Deliver Value to Disability
Claimants: A case study of social security administration and MedVirginia use of MEGAHIT for disability
determination. Social Security Administration. Available at:
http://www.connectopensource.org/sites/connectopensource.org/files/CaseStudy_MedVA_SSA.pdf. Accessed
December 5, 2014.
Fernandes L, O'Connor M. Data governance and data stewardship. Critical issues in the move toward EHRs and
HIE. J AHIMA. 2009;80(5):36-9. PMID: 19507780.
Fernandez-Aleman JL, Seva-Llor CL, Toval A, et al. Free web-based personal health records: An analysis of
functionality. J Med Syst. 2013;37(6)PMID: 24221916.
Exclusion: Not HIE
Ferraccioli G, Salaffi F, Lapadula G. RHEUMA-CARD: Involvement of the patient through a secure systems access
into the treat to target strategy in rheumatology. Ann Rheum Dis. 2013;72(Suppl 3):A1025.
Ferraris VA, Saha SP, Davenport DL, et al. Thoracic surgery in the real world: does surgical specialty affect
outcomes in patients having general thoracic operations? Ann Thorac Surg. 2012;93(4):1041-7; discussion 7-8.
PMID: 22386087.
Exclusion: Not HIE
Ferratt TW, Lederer AL, Hall SR, et al. Surmounting health information network barriers: the greater Dayton area
experience. Health Care Manage Rev. 1998;23(1):70-6. PMID: 9494823.
Fidahussein M, Hook J, Kesterson J, et al. Using a regional health information exchange to improve identification of
post-discharge follow-up providers. J Gen Intern Med. 2011;26:S163-S4.
Figge HL. Interoperable health information exchange between medication therapy management services and the
medical home. Am J Health-Syst Pharm. 2010;67(3):190-1. PMID: 20101060.
Exclusion: Not HIE
Finch TL, Mair FS, May CR. Teledermatology in the UK: lessons in service innovation. Br J Dermatol.
2007;156(3):521-7. PMID: 17300243.
Exclusion: Not HIE
Fincham JE. Significant Potential for Health Information Exchange in Enhancing Quality of Care and Reducing
Hospital Admissions in the United States. Am Health Drug Benefits. 2012;5(6):340-1.
Fine AM, Goldmann DA, Forbes PW, et al. Incorporating vaccine-preventable disease surveillance into the National
Health Information Network: leveraging children's hospitals. Pediatr. 2006;118(4):1431-8. PMID: 17015533.
D-17
Finn Z, McNeill MH, Cooper LS, et al. Aligning HIE. A model to organize networks on core principles,
collaborative activities. J AHIMA. 2010;81(8):48-51. PMID: 20795532.
Finnell JT, Overhage JM, Dexter PR, et al. Community clinical data exchange for emergency medicine patients.
AMIA Annu Symp Proc. 2003:235-8. PMID: 14728169.
Exclusion: Not HIE
Fleischman W, Angiollilo J, Kuperman G, et al. Improving identification of frequent emergency department users
using a regional health information exchange. Acad Emerg Med. 2012;19:S47-S8.
Florence C, Shepherd J, Brennan I, et al. An economic evaluation of anonymised information sharing in a

partnership between health services, police and local government for preventing violence-related injury. Inj Prev.
2014;20(2):108-14.
Exclusion: Not HIE
Flynn D, Gregory P, Makki H, et al. Expectations and experiences of eHealth in primary care: a qualitative practice-
based investigation. Int J Med Inform. 2009;78(9):588-604. PMID: 19482542.
Exclusion: Not HIE
Follen M, Castaneda R, Mikelson M, et al. Implementing health information technology to improve the process of
health care delivery: a case study. Dis Manag. 2007;10(4):208-15. PMID: 17718659.
Exclusion: Not HIE
Fonkych K, Taylor R. The state and pattern of health information technology adoption: Rand Corporation; 2005.
Exclusion: Not HIE
Ford EW, Menachemi N, Phillips MT. Predicting the adoption of electronic health records by physicians: when will
health care be paperless? J Am Med Inform Assoc. 2006;13(1):106-12. PMID: 16221936.
Exclusion: Not HIE
Forland L. Evaluating the implementation of an electronic medical record system for a health organization-affiliated
family practice clinic: ProQuest; 2007.
Exclusion: Not HIE
Foster-Fishman PG, Salem DA, Allen NA, et al. Facilitating interorganizational collaboration: the contributions of
interorganizational alliances. Am J Community Psychol. 2001;29(6):875-905. PMID: 11800511.
Exclusion: Not HIE
Foundation of Research and Education AHIMA. State Level Health Information Exchange. Final Report Part I:
Roles in Ensuring Governance and Advancing Interoperablity. 2008. Available at:
http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_040348.pdf. Accessed December 5, 2014.
Foundation of Research and Education AHIMA. State Level Health Information Exchange. Final Report Part II:
Coordinating Policies That Impact Access, Use, and Control of Health Information. Executive Summary 2008.
Available at: http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_045661.pdf. Accessed
December 19, 2014.
Foundation of Research and Education of American Health Information Management Association. Development of
State Level Health Information Exchange Initiatives Final Report: Extension Tasks. 2007. Available at:
D-18
Foundation of Research and Education of American Health Information Management Association. State Level
Health Information Exchange Initiative Development Workbook: a Guide to Key Issues, Options and Strategies.
2007. Available at: http://library.ahima.org/xpedio/groups/public/documents/ahima/bok1_038398.pdf. Accessed
December 5, 2014.
Foundation of Research and Education of American Health Information Management Association. State Level
Health Information Exchange. Final Report Part II: Coordinating Policies That Impact Access, Use, and Control of
Health Information. 2008. Available at:
Foxhall K. Stating the case. A new report from AHIMA analyzes state-level health information exchanges. Healthc
Inform. 2006;23(11):24. PMID: 17144328.
Frady N. Healthcare collaboration for the 21st century: direct project. Tenn Med. 2013;106(8):32. PMID: 24027884.
Frame A, LaMantia M, Reddy Bynagari BB, et al. Development and Implementation of an Electronic Decision
Support to Manage the Health of a High-Risk Population: The enhanced Electronic Medical Record Aging Brain
Care Software (eMR-ABC). EGEMS (Wash DC). 2013;1(1)
Exclusion: Not HIE
Francis LP. The physician-patient relationship and a National Health Information network. J Law Med Ethics.
2010;38(1):36-49. PMID: 20446982.
Franklin BD, Reynolds M, Sadler S, et al. The effect of the electronic transmission of prescriptions on dispensing
errors and prescription enhancements made in English community pharmacies: a naturalistic stepped wedge study.
BMJ Qual Saf. 2014PMID: 24742778.
Exclusion: Not HIE
Friedman CP, Iakovidis I, Debenedetti L, et al. Across the Atlantic cooperation to address international challenges in
eHealth and health IT: managing toward a common goal. Int J Med Inform. 2009;78(11):778-84. PMID: 19734085.
Friedmann BE, Shapiro JS, Kannry J, et al. Analyzing workflow in emergency departments to prepare for health
information exchange. AMIA Annu Symp Proc. 2006:926. PMID: 17238545.
Exclusion: Not HIE
Frisse ME. State and community-based efforts to foster interoperability. Health Aff (Millwood). 2005;24(5):1190-6.
PMID: 16162562.
Frisse ME. Health information exchange in Memphis: impact on the physician-patient relationship. J Law Med
Ethics. 2010;38(1):50-7. PMID: 20446983.
Exclusion: Not HIE
Frisse ME, Holmes RL. Estimated financial savings associated with health information exchange and ambulatory
care referral. J Biomed Inform. 2007;40(6 Suppl):S27-32. PMID: 17942374.
Frisse ME, King JK, Rice WB, et al. A regional health information exchange: architecture and implementation.
D-19
Frohlich J, Karp S, Smith MD, et al. Retrospective: lessons learned from the Santa Barbara project and their
implications for health information exchange. Health Aff (Millwood). 2007;26(5):w589-91. PMID: 17670777.
Fu PC, Jr., Rosenthal D, Pevnick JM, et al. The impact of emerging standards adoption on automated quality
reporting. J Biomed Inform. 2012;45(4):772-81. PMID: 22820003.
Fuji KT, Gait KA, Siracuse MV, et al. Electronic health record adoption and use by Nebraska pharmacists. Perspect
Health Inf Manag. 2011;8:1d. PMID: 21796266.
Exclusion: Not HIE
Fung KW, Kayaalp M, Callaghan F, et al. Comparison of electronic pharmacy prescription records with manually
collected medication histories in an emergency department. Ann Emerg Med. 2013;62(3):205-11. PMID: 23688770.
Exclusion: Not HIE
Gagnon M, Legare F, Labrecque M, et al. Interventions for promoting information and communication technologies
adoption in healthcare professionals. Cochrane Database Syst Rev. 2009(2). PMID: 19160265.
Ganguly S, Kataria P, Juric R, et al. Sharing information and data across heterogeneous e-health systems. Telemed J
E Health. 2009;15(5):454-64. PMID: 19548826.
Exclusion: Not HIE
Garets DE. Why RHIOs aren't working: views from an American who can see White Rock, British Columbia, from
his backyard. Healthc Q. 2008;11(2):102-3. PMID: 18700271.
Garg N, Kuperman G, Onyile A, et al. Validating health information exchanges data for quality measurement across
four hospitals. Acad Emerg Med. 2014;21(5):S131.
Garrouste-Orgeas M, Timsit JF, Tafflet M, et al. Excess risk of death from intensive care unit-acquired nosocomial
bloodstream infections: a reappraisal.[Erratum appears in Clin Infect Dis. 2006 Jun 15;42(12):1818]. Clin Infect Dis.
2006;42(8):1118-26. PMID: 16575729.
Exclusion: Not HIE
Gaynor M, Lenert L, Wilson KD, et al. Why common carrier and network neutrality principles apply to the
Nationwide Health Information Network (NWHIN). J Am Med Inform Assoc. 2014;21(1):2-7. PMID: 23837992.
Geissbuhler A. Lessons learned implementing a regional health information exchange in Geneva as a pilot for the
Swiss national eHealth strategy. Int J Med Inf. 2013;82(5):e118-24. PMID: 23332387.
Genes N, Shapiro JS, Hwang U, et al. GEDI WISE: Notifications about geriatric ED visits via health information
exchange is feasible and may reduce admissions. Acad Emerg Med. 2014;21(5):S273.
Georgiou A, Tariq A, Westbrook JI. The temporal landscape of residential aged care facilities--implications for
context-sensitive health technology. Stud Health Technol Inform. 2013;194:69-74. PMID: 23941933.
Exclusion: Not HIE
Geurts MM, Ivens M, van Gelder E, et al. Development of a web-based pharmaceutical care plan to facilitate
collaboration between healthcare providers and patients. Inform Prim Care. 2013;21(1):53-9. PMID: 24629657.
D-20
Exclusion: Not HIE
Ghosh T, Marquard J. Development of Regional Health Information Organizations (RHIOs): Knowledge networks
and collaboration. Int J Public Pol. 2007;2(3-4):298-315.
Gichoya J, Gamache RE, Vreeman DJ, et al. An evaluation of the rates of repeat notifiable disease reporting and
patient crossover using a health information exchange-based automated electronic laboratory reporting system.
AMIA Annu Symp Proc. 2012;2012:1229-36. PMID: 23304400.
Exclusion: Not HIE
Glaser JP, DeBor G, Stuntz L. The New England Healthcare EDI Network. J Healthc Inf Manag. 2003;17(4):42-50.
PMID: 14558371.
Glickman SW, Kit Delgado M, Hirshon JM, et al. Defining and measuring successful emergency care networks: a
research agenda. Acad Emerg Med. 2010;17(12):1297-305. PMID: 21122011.
Goedert J. Governance: the HIE differentiator. Health Data Manag. 2009;17(8):26. PMID: 19697558.
Goedert J. Lesson from the HIE front. Organizations share lessons learned in the effort to develop health
information exchanges and regional health information organizations. Health Data Manag. 2009;17(2):28-30.
PMID: 19244811.
Gold MR, McLaughlin CG, Devers KJ, et al. Obtaining providers' 'buy-in' and establishing effective means of
information exchange will be critical to HITECH's success. Health Aff. 2012;31(3):514-26. PMID: 22392662.
Goldberg L, Lide B, Lowry S, et al. Usability and accessibility in consumer health informatics current trends and
future challenges. Am J Prev Med. 2011;40(5 Suppl 2):S187-97. PMID: 21521594.
Exclusion: Not HIE
Gordon P, Camhi E, Hesse R, et al. Processes and outcomes of developing a continuity of care document for use as a
personal health record by people living with HIV/AIDS in New York City. Int J Med Inf. 2012;81(10):e63-73.
PMID: 22841825.
Exclusion: Not HIE
Gore MJ. Gaining links: health information networks arise--with integration challenges. Clin Lab Sci. 1996;9(2):70-
7. PMID: 10163348.
Goroll AH, Simon SR, Tripathi M, et al. Community-wide implementation of health information technology: the
Massachusetts eHealth Collaborative experience. J Am Med Inform Assoc. 2009;16(1):132-9. PMID: 18952937.
Gottlieb LK, Stone EM, Stone D, et al. Regulatory and policy barriers to effective clinical data exchange: lessons
learned from MedsInfo-ED. Health Aff. 2005;24(5):1197-204. PMID: 16162563.
Grannis SJ, Biondich PG, Mamlin BW, et al. How disease surveillance systems can serve as practical building
blocks for a health information infrastructure: the Indiana experience. AMIA Annu Symp Proc. 2005:286-90. PMID:
16779047.
D-21
Grant RW, Wald JS, Schnipper JL, et al. Practice-linked online personal health records for type 2 diabetes mellitus:
a randomized controlled trial. Arch Intern Med. 2008;168(16):1776-82. PMID: 18779465.
Exclusion: Not HIE
Grantham D. Confidentiality alternatives for exchanging electronic medical records take shape. Behav Healthc.
2013;33(3):37-9. PMID: 23821917.
Graumlich JF, Novotny NL, Stephen Nace G, et al. Patient readmissions, emergency visits, and adverse events after
software-assisted discharge from hospital: cluster randomized trial. J Hosp Med. 2009;4(7):E11-9. PMID:
19479782.
Exclusion: Not HIE
Gravely SD, Whaley ES. The next step in health data exchanges: trust and privacy in exchange networks. J Healthc
Inf Manag. 2009;23(2):33-7. PMID: 19382738.
Green SD, Thomas JD. Interdisciplinary collaboration and the electronic medical record. Pediatric nursing.
2008;34(3):225-7, 40. PMID: 18649812.
Exclusion: Not HIE
Grinspan Z, Shapiro JS, Abramson EL, et al. Predicting frequent emergency department users among people with
epilepsy, VIA health information exchange. 1535-7597. New York, United States: Center for Healthcare
Informatics and Policy, Weill Cornell Medical Center. 2014. Available at:
http://www.embase.com/search/results?subaction=viewrecord&from=export&id=L71433373
https://www.aesnet.org/sites/default/files/file_attach/2013%20Abstract%20Supplement-14-1-s1.pdf
http://resolver.lib.washington.edu/resserv?sid=EMBASE&issn=15357597&id=doi:&atitle=Predicting+frequent+em
ergency+department+users+among+people+with+epilepsy%2C+VIA+health+information+exchange&stitle=Epilep
sy+Curr.&title=Epilepsy+Currents&volume=14&issue=&spage=262&epage=&aulast=Grinspan&aufirst=Zachary&
auinit=Z.&aufull=Grinspan+Z.&coden=&isbn=&pages=262-&date=2014&auinit1=Z&auinitm=. Accessed
December 9, 2014.
Grinspan ZM, Abramson EL, Banerjee S, et al. Potential value of health information exchange for people with
epilepsy: crossover patterns and missing clinical data. AMIA Annu Symp Proc. 2013;2013:527-36. PMID:
24551355.
Exclusion: Not HIE
Grinspan ZM, Abramson EL, Banerjee S, et al. People with epilepsy who use multiple hospitals; Prevalence and
associated factors assessed via a health information exchange. Epilepsia. 2014;55(5):734-45. PMID: 24598038.
Grinspan ZM, Berg L, Onyile A, et al. Medical information fragmentation for people with epilepsy in new york city
differs by type of visit. Epilepsy Currents. 2013;13:315.
Exclusion: Not HIE
Grossman JM, Cross DA, Boukus ER, et al. Transmitting and processing electronic prescriptions: experiences of
physician practices and pharmacies. J Am Med Inform Assoc. 2012;19(3):353-9. PMID: 22101907.
Exclusion: Not HIE
Guilbert TW, Arndt B, Temte J, et al. The theory and application of UW ehealth-PHINEX, a clinical electronic
health record-public health information exchange. WMJ. 2012;111(3):124-33. PMID: 22870558.
Exclusion: Not HIE
D-22
Gummadi S, Housri N, Zimmers TA, et al. Electronic medical record: A balancing act of patient safety, privacy and
health care delivery. Am J Med Sci. 2014;348(3):238-43. PMID: 24879530.
Exclusion: Not HIE
Haarbrandt B, Schwartze J, Gusew N, et al. Primary Care Providers' Acceptance of Health Information Exchange
Utilizing IHE XDS... International Conference on Informatics, Management, and Technology in Healthcare
(ICIMTH) Conference, July 5-7th, Athens, Greece. Stud Health Technol Inform. 2013;190:106-8.
Exclusion: Not HIE
Haarbrandt B, Schwartze J, Gusew N, et al. Primary Care Provider's Acceptance of Health Information Exchange
Utilizing IHE XDS. Stud Health Technol Inform. 2013;192:998. PMID: 23920772.
Exclusion: Not HIE
HACKL W, HOERBST A, AMMENWERTH E. The Electronic Health Record in Austria: Рhуsicians' Acceptance
Is Influenced by Negative Emotions. Stud Health Technol Inform. 2009;150:140-4. PMID: 19745284.
Exclusion: Not HIE
Hadjizacharia P, Green DJ, Plurad D, et al. Cocaine use in trauma: effect on injuries and outcomes. J Trauma.
2009;66(2):491-4. PMID: 19204526.
Exclusion: Not HIE
Haggstrom D, Myers LJ, French DD, et al. Impact of VA health information exchange upon the quality of diabetes
care. 0884-8734. Indianapolis, United States: VA Health Services Research and Development. 2014. Available at:
http://download.springer.com/static/pdf/904/art%253A10.1007%252Fs11606-014-2834-
9.pdf?auth66=1417820297_6efc82cb7a7326b3340d48892739e5e2&ext=.pdf. Accessed December 5, 2014.
Hagland M. Readying for the RHIO revolution. Behav Healthc. 2006;26(3):47-9. PMID: 16649645.
Halamka J, Aranow M, Ascenzo C, et al. Health care IT collaboration in Massachusetts: the experience of creating
regional connectivity. J Am Med Inform Assoc. 2005;12(6):596-601. PMID: 16049225.
Halamka J, Aranow M, Ascenzo C, et al. E-Prescribing collaboration in Massachusetts: early experiences from
regional prescribing projects. J Am Med Inform Assoc. 2006;13(3):239-44. PMID: 16501174.
Exclusion: Not HIE
Hall GC, McMahon AD, Dain MP, et al. Primary-care observational database study of the efficacy of GLP-1
receptor agonists and insulin in the UK. Diabet Med. 2013;30(6):681-6. PMID: 23330649.
Exclusion: Not HIE
Hanmer LA, Roode JD, Isaacs S. Modelling the effect of limited or vulnerable resources on the use of computerised
hospital information systems (CHISs) in South Africa. Stud Health Technol Inform. 2007;130:299-309. PMID:
17917203.
Exclusion: Not HIE
Hansagi H, Olsson M, Hussain A, et al. Is information sharing between the emergency department and primary care
useful to the care of frequent emergency department users? Eur J Emerg Med. 2008;15(1):34-9. PMID: 18180664.
Exclusion: Not HIE
Hargreaves JS. Will electronic personal health records benefit providers and patients in rural America? Telemed J E
Health. 2010;16(2):167-76. PMID: 20082592.
Exclusion: Not HIE
D-23
Harkavy H. Greater than the sum of the parts. Eighteen New York physician practices gain centralized patient
information database with ASP-hosted system. Health Manag Technol. 2004;25(7):40-2. PMID: 15283512.
Harno K, Ruotsalainen P. Sharable EHR systems in Finland. Stud Health Technol Inform. 2006;121:364-70. PMID:
17095834.
Exclusion: Not HIE
Haux R. Individualization, globalization and health--about sustainable information technologies and the aim of
medical informatics. Int J Med Inform. 2006;75(12):795-808. PMID: 16846748.
Exclusion: Not HIE
Hayward-Rowse L, Whittle T. A pilot project to design, implement and evaluate an electronic integrated care
pathway. J Nurs Manag. 2006;14(7):564-71. PMID: 17004967.
Exclusion: Not HIE
Hazard L, Miercort C, Gaffney D, et al. Local-regional radiation therapy after breast reconstruction: what is the
appropriate target volume? A case-control study of patients treated with electron arc radiotherapy and review of the
literature. Am J Clin Oncol. 2004;27(6):555-64. PMID: 15577432.
Exclusion: Not HIE
Heads T. Once In A Lifetime. Remain in Light. 1980

Exclusion: Wrong publication type
Hebel E, Middleton B, Shubina M, et al. Bridging the chasm: effect of health information exchange on volume of
laboratory testing. Arch Intern Med. 2012;172(6):517-9. PMID: 22450942.
Heimly V. Consent-based access to core EHR information: the SUMO-project. Stud Health Technol Inform.
2008;136:431-6. PMID: 18487769.
Heimly V, Berntsen KE. Consent-based access to core EHR information. Collaborative approaches in Norway.
Henderson J, Miller G, Britt H, et al. Effect of computerisation on Australian general practice: does it improve the
quality of care? Qual Prim Care. 2010;18(1):33-47. PMID: 20359411.
Exclusion: Not HIE
Herbst K, Littlejohns P, Rawlinson J, et al. Evaluating computerized health information systems: hardware, software
and human ware: experiences from the Northern Province, South Africa. J Public Health Med. 1999;21(3):305-10.
PMID: 10528958.
Exclusion: Not HIE
Herrin J, da Graca B, Nicewander D, et al. The effectiveness of implementing an electronic health record on
diabetes care and outcomes. Health Serv Res. 2012;47(4):1522-40. PMID: 22250953.
Exclusion: Not HIE
Hersh W. Health care information technology: progress and barriers. JAMA. 2004;292(18):2273-4. PMID:
15536117.
Exclusion: Not HIE
Hicken VN, Thornton SN, Rocha RA. Integration challenges of clinical information systems developed without a
shared data dictionary. Stud Health Technol Inform. 2004;107(Pt 2):1053-7. PMID: 15360973.
Exclusion: Not HIE
D-24
Hincapie A, Warholak T. The impact of health information exchange on health outcomes. Appl Clin Inform.
2011;2(4):499-507. PMID: 23616891.
Holman CD, Bass AJ, Rouse IL, et al. Population-based linkage of health records in Western Australia:
development of a health services research linked database. Aust N Z J Public Health. 1999;23(5):453-9. PMID:
10575763.
Holmquest DL. Another lesson from Santa Barbara. Health Aff. 2007;26(5):w592-4. PMID: 17670778.
Hoyle T, Swanson R. Assessing what child health information systems should be integrated: the Michigan
experience. J Public Health Manag Pract. 2004;Suppl:S66-71. PMID: 15643362.
Hripcsak G, Sengupta S, Wilcox A, et al. Emergency department access to a longitudinal medical record. J Am Med
Exclusion: Not HIE
Huang C, Behara RS, Goo J. Optimal information security investment in a Healthcare Information Exchange: An
economic analysis. Decis Support Syst. 2014;61:1-11.
Hufstader M, Furukawa M, Hogin E. E-prescribing trends in the United States: 2008-2012. Value Health.
2012;15(4):A25.
Exclusion: Not HIE
Hummel J, Gandara BK. Health Information Exchange and Care Coordination of Diabetic Patients Between
Medicine and Dentistry. Diabetes Spectr. 2011;24(4):205-10.
Johansen I, Rasmussen M. Electronic interchange of lab test orders and results between laboratories reduces errors
and gives full traceability. Stud Health Technol Inform. 2010;155:65-8. PMID: 20543311.
Johansson L, Wohed R, Kajbjer K. Medical informatics in a united and healthy Europe. The development of a
Swedish national information structure... XXIInd International Congress of the European Federation for Medical
Informatics. Stud Health Technol Inform. 2009;150:53-7.
Johnson KB, Gadd C. Playing smallball: approaches to evaluating pilot health information exchange systems. J
Biomed Inform. 2007;40(6 Suppl):S21-6. PMID: 17931981.
Jones JB, Shah NR, Bruce CA, et al. Meaningful use in practice using patient-specific risk in an electronic health
record for shared decision making. Am J Prev Med. 2011;40(5 Suppl 2):S179-86. PMID: 21521593.
Exclusion: Not HIE
Jones SS, Rudin RS, Perry T, et al. Health information technology: an updated systematic review with a focus on
meaningful use. Ann Intern Med. 2014;160(1):48-54. PMID: 24573664.
Joshi JK. Clinical Value-Add for Health Information Exchange (HIE). Internet Journal of Medical Informatics.
2011;6(1):1.
D-25
Just BH, Fabian DP, Webb LL, et al. Managing the integrity of patient identity in health information exchange. J
AHIMA. 2009;80(7):62-9. PMID: 19663149.
Exclusion: Not HIE
Kabachinski J. RHIO: the data saga continues. Biomed Instrum Technol. 2009;43(1):47-51. PMID: 19215168.
Karmel R, Gibson D. Event-based record linkage in health and aged care services data: a methodological innovation.
BMC Health Serv Res. 2007;7:154. PMID: 17892601.
Exclusion: Not HIE
Katehakis DG, Sfakianakis S, Tsiknakis M, et al. An infrastructure for Integrated Electronic Health Record services:
the role of XML (Extensible Markup Language). J Med Internet Res. 2001;3(1):E7. PMID: 11720949.
Katz SJ, Moyer CA, Cox DT, et al. Effect of a triage-based E-mail system on clinic resource use and patient and
physician satisfaction in primary care: a randomized controlled trial. J Gen Intern Med. 2003;18(9):736-44. PMID:
12950483.
Exclusion: Not HIE
Keet G. In or out? HIE patient consent 101: How to populate a successful HIE with the right data, while
simultaneously maintaining patient privacy and ensuring patient comfort. Health Manag Technol. 2012;33(6):19.
PMID: 22787948.
Kemper AR, Uren RL, Clark SJ. Adoption of electronic health records in primary care pediatric practices. Pediatr.
2006;118(1):e20-4. PMID: 16818534.
Exclusion: Not HIE
Kern L, Barron Y, Dhopeshwarkar R, et al. Health information exchange and quality of care. J Gen Intern Med.
2011;26((Kern L.; Barron Y.; Dhopeshwarkar R.; Kaushal R.) Weill Cornell Medical College, New York, United
States):S167.
Kern LM, Ancker JS, Abramson E, et al. Evaluating health information technology in community-based settings:
lessons learned. J Am Med Inform Assoc. 2011;18(6):749-53. PMID: 21807649.
Kern LM, Barron Y, Blair AJ, 3rd, et al. Electronic result viewing and quality of care in small group practices. J Gen
Intern Med. 2008;23(4):405-10. PMID: 18373137.
Exclusion: Not HIE
Kern LM, Dhopeshwarkar R, Barron Y, et al. Measuring the effects of health information technology on quality of
care: a novel set of proposed metrics for electronic quality reporting. Jt Comm J Qual Patient Saf. 2009;35(7):359-
69. PMID: 19634804.
Kern LM, Kaushal R. Health information technology and health information exchange in New York State: new
initiatives in implementation and evaluation. J Biomed Inform. 2007;40(6 Suppl):S17-20. PMID: 17945542.
Khan AS, Fleischauer A, Casani J, et al. The next public health revolution: public health information fusion and
social networks. Am J Public Health. 2010;100(7):1237-42. PMID: 20530760.
D-26
Khan S, Maclean CD, Littenberg B. The effect of the Vermont Diabetes Information System on inpatient and
emergency room use: results from a randomized trial. Health Outcomes Res Med. 2010;1(1):e61-e6. PMID:
20975923.
Exclusion: Not HIE
Khan WA, Hussain M, Afzal M, et al. Personalized-detailed clinical model for data interoperability among clinical
standards. Telemed J E Health. 2013;19(8):632-42. PMID: 23875730.
Kho AN, Hynes DM, Goel S, et al. CAPriCORN: Chicago Area Patient-Centered Outcomes Research Network. J
Am Med Inform Assoc. 2014;21(4):607-11. PMID: 24821736.
Exclusion: Not HIE
Kho AN, Lemmon L, Commiskey M, et al. Use of a regional health information exchange to detect crossover of
patients with MRSA between urban hospitals. J Am Med Inform Assoc. 2008;15(2):212-6. PMID: 18096903.
Exclusion: Not HIE
Khurshid A, Diana ML, Luce SD. Health information exchange: metrics to address quality of care and return on
investment. Perspect Health Inf Manag. 2012;9:1e. PMID: 22783153.
Kierkegaard P. eHealth in Denmark: a case study. J Med Syst. 2013;37(6):9991. PMID: 24166019.
Kilbridge PM, Classen DC. The informatics opportunities at the intersection of patient safety and clinical
informatics. J Am Med Inform Assoc. 2008;15(4):397-407. PMID: 18436896.
Kimura M, Nakayasu K, Ohshima Y, et al. SS-MIX: a ministry project to promote standardized healthcare
information exchange. Methods Inf Med. 2011;50(2):131-9. PMID: 21206962.
Exclusion: Not HIE
Kirkby KC. Psychiatric networks in Asia. Int Rev Psychiatry. 2008;20(5):409-12. PMID: 19012124.
Exclusion: Not HIE
Kittler AF, Carlson GL, Harris C, et al. Primary care physician attitudes towards using a secure web-based portal
designed to facilitate electronic communication with patients. Inform Prim Care. 2004;12(3):129-38. PMID:
15606985.
Exclusion: Not HIE
Kloss L. Health Information Exchange: State Level Challenges and Opportunities. American Health Informatics
Management Association. Betheda, MD. Available at: http://www.chita.org/downloads/Kloss.pdf. Accessed April
22, 2015.
Kluge E-HW. Secure e-Health: managing risks to patient health data. Int J Med Inf. 2007;76(5-6):402-6. PMID:
17084665.
Exclusion: Not HIE
Knaup P, Bott O, Kohl C, et al. Electronic patient records: moving from islands and bridges towards electronic
health records for continuity of care. Yearb Med Inform. 2007:34-46. PMID: 17700902.
Koff DA. Introducing Integrating the Healthcare Enterprise--Canada. Can Assoc Radiol J. 2005;56(4):225-31.
PMID: 16419374.
D-27
Exclusion: Not HIE
Kontos EZ, Emmons KM, Puleo E, et al. Communication inequalities and public health implications of adult social
networking site use in the United States. J Health Commun. 2010;15 Suppl 3:216-35. PMID: 21154095.
Exclusion: Not HIE
Korst LM, Aydin CE, Signer JMK, et al. Hospital readiness for health information exchange: development of
metrics associated with successful collaboration for quality improvement. Int J Med Inf. 2011;80(8):e178-88. PMID:
21330191.
Exclusion: Not HIE
Korst LM, Signer JMK, Aydin CE, et al. Identifying organizational capacities and incentives for clinical data-
sharing: the case of a regional perinatal information system. J Am Med Inform Assoc. 2008;15(2):195-7. PMID:
18096916.
Korzeniewski SJ, Grigorescu V, Copeland G, et al. Methodological innovations in data gathering: newborn
screening linkage with live births records, Michigan, 1/2007-3/2008. Matern Child Health J. 2010;14(3):360-4.
PMID: 19353254.
Exclusion: Not HIE
Kouroubali A, Starren J, Barrows RC, Jr., et al. Practical lessons in remote connectivity. Proc AMIA Annu Fall
Symp. 1997:335-9. PMID: 9357643.
Kralewski JE, Zink T, Boyle R. Factors influencing electronic clinical information exchange in small medical group
practices. J Rural Health. 2012;28(1):28-33. PMID: 22236312.
Exclusion: Not HIE
Kremer T. RHIO stabilizes finances: Rochester RHIO committee develops revenue plan to cover $3 million annual
operating cost. Health Manag Technol. 2011;32(9):18. PMID: 21961256.
Kretz JM. National health information network cost and structure. Ann Intern Med. 2006;144(2):145-6; author reply
7. PMID: 16418418.
Krist AH, Woolf SH, Frazier CO, et al. An electronic linkage system for health behavior counseling effect on
delivery of the 5A's. Am J Prev Med. 2008;35(5 Suppl):S350-8. PMID: 18929981.
Exclusion: Not HIE
Krohn R. The business end of HIE. Despite recent developments, exchanges face daunting obstacles to success. J
Healthc Inf Manag. 2010;24(1):6-7. PMID: 20077916.
Kruse SC, Regier V, Rheinboldt TK. Barriers Over Time to Full Implementation of Health Information Exchange in
the United States. JMIR Med Inform. 2014;2(2):e26.
Kukafka R, Khan SA, Hutchinson C, et al. Digital partnerships for health: steps to develop a community-specific
health portal aimed at promoting health and well-being. AMIA Annu Symp Proc. 2007:428-32. PMID: 18693872.
Exclusion: Not HIE
Kuo M-H, Kushniruk AW, Borycki EM, et al. National strategies for health data interoperability. Stud Health
Technol Inform. 2011;164:238-42. PMID: 21335717.
D-28
Kuperman GJ. Health-information exchange: why are we doing it, and what are we doing? J Am Med Inform Assoc.
2011;18(5):678-82. PMID: 21676940.
Kuperman GJ, McGowan JJ. Potential unintended consequences of health information exchange. J Gen Intern Med.
2013;28(12):1663-6. PMID: 23690236.
Kurland LT, Molgaard CA. The patient record in epidemiology. Sci Am. 1981;245(4):54-63. PMID: 7027437.
Exclusion: Not HIE
Kussaibi H, Macary F, Kennedy M, et al. HL7 CDA implementation guide for structured anatomic pathology
reports methodology and tools. Stud Health Technol Inform. 2010;160(Pt 1):289-93. PMID: 20841695.
Exclusion: Not HIE
Laborde DV, Griffin JA, Smalley HK, et al. A framework for assessing patient crossover and health information
exchange value. J Am Med Inform Assoc. 2011;18(5):698-703. PMID: 21705458.
Lagoe RJ, Westert GP. Community wide electronic distribution of summary health care utilization data. BMC Med
Inform Decis Mak. 2006;6:17. PMID: 16549023.
Exclusion: Not HIE
Lammers EJ, Adler-Milstein J, Kocher KE. Effect of health information exchange on repeat imaging in the
emergency department. Acad Emerg Med. 2013;20(5):S15.
Landman AB, Lee CH, Sasson C, et al. Prehospital electronic patient care report systems: early experiences from
emergency medical services agency leaders. PLoS ONE. 2012;7(3):e32692. PMID: 22403698.
Exclusion: Not HIE
Lang RD. Hometown heroes: Small-town Doylestown Hospital has earned distinction for implementing a successful
HIE. Health Manag Technol. 2012;33(10):16-7.
Lapsia V, Lamb K, Yasnoff WA. Where should electronic records for patients be stored? Int J Med Inf.
2012;81(12):821-7. PMID: 23021932.
Lassila KS, Pemble KR, DuPont LA, et al. Assessing the impact of community health information networks: a
multisite field study of the Wisconsin Health Information Network. Top Health Inf Manage. 1997;18(2):64-76.
PMID: 10174731.
Exclusion: Not HIE
Lawrence D. RHIO or not--it works. A pilot program on medication histories in EDs is first for Vermont Health
Information Exchange. Healthc Inform. 2007;24(9):46. PMID: 17927065.
Leao BF, Bernardes MM, Levin J, et al. The Brazilian National Health Informatics Strategy. Stud Health Technol
Inform. 2001;84(Pt 1):38-42. PMID: 11604702.
Lee L, Whitcomb K, Galbreth M, et al. A strong state role in the HIE. Lessons from the South Carolina Health
Information Exchange. J AHIMA. 2010;81(6):46-50; quiz 1. PMID: 20614703.
D-29
Lee M, Gatton TM. Wireless health data exchange for home healthcare monitoring systems. Sensors (Basel).
2010;10(4):3243-60. PMID: 22319296.
Legg M. Standardisation of test requesting and reporting for the electronic health record. Clin Chim Acta.
2014;432:148-56. PMID: 24333615.
Lemay NV, Sullivan T, Jumbe B, et al. Reaching remote health workers in Malawi: Baseline assessment of a pilot
mHealth intervention. J Health Commun. 2012;17(Suppl 1):105-17. PMID: 22548604.
Exclusion: Not HIE
Lemieux-Charles L, Chambers LW, Cockerill R, et al. Evaluating the effectiveness of community-based dementia
care networks: the Dementia Care Networks' Study. Gerontologist. 2005;45(4):456-64. PMID: 16051908.
Exclusion: Not HIE
Lester WT, Grant RW, Barnett GO, et al. Randomized controlled trial of an informatics-based intervention to
increase statin prescription for secondary prevention of coronary disease. J Gen Intern Med. 2006;21(1):22-9.
PMID: 16423119.
Exclusion: Not HIE
Leventhal R. Health Information Exchange: Moving Forward or Stuck in Neutral? Healthc Inform. 2014;31(3):14-
20. PMID: 24941600.
Leventhal R. Sutter health goes next-level with data exchange. Healthc Inform. 2014;31(5):36-7. PMID: 25230451.
Levin-Epstein M. Health information exchanges not ready for prime time. Manag Care. 2014;23(6):31-5. PMID:
25109045.
Lewis SH, Holtry RS, Loschen WA, et al. The collaborative experience of creating the National Capital Region
Disease Surveillance Network. J Public Health Manag Pract. 2011;17(3):248-54. PMID: 21464687.
Li J-s, Zhou T-s, Chu J, et al. Design and development of an international clinical data exchange system: the
international layer function of the Dolphin Project. J Am Med Inform Assoc. 2011;18(5):683-9. PMID: 21571747.
Li YC, Kuo HS, Jian WS, et al. Building a generic architecture for medical information exchange among healthcare
providers. Int J Med Inf. 2001;61(2-3):241-6. PMID: 11311678.
Lichtner V, Galliers JR, Wilson S. A pragmatics' view of patient identification. Qual Saf Health Care. 2010;19
Suppl 3:i13-9. PMID: 20513792.
Exclusion: Not HIE
Lim C, Dokmak S, Cauchy F, et al. Selective policy of no drain after pancreaticoduodenectomy is a valid option in
patients at low risk of pancreatic fistula: a case-control analysis. World J Surg. 2013;37(5):1021-7. PMID:
23412469.
Exclusion: Not HIE
Little L. Privacy, trust, and identity issues for ubiquitous computing. Soc Sci Comput Rev. 2008;26(1):3-5.
D-30
Liu C-F, Hwang H-G, Chang H-C. E-healthcare maturity in Taiwan. Telemed J E Health. 2011;17(7):569-73.
PMID: 21718093.
Exclusion: Not HIE
Liu D, Wang X, Pan F, et al. Web-based infectious disease reporting using XML forms. Int J Med Inf.
2008;77(9):630-40. PMID: 18060833.
Exclusion: Not HIE
Liu GC, Cooper JG, Schoeffler KM, et al. Standards for the electronic health record, emerging from health care's
Tower of Babel. Proc AMIA Symp. 2001:388-92. PMID: 11825216.
Exclusion: Not HIE
Liu S, Zhou B, Xie G, et al. Beyond regional health information exchange in China: a practical and industrial-
strength approach. AMIA Annu Symp Proc. 2011;2011:824-33. PMID: 22195140.
Lluch M. Healthcare professionals' organisational barriers to health information technologies-a literature review. Int
J Med Inform. 2011;80(12):849-62. PMID: 22000677.
Lobach DF, Silvey GM, Willis JM, et al. Coupling direct collection of health risk information from patients through
kiosks with decision support for proactive care management. AMIA Annu Symp Proc. 2008:429-33. PMID:
18999181.
Exclusion: Not HIE
Lomax A, Grossmann M, Cozzi L, et al. The exchange of radiotherapy data as part of an electronic patient-referral
system. Int J Radiat Oncol Biol Phys. 2000;47(5):1449-56. PMID: 10889401.
M. W S. Health information exchange can save money by reducing admissions from the emergency department.
AHRQ Research Activities. 2012(381):13.
Mabry LM, Bailey JE, Wan J, et al. Health information exchange use improves adherence with evidence-based
guidelines for neuroimaging in the emergency evaluation of headache. J Investig Med. 2011;59(2):533.
MacFarlane A, Murphy AW, Clerkin P. Telemedicine services in the Republic of Ireland: an evolving policy
context. Health Policy. 2006;76(3):245-58. PMID: 16026889.
Exclusion: Not HIE
MacPhail LH, Neuwirth EB, Bellows J. Coordination of diabetes care in four delivery models using an electronic
health record. Med Care. 2009;47(9):993-9. PMID: 19648836.
Exclusion: Not HIE
Maenpaa T, Suominen T, Asikainen P, et al. The outcomes of regional healthcare information systems in health
care: a review of the research literature. Int J Med Inf. 2009;78(11):757-71. PMID: 19656719.
Maffei R, Burciago D, Dunn K. Determining business models for financial sustainability in regional health
information organizations (RHIOs): a review. Popul Health Manag. 2009;12(5):273-8. PMID: 19848569.
Maglogiannis I, Constantinos D, Kazatzopoulos L. Enabling collaborative medical diagnosis over the Internet via
peer-to-peer distribution of electronic health records. J Med Syst. 2006;30(2):107-16. PMID: 16705995.
D-31
Magnuson JA, Klockner R, Ladd-Wilson S, et al. Security aspects of electronic data interchange between a state
health department and a hospital emergency department. J Public Health Manag Pract. 2004;10(1):70-6. PMID:
15018344.
Mahon BE, Shea KM, Dougherty NN, et al. Implications for registry-based vaccine effectiveness studies from an
evaluation of an immunization registry: a cross-sectional study. BMC Public Health. 2008;8:160. PMID: 18479517.
Exclusion: Not HIE
Maiorana A, Steward WT, Koester KA, et al. Trust, confidentiality, and the acceptability of sharing HIV-related
patient data: lessons learned from a mixed methods study about Health Information Exchanges. Implement Sci.
2012;7:34. PMID: 22515736.
Exclusion: Not HIE
Mancuso M. Collaborating our way into interoperability. Health Manag Technol. 2014;35(6):24. PMID: 25058982.
Mandl KD, Mandel JC, Murphy SN, et al. The SMART Platform: early experience enabling substitutable
applications for electronic health records. J Am Med Inform Assoc. 2012;19(4):597-603. PMID: 22427539.
Mantzana V, Themistocleous M, Morabito V, et al. Evaluating actors and factors associated with healthcare
information systems. Evaluating Information Systems: Public and Private Sector2008:179-98.
Exclusion: Not HIE
Manzotti A, Chemello C, Pullen C, et al. Computer-assisted total knee arthroplasty after prior femoral fracture
without hardware removal. Orthopedics. 2012;35(10 Suppl):34-9. PMID: 23026250.
Exclusion: Not HIE
Marchibroda JM. The impact of health information technology on collaborative chronic care management. J Manage
Care Pharm. 2008;14(2 Suppl):S3-11. PMID: 18331114.
Marelli C, Gunnarsson C, Ross S, et al. Statins and risk of cancer: a retrospective cohort analysis of 45,857 matched
pairs from an electronic medical records database of 11 million adult Americans. J Am Coll Cardiol.
2011;58(5):530-7. PMID: 21777752.
Exclusion: Not HIE
Marquard J, Brennan PF, Grindrod D, et al. Health information exchange networks: understanding stakeholder
views. AMIA Annu Symp Proc. 2005:1044. PMID: 16779331.
Marrs KA, Kahn MG. Extending a clinical repository to include multiple sites. Proc Annu Symp Comput Appl Med
Care. 1995:387-91. PMID: 8563308.
Marshall GF, Gillespie W, Fox SJ. Privacy and security in Pennsylvania: ensuring privacy and security of health
information exchange in Pennsylvania. J Healthc Inf Manag. 2009;23(2):38-44. PMID: 19382739.
Martiez I, Escayola J, Martinez-Espronceda M, et al. Seamless integration of ISO/IEEE11073 personal health

devices and ISO/EN13606 electronic health records into an end-to-end interoperable solution. Telemed J E Health.
2010;16(10):993-1004. PMID: 21087123.
D-32
Martin Z. Virginia RHIO taking baby steps. Health Data Manag. 2007;15(2):120. PMID: 17375855.
Masi M, Pugliese R, Tiezzi F. Security analysis of standards-driven communication protocols for healthcare
scenarios. J Med Syst. 2012;36(6):3695-711. PMID: 22447202.
Mastebroek M, Naaldenberg J, Lagro-Janssen AL, et al. Health information exchange in general practice care for
people with intellectual disabilities--a qualitative review of the literature. Res Dev Disabil. 2014;35(9):1978-87.
PMID: 24864050.
Exclusion: Not HIE
McBride M. Health information exchange will improve quality of patient care, physicians believe. Ophthalmology
Times. 2012;37(21):89.
McCarter D, Lenart D. A "most-wired" hospital targets information sharing. Process improvements include faster
communication of key patient indicators. Nurs Manage. 2007;Suppl:24, 6, 32. PMID: 18159650.
Exclusion: Not HIE
McCormick D, Bor DH, Woolhandler S, et al. Giving office-based physicians electronic access to patients' prior
imaging and lab results did not deter ordering of tests. Health Aff (Millwood). 2012;31(3):488-96. PMID:
22392659.
Exclusion: Not HIE
McCray JC. Delivering health information statewide via the Internet in a collaborative environment: impact on
individual member institutions. Bull Med Libr Assoc. 1999;87(3):264-9. PMID: 10427425.
Exclusion: Not HIE
McCray JC, Maloney K. Improving access to knowledge-based health sciences information: early results from a
statewide collaborative effort. Bull Med Libr Assoc. 1997;85(2):136-40. PMID: 9160149.
Exclusion: Not HIE
McCullough JC. The Adoption and Use of Health Information Technologies in Three Settings.
[Dissertation].University of California, Los Angeles. Los Angeles. Available at:
https://escholarship.org/uc/item/6br9w3dm#page-1. Accessed April 22, 2015.
McCullough JC. The adoption and use of health Information Technologies in three settings. Diss Abstr Int.
2014;75(3-B(E)):No Pagination Specified.
McDonald C. Protecting patients in health information exchange: a defense of the HIPAA privacy rule. Health Aff.
2009;28(2):447-9. PMID: 19276002.
McDonald C, Overhage J, Barnes M, et al. The Indiana Network for Patient Care: a working local health
information infrastructure. Health Aff. 2005;24:1214-20. PMID: 16162565.
McDonald CJ, Schadow G, Barnes M, et al. Open Source software in medical informatics--why, how and what. Int J
Med Inform. 2003;69(2-3):175-84. PMID: 12810121.
Exclusion: Not HIE
D-33
McGowan J, Evans J, Michl K. Networking a need: a cost-effective approach to statewide health information
delivery. Proc Annu Symp Comput Appl Med Care. 1995:571-5. PMID: 8563350.
Exclusion: Not HIE
McGowan J, Kuperman G, Olinger L, et al. Strengthening Health Information Exchange: Final Report HIE
Unintended Consequences Work Group. Rockville, MD: Westat. 2012. Available at:
http://www.healthit.gov/sites/default/files/hie_uc_workgroup_final_report.pdf. Accessed January 7, 2014.
McGraw D, Dempsey JX, Harris L, et al. Privacy as an enabler, not an impediment: building trust into health
information exchange. Health Aff. 2009;28(2):416-27. PMID: 19275998.
McKenna R. Using information and communications technology to enable the exchange of information between
New Zealand clinicians and health providers. N Z Med J. 2010;123(1314):92-104. PMID: 20581917.
Exclusion: Not HIE
Meade B, Buckley D, Boland M. What factors affect the use of electronic patient records by Irish GPs? Int J Med
Inform. 2009;78(8):551-8. PMID: 19375381.
Exclusion: Not HIE
Mears GD, Rosamond WD, Lohmeier C, et al. A link to improve stroke patient care: a successful linkage between a
statewide emergency medical services data system and a stroke registry. Acad Emerg Med. 2010;17(12):1398-404.
PMID: 21122025.
Mehrotra A, Pearson SD, Coltin KL, et al. The response of physician groups to P4P incentives. Am J Manag Care.
2007;13(5):249-55. PMID: 17488190.
Exclusion: Not HIE
Mendelson DS, Bak PRG, Menschik E, et al. Informatics in radiology: image exchange: IHE and the evolution of
image sharing. Radiographics. 2008;28(7):1817-33. PMID: 18772272.
Meter RK. The Synapse health information network. Linking Nebraska and the midwest. Ann N Y Acad Sci.
1992;670:98-100. PMID: 1309108.
Exclusion: Not HIE
Metsemakers JF, Hoppener P, Knottnerus JA, et al. Computerized health information in The Netherlands: a
registration network of family practices. Br J Gen Pract. 1992;42(356):102-6. PMID: 1493025.
Exclusion: Not HIE
Metz JM, Coyle C, Hudson C, et al. An Internet-based cancer clinical trials matching resource. J Med Internet Res.
2005;7(3):e24. PMID: 15998615.
Exclusion: Not HIE
Milburn JA, Driver CP, Youngson GG, et al. The accuracy of clinical data: a comparison between central and local
data collection. Surgeon. 2007;5(5):275-8. PMID: 17958226.
Exclusion: Not HIE
Miller RH, Sim I. Physicians' use of electronic medical records: barriers and solutions. Health Aff. 2004;23(2):116-
26. PMID: 15046136.
Exclusion: Not HIE
D-34
Miriovsky BJ, Shulman LN, Abernethy AP. Importance of health information technology, electronic health records,
and continuously aggregating data to comparative effectiveness research and learning health care. J Clin Oncol.
2012;30(34):4243-8. PMID: 23071233.
Moehr JR, McDaniel JG. Adoption of security and confidentiality features in an operational community health
information network: the Comox Valley experience--case example. Int J Med Inf. 1998;49(1):81-7. PMID: 9723805.
Montori VM, Dinneen SF, Gorman CA, et al. The impact of planned care and a diabetes electronic management
system on community-based diabetes care: the Mayo Health System Diabetes Translation Project. Diabetes Care.
2002;25(11):1952-7. PMID: 12401738.
Exclusion: Not HIE
Morris G, Afzal S, Bhasker M, et al. Health Information Exchange Driven Subscription and Notification Services:
Market Assessment and Policy Considerations. Office of the National Coordinator for Health Information
Technology. 2012. Availible at: Accessed April 10, 2015.
Morris G, Afzal S, Finney D. Consumer Engagement in Health Information Exchange. Office of the National
Coordinator for Health Information Technology. 2012
Morrissey J. The evolution of a CHIN (community health information network). Mod Healthc. 2000;Suppl:42-3.
PMID: 11067123.
Morrissey J. Health information exchange. Hosp Health Netw. 2011;85(2):22-7. PMID: 21485258.
Mostashari F, Tripathi M, Kendall M. A tale of two large community electronic health record extension projects.
Health Aff. 2009;28(2):345-56. PMID: 19275989.
Exclusion: Not HIE
Munck LK, Hansen KR, Grethe Molbak A, et al. The use of shared medication record as part of medication
reconciliation at hospital admission is feasible. Dan Med J. 2014;61(5). PMID: 24814735.
Munoz RT, Fox MD, Gomez MR. Presumed consent models and health information exchanges: hard nudges and
ambiguous benefits. Am J Bioeth. 2013;13(6):14-5. PMID: 23641837.
Muscatello DJ, Churches T, Kaldor J, et al. An automated, broad-based, near real-time public health surveillance
system using presentations to hospital Emergency Departments in New South Wales, Australia. BMC Public Health.
2005;5:141. PMID: 16372902.
Exclusion: Not HIE
Myers JS, Shannon RP. Chasing high performance: best business practices for using health information technology
to advance patient safety. Am J Manag Care. 2012;18(4):e121-5. PMID: 22554037.
Myneni S, Patel VL. Assessment of collaboration and interoperability in an information management system to
support bioscience research. AMIA Annu Symp Proc. 2009;2009:463-7. PMID: 20351900.
Exclusion: Not HIE
D-35
Nagy PG, Pierce B, Otto M, et al. Quality Control Management and Communication Between Radiologists and
Technologists. J Am Coll Radiol. 2008;5(6):759-65. PMID: 18514956.
Exclusion: Not HIE
Navas H, Lopez Osornio A, Gambarte L, et al. Implementing rules to improve the quality of concept post-
coordination with SNOMED CT. Stud Health Technol Inform. 2010;160(Pt 2):1045-9. PMID: 20841843.
Exclusion: Not HIE
Neame R. Privacy and security issues in a wide area health communications network. Int J Biomed Comput.
1996;43(1-2):123-7. PMID: 8960932.
Nesbitt TS, Dharmar M, Katz-Bell J, et al. Telehealth at UC Davis--a 20-year experience. Telemed J E Health.
2013;19(5):357-62. PMID: 23343257.
Exclusion: Not HIE
Newgard C, Malveau S, Staudenmayer K, et al. Evaluating the use of existing data sources, probabilistic linkage,
and multiple imputation to build population-based injury databases across phases of trauma care. Acad Emerg Med.
2012;19(4):469-80. PMID: 22506952.
Exclusion: Not HIE
Newgard CD, Zive D, Malveau S, et al. Developing a statewide emergency medical services database linked to
hospital outcomes: a feasibility study. Prehosp Emerg Care. 2011;15(3):303-19. PMID: 21612384.
Exclusion: Not HIE
Nguile-Makao M, Zahar J-R, Francais A, et al. Attributable mortality of ventilator-associated pneumonia: respective
impact of main characteristics at ICU admission and VAP onset using conditional logistic regression and multi-state
models. Intensive Care Med. 2010;36(5):781-9. PMID: 20232046.
Exclusion: Not HIE
Nicholson C, Jackson C, Tweeddale M, et al. International exchange. Electronic patient records: achieving best
practice in information transfer between hospital and community providers -- an integration success story. Qual
Prim Care. 2003;11(3):233-40.
Exclusion: Not HIE
Nirel N, Rosen B, Sharon A, et al. The impact of an integrated hospital-community medical information system on
quality and service utilization in hospital departments. Int J Med Inform. 2010;79(9):649-57. PMID: 20655276.
Exclusion: Not HIE
Noblin AM. Privacy policy analysis for health information networks and regional health information organizations.
Health Care Manag (Frederick). 2007;26(4):331-40. PMID: 17992107.
Nocella KC, Horowitz KJ, Young JJ. Against all odds: designing and implementing a grassroots, community-
designed RHIO in a rural region. J Healthc Inf Manag. 2008;22(2):34-41. PMID: 19266993.
Exclusion: Not HIE
NORC. Evaluation of the State Health Information Exchange Cooperative Agreement Program: Case Study Report:
Experiences from Maine in Enabling Health Information Exchange (HIE). University of Chicago: Office of the
National Coordinator for Health Information Technology. 2012. Available at:
http://www.healthit.gov/sites/default/files/me_casestudyreportfinal.pdf. Accessed April 10, 2015.
D-36
Experiences from Nebraska in Enabling Health Information Exchange (HIE). University of Chicago: Office of the
http://www.healthit.gov/sites/default/files/ne_casestudyreport_final.pdf. Accessed April 10, 2015.
Experiences from Texas in Enabling Health Information Exchange (HIE) University of Chicago: Office of the
http://www.healthit.gov/sites/default/files/tx_casestudyreport_final.pdf. Accessed April 10, 2015.
Experiences from Washington State in Enabling Health Information Exchange (HIE) University of Chicago: Office
of the National Coordinator for Health Information Technology 2012. Available at:
http://www.healthit.gov/sites/default/files/wa_casestudyreport_final.pdf. Accessed April 10, 2015.
Experiences from Wisconsin in Enabling Health Information Exchange (HIE). University of Chicago: Office of the
http://www.healthit.gov/sites/default/files/wicasestudyreport_final.pdf. Accessed April 10, 2015.
Noss B, Zall RJ. A review of CHIN initiatives: what works and why. J Healthc Inf Manag. 2002;16(2):35-9. PMID:
11941918.
Nykanen P, Karimaa E. Evaluation during design of a regional seamless network of social and health care services--
information technology perspective. Stud Health Technol Inform. 2002;90:539-42. PMID: 15460751.
Exclusion: Not HIE
O'Donnell HC, Patel V, Kern LM, et al. Healthcare consumers' attitudes towards physician and personal use of
health information exchange. J Gen Intern Med. 2011;26(9):1019-26. PMID: 21584839.
Exclusion: Not HIE
Office of the National Coordinator for Health Information Technology DoH, Human S. 2014 Edition Release 2
Electronic Health Record (EHR) certification criteria and the ONC HIT Certification Program; regulatory
flexibilities, improvements, and enhanced health information exchange. Final rule. Fed Regist. 2014;79(176):54429-
80. PMID: 25233533.
Office of the National Coordinator for Health Information Technology DoH, Human S. Medicare and Medicaid
Programs; Electronic Health Record Incentive Program—Stage 3; 2015 Edition Health Information Technology
(Health IT) Certification Criteria, 2015 Edition Base Electronic Health Record (EHR) Definition, and ONC Health
IT Certification Program Modifications; Proposed Rules. March 30, 2015 2015. Available at:
http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/Stage3_Rule.pdf.
Ogunyemi OI, Meeker D, Kim H-E, et al. Identifying appropriate reference data models for comparative
effectiveness research (CER) studies based on data from clinical information systems. Med Care. 2013;51(8 Suppl
3):S45-52. PMID: 23774519.
D-37
Ohno-Machado L, Agha Z, Bell DS, et al. pSCANNER: patient-centered Scalable National Network for
Effectiveness Research. J Am Med Inform Assoc. 2014;21(4):621-6. PMID: 24780722.
Exclusion: Not HIE
O'Leary KJ, Liebovitz DM, Feinglass J, et al. Creating a better discharge summary: improvement in quality and
timeliness using an electronic discharge summary. J Hosp Med. 2009;4(4):219-25. PMID: 19267397.
Exclusion: Not HIE
Oliveira IC, Cunha JPS. Integration services to enable regional shared electronic health records. Stud Health
Technol Inform. 2011;169:310-4. PMID: 21893763.
Oliver AL, Montgomery K. A network approach to outpatient service delivery systems: resources flow and system
influence. Health Serv Res. 1996;30(6):771-89. PMID: 8591929.
Exclusion: Not HIE
Oliver N, Sohrab S. Connecting the disconnected: what FSM is doing? Pac Health Dialog. 2010;16(1):137-40.
PMID: 20968246.
Olsen J, Baisch MJ. An integrative review of information systems and terminologies used in local health
departments. J Am Med Inform Assoc. 2014;21(e1):e20-7. PMID: 24036156.
Exclusion: Not HIE
Olson KL, Grannis SJ, Mandl KD. Privacy protection versus cluster detection in spatial epidemiology. Am J Public
Health. 2006;96(11):2002-8. PMID: 17018828.
Exclusion: Not HIE
Onyile A, Shapiro JS, Kuperman G. Patient crossover rates vary by disease in a health information exchange. Ann
Emerg Med. 2011;58(4):S294-S5.
Orlova AO, Dunnagan M, Finitzo T, et al. Electronic health record - public health (EHR-PH) system prototype for
interoperability in 21st century healthcare systems. AMIA Annu Symp Proc. 2005:575-9. PMID: 16779105.
Orphanoudakis S. HYGEIAnet: the integrated regional health information network of Crete. Stud Health Technol
Inform. 2004;100:66-78. PMID: 15718565.
Overhage JM, Dexter PR, Perkins SM, et al. A randomized, controlled trial of clinical information shared from
another institution. Ann Emerg Med. 2002;39(1):14-23. PMID: 11782726.
Exclusion: Not HIE
Overhage JM, Tierney WM, McDonald CJ. Design and implementation of the Indianapolis Network for Patient
Care and Research. Bull Med Libr Assoc. 1995;83(1):48-56. PMID: 7703939.
Ozkaynak M, Marquard J, Hsieh Y, et al. Are lay people ready for health information exchange? AMIA Annu Symp
Proc. 2007:1065. PMID: 18694163.
Exclusion: Not HIE
Page D. Health information exchanges hold promise, pose perils. Hosp Health Netw. 2010;84(1):12. PMID:
20166483.
D-38
Pagliari C. Implementing the National Programme for IT: what can we learn from the Scottish experience? Inform
Prim Care. 2005;13(2):105-11. PMID: 15992495.
Pagliari C, Donnan P, Morrison J, et al. Adoption and perception of electronic clinical communications in Scotland.
Inform Prim Care. 2005;13(2):97-104. PMID: 15992494.
Pagliari C, Singleton P, Detmer DE. Time for a reality check of NPfIT’s problems. BMJ. 2009;338. PMID:
19223355.
Pan E, Cusack CM, Hook JM, et al. Cost of interconnecting health information exchanges to form a national
network. AMIA Annu Symp Proc. 2007:583-7. PMID: 18693903.
Pare G, Trudel MC. Knowledge barriers to PACS adoption and implementation in hospitals. Int J Med Inform.
2007;76(1):22-33. PMID: 16478675.
Exclusion: Not HIE
Park SC, Finnell JT. Indianapolis emergency medical service and the Indiana Network for Patient Care: evaluating
the patient match algorithm. AMIA Annu Symp Proc. 2012;2012:1221-8. PMID: 23304399.
Parrish F, Do N, Bouhaddou O, et al. Implementation of RxNorm as a terminology mediation standard for

exchanging pharmacy medication between federal agencies. AMIA Annu Symp Proc. 2006:1057. PMID: 17238676.
Parv L, Saluse J, Aaviksoo A, et al. Economic impact of a nationwide interoperable e-Health system using the
PENG evaluation tool. Stud Health Technol Inform. 2012;180:876-80. PMID: 22874318.
Patel V, Abramson EL, Edwards A, et al. Physicians' potential use and preferences related to health information
exchange. Int J Med Inf. 2011;80(3):171-80. PMID: 21156351.
Exclusion: Not HIE
Patel VN, Dhopeshwarkar RV, Edwards A, et al. Low-income, ethnically diverse consumers' perspective on health
information exchange and personal health records. Inform Health Soc Care. 2011;36(4):233-52. PMID: 21851182.
Exclusion: Not HIE
Patel VN, Dhopeshwarkar RV, Edwards A, et al. Consumer support for health information exchange and personal
health records: a regional health information organization survey. J Med Syst. 2012;36(3):1043-52. PMID:
20703633.
Paulus RA, Davis K, Steele GD. Continuous innovation in health care: implications of the Geisinger experience.
Health Aff (Millwood). 2008;27(5):1235-45. PMID: 18780906.
Exclusion: Not HIE
Payton FC, Brennan PF, Silvers JB. Cost justification of a community health information network: the
ComputerLink for AD caregivers. Proc Annu Symp Comput Appl Med Care. 1995:566-70. PMID: 8563348.
Exclusion: Not HIE
Pemble KR. Regional health information networks: the Wisconsin Health Information Network, a case study. Proc
Annu Symp Comput Appl Med Care. 1994:401-5. PMID: 7949958.
D-39
Pevnick JM, Claver M, Dobalian A, et al. Provider stakeholders' perceived benefit from a nascent health information
exchange: a qualitative analysis. J Med Syst. 2012;36(2):601-13. PMID: 20703673.
Pfoh E, Abramson E, Edwards A, et al. The Comparative Value of 3 Electronic Sources of Medication Data. Am J
Manag Care. 10/20/14 ed2014 Available at:
http://www.ajmc.com/publications/ajpb/2014/ajpb_septemberoctober2014/The-Comparative-Value-of-3-Electronic-
Sources-of-Medication-Data. Accessed April 22, 2015.
Exclusion: Not HIE
Phillips BO, Welch EE. Challenges for developing RHIOs in rural America: a study in Appalachian Ohio. J Healthc
Inf Manag. 2007;21(3):37-43. PMID: 19195292.
Pinborough-Zimmerman J, Bilder D, Satterfield R, et al. The impact of surveillance method and record source on
autism prevalence: collaboration with Utah Maternal and Child Health programs. Matern Child Health J.
2010;14(3):392-400. PMID: 19475366.
Exclusion: Not HIE
Pirnejad H, Niazkhani Z, van der Sijs H, et al. Impact of a computerized physician order entry system on nurse-
physician collaboration in the medication process. Int J Med Inf. 2008;77(11):735-44. PMID: 18514020.
Exclusion: Not HIE
Porteous T, Bond C, Robertson R, et al. Electronic transfer of prescription-related information: comparing views of
patients, general practitioners, and pharmacists... including commentary by Lockyer M. Br J Gen Pract.
2003;53(488):204-9. PMID: 14694696.
Exclusion: Not HIE
Posner KL, Van Norman GA, Chan V. Adverse cardiac outcomes after noncardiac surgery in patients with prior
percutaneous transluminal coronary angioplasty. Anesth Analg. 1999;89(3):553-60. PMID: 10475280.
Exclusion: Not HIE
Powell J, Fitton R, Fitton C. Sharing electronic health records: the patient view. Inform Prim Care. 2006;14(1):55-7.
PMID: 16848967.
Exclusion: Not HIE
Prestigiacomo J. HIE Sustainability Secrets. Healthc Inform. 2011;28(11):24-8. PMID: 22121569.

Prestigiacomo J. Overcoming interoperability challenges through HIE. Huntington Hospital creates its own
community information exchange to coordinate care, aid practice viability. Healthc Inform. 2012;29(5):36-7. PMID:
22655443.
Prestigiacomo J. Tennessee HIE to begin data exchange. Middle Tennessee eHealth Connect readies its core hospital
contributors and seeks payer participation. Healthc Inform. 2012;29(5):35. PMID: 22655442.
Proeschold-Bell RJ, Belden CM, Parnell H, et al. A randomized controlled trial of health information exchange
between human immunodeficiency virus institutions. J Public Health Manag Pract. 2010;16(6):521-8. PMID:
20885182.
Exclusion: Not HIE
Protti D. US regional health information organizations and the nationwide health information network: any lessons
for Canadians? Healthc Q. 2008;11(2):96-101. PMID: 18700270.
D-40
Protti D. Reflections on international EHR journeys. Healthcare Information Management and Communications.
2009;23(4):6-9.
Protti D, Bowden T, Johansen I. Adoption of information technology in primary care physician offices in New
Zealand and Denmark, part 1: healthcare system comparisons. Inform Prim Care. 2008;16(3):183-7. PMID:
19094404.
Exclusion: Not HIE
Zealand and Denmark, part 2: historical comparisons. Inform Prim Care. 2008;16(3):189-93. PMID: 19094405.
Exclusion: Not HIE
Zealand and Denmark, Part 4: Benefits comparisons. Inform Prim Care. 2008;16(4):291-6. PMID: 19192331.
Exclusion: Not HIE
Protti D, Edworthy S, Johansen I. Adoption of information technology in primary care physician offices in Alberta
and Denmark, Part 1: Historical, technical and cultural forces. Healthc Q. 2007;10(3):95-102, 4. PMID: 17626551.
Exclusion: Not HIE
Protti D, Johansen I, Perez-Torres F. Comparing the application of Health Information Technology in primary care
in Denmark and Andalucia, Spain. Int J Med Inform. 2009;78(4):270-83. PMID: 18819836.
Exclusion: Not HIE
Protti D, Nilsson G. Swedish GPs use Electronic Patient Records. Can Med Assoc J. 2005;10
Exclusion: Not HIE
Protti D, Smit C. GP’s have been using EMRs in the Netherlands for over twenty years. Canada Health Infoway.
2005
Exclusion: Not HIE
Protti D, Treweek S. Scottish physicians are also active users of electronic medical records. Canada Health Infoway.
2005
Exclusion: Not HIE
Protti D, Wright G, Treweek S, et al. Primary care computing in England and Scotland: a comparison with
Denmark. Inform Prim Care. 2006;14(2):93-9. PMID: 17059698.
Quinn R. Transaction portal cuts costs. New York payers and providers discover that IT collaboration and the
sharing of information affords savings that no organization could achieve on its own. Health Manag Technol.
2003;24(12):40-2. PMID: 14679731.
Quintana Y, Howard S, Norland M, et al. Pond4Kids - an multi-site online Pediatric Oncology Research Database
for collaborative protocol research. AMIA Annu Symp Proc. 2005:1090. PMID: 16779377.
Quintana Y, Patel AN, Arreola M, et al. POND4Kids: A global web-based database for pediatric hematology and
oncology outcome evaluation and collaboration. Stud Health Technol Inform. 2013;183:251-6. PMID: 23388293.
Exclusion: Not HIE
D-41
Quintana Y, Patel AN, Naidu PE, et al. POND4Kids: A web-based pediatric cancer database for hospital-based
cancer registration and clinical collaboration. Stud Health Technol Informatics. 2011;164:227-31. PMID: 21335715.
Exclusion: Not HIE
Rajda J, Vreeman DJ, Wei HG. Semantic interoperability of Health Risk Assessments. AMIA Annu Symp Proc.
2011;2011:1134-43. PMID: 22195174.
Ralston JD, Silverberg MJ, Grothaus L, et al. Use of web-based shared medical records among patients with HIV.
Am J Manag Care. 2013;19(4):e114-24. PMID: 23725449.
Exclusion: Not HIE
Raths D. No practice run. Getting large physician practices and IPAs to buy into a RHIO is paramount to its
survival. Healthc Inform. 2007;24(9):41-2. PMID: 17927063.
Rawson NSB. Access to linked administrative healthcare utilization data for pharmacoepidemiology and
pharmacoeconomics research in Canada: anti-viral drugs as an example. Pharmacoepidemiol Drug Saf.
2009;18(11):1072-9. PMID: 19650154.
Reed-Fourquet LL, Durand D, Johnson L, et al. CHIME-Net, the Connecticut Health Information Network: a pilot
study. Proc Annu Symp Comput Appl Med Care. 1995:561-5. PMID: 8563347.
Reid RJ, Wagner EH. Strengthening primary care with better transfer of information. CMAJ. 2008;179(10):987-8.
PMID: 18981432.
Reiss SM, American Pharmacists A. Integrating pharmacogenomics into pharmacy practice via medication therapy
management. J Am Pharm Assoc (2003). 2011;51(6):e64-74. PMID: 22001957.
Research K. Health information exchanges: rapid growth in an evolving market. Orem, Utah:2011. Available at:
www.klasresearch.com. Accessed April 22, 2015.
Rigby M, Roberts R, Williams J, et al. Integrated record keeping as an essential aspect of a primary care led health
service. BMJ. 1998;317(7158):579-82. PMID: 9721116.
Riley L, Smith G. Developing and implementing IS: A case study analysis in social services. J Inform Technol.
1997;12(4):305-21.
Exclusion: Not HIE
Rode D. Connecting the dots. Outlining the organizations involved with EHRs and HIE. J AHIMA. 2007;78(4):18-
20. PMID: 17455840.
Roop ES. Ten elements of a successful HIE. For the Record (Great Valley Publishing Company, Inc).
2011;23(3):3p.
Roos NP, Black CD, Frohlich N, et al. A population-based health information system. Med Care. 1995;33(12
Suppl):DS13-20. PMID: 7500666.
D-42
Rosen R, Florin D, Hutt R. An anatomy of GP referral decisions. A qualitative study on GPs’ views on their role in
supporting patient choice. King’s Fund, United Kingdom. Available at:
http://www.kingsfund.org.uk/publications/anatomy-gp-referral-decisions. Accessed April 22, 2015.
Exclusion: Not HIE
Rosenfeld S, Bernasek C, Mendelson D. Medicare's next voyage: encouraging physicians to adopt health
information technology. Health Aff (Millwood). 2005;24(5):1138-46. PMID: 16162556.
Rosenman M, Szucs K, Finnell SME, et al. Development and Testing of Health Information Exchange Methods for
Alerting Infection Preventionists About Multi-Drug Resistant Organisms: Making Unstructured Microbiology
Culture Data Usable. Am J Infect Control. 2014;42:S62-3.
Roshanov PS, Fernandes N, Wilczynski JM, et al. Features of effective computerised clinical decision support
systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. PMID: 23412440.
Exclusion: Not HIE
Rubin RD. The Community Health Information Movement: Where It’s Been, Where It’s Going. In: O’Carroll P,
Ripp L, Yasnoff W, Ward ME, Martin E, eds. Public Health Informatics and Information Systems: Springer New
York; 2003:595-616.
Rudin RS. Using information technology to exchange health information among healthcare providers: Measuring
usage and understanding value. Diss Abstr Int. 2012;73(4-B):2158.
Rudin RS, Salzberg CA, Szolovits P, et al. Care transitions as opportunities for clinicians to use data exchange
services: how often do they occur? J Am Med Inform Assoc. 2011;18(6):853-8. PMID: 21531703.
Exclusion: Not HIE
Rudin RS, Schneider EC, Volk LA, et al. Simulation Suggests that medical group mergers won't undermine the
potential utility of health information exchanges. Health Aff. 2012;31(3):548-59. PMID: 22392665.
Exclusion: Not HIE
Ruotsalainen P. A cross-platform model for secure Electronic Health Record communication. Int J Med Inform.
2004;73(3):291-5. PMID: 15066561.
Russler D. Disease registries on the nationwide health information network. J Diabetes Sci Technol. 2011;5(3):535-
42. PMID: 21722569.
Exclusion: Not HIE
Ryan DH. A Scottish record linkage study of risk factors in medical history and dementia outcome in hospital
patients. Dementia. 1994;5(6):339-47. PMID: 7866488.
Exclusion: Not HIE
Sackett KM, Erdley WS, Jones J. The Western New York regional electronic health record initiative: Healthcare
informatics use from the registered nurse perspective. Stud Health Technol Inform. 2006;122:248-52. PMID:
17102258.
Exclusion: Not HIE
Saef SH, Bourne CL, Bush JS, et al. The impact of a health information exchange on resource use and medicare-
allowable charges at eleven emergency departments operated by four major hospital systems in a midsized
southeastern city: An observational study using clinician estimates. Ann Emerg Med. 2013;62(4):S97.
D-43
Sairamesh J, Griss ML, Weber PA, et al. Innovation in healthcare intelligence: cross-sector convergence beyond
electronic medical records. Am J Prev Med. 2011;40(5 Suppl 2):S234-7. PMID: 21521599.
Salzberg CA, Jang Y, Rozenblum R, et al. Policy initiatives for health information technology: a qualitative study of
U.S. expectations and Canada's experience. Int J Med Inf. 2012;81(10):713-22. PMID: 22902272.
Exclusion: Not HIE
Sands DZ. Help for physicians contemplating use of e-mail with patients. J Am Med Inform Assoc. 2004;11(4):268-
9. PMID: 15252925.
Schiefelbein EL, Olson JA, Moxham JD. Patterns of Health Care Utilization among Vulnerable Populations in
Central Texas Using Data from a Regional Health Information Exchange. J Health Care Poor Underserved.
2014;25(1):37-51. PMID: 24509011.
Schnall R, Bakken S. Testing the Technology Acceptance Model: HIV case managers' intention to use a continuity
of care record with context-specific links. Inform Health Soc Care. 2011;36(3):161-72. PMID: 21848452.
Exclusion: Not HIE
Schnall R, Cimino JJ, Bakken S. Development of a prototype continuity of care record with context-specific links to
meet the information needs of case managers for persons living with HIV. Int J Med Inf. 2012;81(8):549-55. PMID:
22632821.
Schnall R, Odlum M, Gordon P, et al. Barriers to implementation of a Continuity of Care Record (CCR) in
HIV/AIDS care. Stud Health Technol Inform. 2009;146:248-52. PMID: 19592843.
Exclusion: Not HIE
Schnall R, Smith AB, Sikka M, et al. Employing the FITT framework to explore HIV case managers' perceptions of
two electronic clinical data (ECD) summary systems. Int J Med Inf. 2012;81(10):e56-62. PMID: 22841702.
Exclusion: Not HIE
Schneider ME. Interoperability issues limit health-record, data sharing. Caring for the Ages. 2013;14(1):9.
Schnipper JL, Hamann C, Ndumele CD, et al. Effect of an electronic medication reconciliation application and
process redesign on potential adverse drug events: a cluster-randomized trial. Arch Intern Med. 2009;169(8):771-80.
PMID: 19398689.
Exclusion: Not HIE
Schwartze J, Haarbrandt B, Rochon M, et al. Design and Implementation of an Informed Consent Process for a
Standardized Health Information Exchange Solution on the Example of the Lower Saxony Bank of Health. Stud
Exclusion: Not HIE
Sek ACH, Cheung NT, Choy KM, et al. A territory-wide electronic health record--from concept to practicality: the
Hong Kong experience. Stud Health Technol Inform. 2007;129(Pt 1):293-6. PMID: 17911725.
Selenke J. EHR bliss. A small family practice reaps the benefits of a Web-based EHR. Health Manag Technol.
2007;28(12):38-9. PMID: 18210973.
Exclusion: Not HIE
D-44
Sensmeier J. Laying the foundation for a secure, interoperable, nationwide health information network. Comput
Inform Nurs. 2009;27(3):195-6. PMID: 19411951.
Exclusion: Not HIE
Shade SB, Chakravarty D, Koester KA, et al. Health information exchange interventions can enhance quality and
continuity of HIV care. Int J Med Inf. 2012;81(10):e1-9. PMID: 22854158.
Shaikh AR, Prabhu Das I, Vinson CA, et al. Cyberinfrastructure for consumer health. Am J Prev Med. 2011;40(5
Suppl 2):S91-6. PMID: 21521603.
Shank N. Behavioral health providers' beliefs about health information exchange: a statewide survey. J Am Med
Exclusion: Not HIE
Shapiro JS. Evaluating public health uses of health information exchange. J Biomed Inform. 2007;40(6 Suppl):S46-
9. PMID: 17919985.
Shapiro JS, Bartley J, Kuperman G. Initial Experience with Opt-in Consent at the New York Clinical Information
Exchange (NYCLIX). AMIA Annu Symp Proc. 2009:1029.
Shapiro JS, Genes N, Kuperman G, et al. Health information exchange, biosurveillance efforts, and emergency
department crowding during the spring 2009 H1N1 outbreak in New York City. Ann Emerg Med. 2010;55(3):274-9.
PMID: 20079955.
Shapiro JS, Kannry J, Kushniruk AW, et al. Emergency physicians' perceptions of health information exchange. J
Am Med Inform Assoc. 2007;14(6):700-5. PMID: 17712079.
Exclusion: Not HIE
Shapiro JS, Kannry J, Lipton M, et al. Approaches to patient health information exchange and their impact on
emergency medicine. Ann Emerg Med. 2006;48(4):426-32. PMID: 16997679.
Exclusion: Not HIE
Shapiro JS, Onyile A, Genes N, et al. Validating health information exchange data for quality measurement. Ann
Emerg Med. 2013;62(4):S94.
Shapiro JS, Onyile A, Patel VR, et al. Enabling 72-hour emergency department returns measurement with regional
data from a health information exchange. Ann Emerg Med. 2011;58(4):S295.
Shapiro JS, Vaidya SR, Kuperman G. Preparing for the evaluation of health information exchange. AMIA Annu
Symp Proc. 2008:1128. PMID: 18999179.
Shaw KJ, Gutierrez M, Fridman M, et al. Health care costs associated with changing clinics and "walk-in"
deliveries: evidence supporting a regionalized health information network. Am J Obstet Gynecol.
2008;198(6):707.e1-8; discussion .e8. PMID: 18448082.
D-45
Shekelle PG, Morton SC, Keeler EB. Costs and benefits of health information technology. Evid Rep Technol Assess
(Full Rep). 2006(132):1-71. PMID: 17627328.
Shields AE, Shin P, Leu MG, et al. Adoption of health information technology in community health centers: results
of a national survey. Health Aff (Millwood). 2007;26(5):1373-83. PMID: 17848448.
Exclusion: Not HIE
Shih FJ, Fan YW, Chiu CM, et al. Needs for providing overseas organ transplant medical function and information
with eHealth telecare systems-instrument development for health professionals in Taiwan. Transplant Proc.
2014;46(4):1014-8. PMID: 24815115.
Shih FJ, Shih FJ, Pan YJ, et al. Dilemma of applying telehealth for overseas organ transplantation: comparison on
perspectives of health professionals and e-health information and communication technologists in Taiwan.
Transplant Proc. 2014;46(4):1019-21. PMID: 24815116.
Exclusion: Not HIE
Shy BD, Shapiro JS, Shearer PL, et al. A conceptual framework for improved analyses of 72-hour return cases. Am
J Emerg Med. (0)PMID: 25303847.
Exclusion: Not HIE
Silva PL. Planning for productivity. A Michigan health plan leverages its PM and EMR systems to improve the
bottom line and speed access to business intelligence. Health Manag Technol. 2008;29(4):32-3, 7. PMID: 18468217.
Exclusion: Not HIE
Simon JS, Rundall TG, Shortell SM. Adoption of order entry with decision support for chronic care by physician
organizations. J Am Med Inform Assoc. 2007;14(4):432-9. PMID: 17460136.
Exclusion: Not HIE
Simon SR, Evans JS, Benjamin A, et al. Patients' attitudes toward electronic health information exchange:
qualitative study. J Med Internet Res. 2009;11(3):e30. PMID: 19674960.
Exclusion: Not HIE
Simon SR, Kaushal R, Cleary PD, et al. Physicians and electronic health records: a statewide survey. Arch Intern
Med. 2007;167(5):507-12. PMID: 17353500.
Exclusion: Not HIE
Simonaitis L, Belsito A, Warvel J, et al. Extensible Stylesheet Language Formatting Objects (XSL-FO): a tool to
transform patient data into attractive clinical reports. AMIA Annu Symp Proc. 2006:719-23. PMID: 17238435.
Exclusion: Not HIE
Simons WW, Halamka JD, Kohane IS, et al. Integration of the personally controlled electronic medical record into
regional inter-regional data exchanges: a national demonstration. AMIA Annu Symp Proc. 2006:1099. PMID:
17238718.
Slade K, Lambert MJ, Harmon SC, et al. Improving psychotherapy outcome: The use of immediate electronic
feedback and revised clinical support tools. Clin Psychol Psychother. 2008;15(5):287-303. PMID: 19115449.
Exclusion: Not HIE
Smith E, Kaufman JH. Lowering the barrier to a decentralized NHIN using the open healthcare framework. Stud
D-46
Smith ME, Newcombe HB. Automated follow-up facilities in Canada for monitoring delayed health effects. Am J
Public Health. 1980;70(12):1261-8. PMID: 7435743.
Exclusion: Not HIE
Social Security A. Obtaining evidence beyond the current "special arrangement sources." Interim final rule with
request for comments. Fed Regist. 2014;79(113):33681-3. PMID: 24922983.
Sokolova M, El Emam K, Arbuckle L, et al. P2P watch: personal health information detection in peer-to-peer file-
sharing networks. J Med Internet Res. 2012;14(4):e95. PMID: 22776692.
Solberg D. 'Pipe dream' HIE proves challenging. Health Manag Technol. 2009;30(7):22. PMID: 19739562.
Solomon MR. Regional health information organizations: a vehicle for transforming health care delivery? J Med
Syst. 2007;31(1):35-47. PMID: 17283921.
Solutions DCfH. Health Information Exchange (HIE) Business Models: The Path to Sustainable Financial Success.
Available at: http://www.providersedge.com/ehdocs/ehr_articles/Health_Info_Exchange_Business_Models.pdf.
Soti P, Pandey S. Business process optimization for RHIOs. J Healthc Inf Manag. 2007;21(1):40-7. PMID:
17299924.
Spahni S, Guardia A, Boggini T, et al. Design and Implementation of a Shared Treatment Plan in a Federated Health
Information Exchange... MEDINFO 2013. Stud Health Technol Inform. 2013;192:1090. PMID: 23920864.
Spath MB, Grimson J. Applying the archetype approach to the database of a biobank information management
system. Int J Med Inf. 2011;80(3):205-26. PMID: 21131230.
Exclusion: Not HIE
Spil TA, Schuring RW, Stegwee RA, et al. Towards a better understanding of the e-health user: comparing USE IT
and Requirements study for an Electronic Patient Record. Available at: http://doc.utwente.nl/55471/. Accessed
April 22, 2015.
Exclusion: Not HIE
Sprivulis P, Walker J, Johnston D, et al. The economic benefits of health information exchange interoperability for
Australia. AMIA Annu Symp Proc. 2005:1119. PMID: 16779406.
Sprivulis P, Walker J, Johnston D, et al. The economic benefits of health information exchange interoperability for
Australia. Aust Health Rev. 2007;31(4):531-9. PMID: 17973611.
Sridhar S, Brennan PF, Wright SJ, et al. Optimizing financial effects of HIE: a multi-party linear programming
approach. J Am Med Inform Assoc. 2012;19(6):1082-8. PMID: 22733978.
Stair TO. Reduction of redundant laboratory orders by access to computerized patient records. The Journal of
emergency medicine. 1998;16(6):895-7. PMID: 9848709.
Exclusion: Not HIE
D-47
Stansfield K, Yetman L, Renwick C. eDoc evaluation - At eighteen months into the challenge. Stud Health Technol
Informatics. 2009;143:414-8. PMID: 19380970.
Exclusion: Not HIE
Starr P. Smart technology, stunted policy: developing health information networks. Health Aff. 1997;16(3):91-105.
PMID: 9141326.
Stiell A, Forster AJ, Stiell IG, et al. Prevalence of information gaps in the emergency department and the effect on
patient outcomes. CMAJ. 2003;169(10):1023-8. PMID: 14609971.
Exclusion: Not HIE
Stolyar A, Lober WB, Drozd DR, et al. Feasibility of data exchange with a Patient-centered Health Record. AMIA
Annu Symp Proc. 2005:1123. PMID: 16779410.
Exclusion: Not HIE
Stoves J, Connolly J, Cheung CK, et al. Electronic consultation as an alternative to hospital referral for patients with
chronic kidney disease: a novel application for networked electronic health records to improve the accessibility and
efficiency of healthcare. Qual Saf Health Care. 2010;19(5):e54-e. PMID: 20554576.
Strasberg HR, Hubbs PR, Rindfleisch TC, et al. Analysis of information needs of users of the Stanford Health
Information Network for Education. Proc AMIA Symp. 1999;Annual Symposium.:965-9. PMID: 10566504.
Exclusion: Not HIE
Stroetmann V, Thiel R, Stroetmann KA, et al. Understanding the role of device level interoperability in promoting
health - lessons learned from the SmartPersonalHealth Project. Yearb Med Inform. 2011;6(1):87-91. PMID:
21938330.
Sucurovic S. Implementing security in a distributed web-based EHCR. Int J Med Inform. 2007;76(5-6):491-6.
PMID: 17084662.
Suhanic W, Crandall I, Pennefather P. An informatics model for guiding assembly of telemicrobiology workstations
for malaria collaborative diagnostics using commodity products and open-source software. Malar J. 2009;8:164.
PMID: 19615074.
Sullivan FM, McEwan N, Murphy G. Regional repositories, reintermediation and the new GMS contract:
cardiovascular disease in Tayside. Inform Prim Care. 2003;11(4):215-21. PMID: 14980061.
Exclusion: Not HIE
Szende A. Ontario's province-wide paediatric electronic health record. Stud Health Technol Inform. 2009;143:99-
103. PMID: 19380922.
Takada A, Guo sJ, Tanaka K, et al. Dolphin project--cooperative regional clinical system centered on clinical
information center. J Med Syst. 2005;29(4):391-400. PMID: 16178336.
Takeda H, Matsumura Y, Kuwata S, et al. An assessment of PKI and networked electronic patient record system:
lessons learned from real patient data exchange at the platform of OCHIS (Osaka Community Healthcare
Information System). Int J Med Inform. 2004;73(3):311-6. PMID: 15066564.
Exclusion: Not HIE
D-48
Takeda H, Matsumura Y, Nakagawa K, et al. Healthcare public key infrastructure (HPKI) and non-profit
organization (NPO): essentials for healthcare data exchange. Stud Health Technol Inform. 2004;107(Pt 2):1273-6.
PMID: 15361019.
Tamblyn R, Poissant L, Huang A, et al. Estimating the information gap between emergency department records of
community medication compared to on-line access to the community-based pharmacy records. J Am Med Inform
Assoc. 2014;21(3):391-8. PMID: 23956015.
Exclusion: Not HIE
Tambouris E, Williams MH, Makropoulos C. Co-operative health information networks in Europe: experiences
from Greece and Scotland. J Med Internet Res. 2000;2(2):E11. PMID: 11720930.
Exclusion: Not HIE
Tan SL, Lewis RA. Picture archiving and communication systems: a multicentre survey of users experience and
satisfaction. Eur J Radiol. 2010;75(3):406-10. PMID: 19523778.
Exclusion: Not HIE
Tchwenko SN, Parnell H, Messer LC. Health outcomes following a health information exchange intervention for
HIV patients. Am J Epidemiol. 2012;175:S13.
Teixeira PA, Gordon P, Camhi E, et al. HIV patients' willingness to share personal health information electronically.
Patient Educ Couns. 2011;84(2):e9-12. PMID: 20724095.
Exclusion: Not HIE
Tello-Leal E, Chiotti O, Villarreal PD. Process-oriented integration and coordination of healthcare services across
organizational boundaries. J Med Syst. 2012;36(6):3713-24. PMID: 22434534.
Exclusion: Not HIE
Tennison J, Rajeev D, Woolsey S, et al. The Utah Beacon Experience: Integrating Quality Improvement, Health
Information Technology, and Practice Facilitation to Improve Diabetes Outcomes in Small Healthcare Facilities.
EGEMS (Wash DC). 2014;2(3)
Exclusion: Not HIE
Terry K. Electronic exchange of health information dials in new patient consent questions. Med Econ.
2014;91(13):46-50. PMID: 25174225.
Tham E, Ross SE, Mellis BK, et al. Interest in health information exchange in ambulatory care: a statewide survey.
Appl Clin Inform. 2010;1(1):1-10. PMID: 23616824.
Thielst CB. Regional health information networks and the emerging organizational structures. J Healthc Manag.
2007;52(3):146-50. PMID: 17552351.
Thorn SA. Emergency physicians' perspectives on the usability of health information exchange. Diss Abstr Int.
2012;72(7-A):2200.
Thorn SA, Carter MA. The Potential of Health Information Exchange to Assist Emergency Nurses. J Emerg Nurs.
2013;39(5):e91-6. PMID: 23369772.
D-49
Thornewill J, Dowling AF, Cox BA, et al. Information infrastructure for consumer health: a health information
exchange stakeholder study. Am J Prev Med. 2011;40(5 Suppl 2):S123-33. PMID: 21521585.
Tierney WM, Overhage JM, McDonald CJ. Toward electronic medical records that improve care. Ann Intern Med.
1995;122(9):725-6. PMID: 7702235.
Ting S, Kwok S, Tsang A, et al. Experiences Sharing of Implementing Template-Based Electronic Medical Record
System (TEMRS) in a Hong Kong Medical Organization. J Med Syst. 2011;35(6):1605-15. PMID: 20703758.
Exclusion: Not HIE
Tjora A, Tran T, Faxvaag A. Privacy vs usability: a qualitative exploration of patients' experiences with secure
Internet communication with their general practitioner. J Med Internet Res. 2005;7(2):e15. PMID: 15998606.
Exclusion: Not HIE
Tomines A, Readhead H, Readhead A, et al. Applications of Electronic Health Information in Public Health: Uses,
Opportunities and Barriers. EGEMS (Wash DC). 2013;1(2)
Törnvall E, Wilhelmsson S. Nursing documentation for communicating and evaluating care. J Clin Nurs.
2008;17(16):2116-24. PMID: 18710374.
Exclusion: Not HIE
Toussaint JS, Queram C, Musser JW. Connecting statewide health information technology strategy to payment
reform. Am J Manag Care. 2011;17(3):e80-8. PMID: 21504263.
Exclusion: Not HIE
Trigg LJ. Social construction of the patient through problems of safety, uninsurance, and unequal treatment. ANS
Adv Nurs Sci. 2009;32(3):E17-27. PMID: 19707084.
Exclusion: Not HIE
Triska OH, Church J, Wilson D, et al. Physicians' perceptions of integration in three Western Canada Health
Regions. Healthc Manage Forum. 2005;18(3):18-24. PMID: 16323465.
Exclusion: Not HIE
Tsiknakis M, Brochhausen M, Nabrzyski J, et al. A semantic grid infrastructure enabling integrated access and
analysis of multilevel biomedical data in support of postgenomic clinical trials on cancer. IEEE Trans Inf Technol
Biomed. 2008;12(2):205-17. PMID: 18348950.
Exclusion: Not HIE
Tsiknakis M, Kouroubali A. Organizational factors affecting successful adoption of innovative eHealth services: a
case study employing the FITT framework. Int J Med Inf. 2009;78(1):39-52. PMID: 18723389.
Exclusion: Not HIE
Tufano JT. Information and communication technologies in patient-centered healthcare redesign: Qualitative studies
of provider experience [Ph.D.]. Ann Arbor, University of Washington; 2009.
Tuttle MS, Nelson SJ. The role of the UMLS in 'storing' and 'sharing' across systems. Int J Biomed Comput.
1994;34(1-4):207-37. PMID: 8125633.
Ullman K. Indiana data network provides one stop for inter-hospital connectivity. How an Indiana-based regional
health data exchange helps CIOs save time and money. Healthc Inform. 2010;27(8):32. PMID: 20853808.
D-50
Unertl KM, Weinger M, Johnson K. Variation in use of informatics tools among providers in a diabetes clinic.
Exclusion: Not HIE
Vaidya SR, Shapiro JS, Papa AV, et al. Perceptions of health information exchange in home healthcare. Comput
Inform Nurs. 2012;30(9):503-9. PMID: 22584878.
Exclusion: Not HIE
van der Linden H, Kalra D, Hasman A, et al. Inter-organizational future proof EHR systems. A review of the
security and privacy related issues. Int J Med Inf. 2009;78(3):141-60. PMID: 18760661.
van der Linden MW, Plat AW, Erkens JA, et al. Large impact of antidiabetic drug treatment and hospitalizations on
economic burden of diabetes mellitus in The Netherlands during 2000 to 2004. Value Health. 2009;12(6):909-14.
PMID: 19508664.
Exclusion: Not HIE
Van der Velde ET, Atsma DE, Foeken H, et al. Remote monitoring of patients with implanted devices: data
exchange and integration. Eur J Prev Cardiolog. 2013;20(2 Suppl):8-12. PMID: 23702984.
Exclusion: Not HIE
Van Eaton EG, Devlin AB, Devine EB, et al. Achieving and Sustaining Automated Health Data Linkages for
Learning Systems: Barriers and Solutions. EGEMS (Wash DC). 2014;2(2)
van Walraven C, Taljaard M, Bell CM, et al. Information exchange among physicians caring for the same patient in
the community. CMAJ. 2008;179(10):1013-8. PMID: 18981442.
Exclusion: Not HIE
van Wingerde FJ, Sun Y, Harary O, et al. Linking multiple heterogeneous data sources to practice guidelines. Proc
AMIA Symp. 1998;Annual Symposium.:391-5. PMID: 9929248.
Velamuri S. QRDA--technology overview and lessons learned. J Healthc Inf Manag. 2010;24(3):41-8. PMID:
20677471.
Vest JR, Gamm LD. Health information exchange: persistent challenges and new strategies. J Am Med Inform
Assoc. 2010;17(3):288-94. PMID: 20442146.
Vest JR, Jasperson J. What should we measure? Conceptualizing usage in health information exchange. J Am Med
Exclusion: Not HIE
Vest JR, Menachemi N, Ford EW. Governance's role in local health departments' information system and
technology usage. J Public Health Manag Pract. 2012;18(2):160-8. PMID: 22286285.
Exclusion: Not HIE
Virga PH, Jin B, Thomas J, et al. Electronic health information technology as a tool for improving quality of care
and health outcomes for HIV/AIDS patients. Int J Med Inf. 2012;81(10):e39-45. PMID: 22890224.
Exclusion: Not HIE
D-51
Viswanathan KP, Bass R, Wijetunge G, et al. Rural mass casualty preparedness and response: the Institute of
Medicine's Forum on Medical and Public Health Preparedness for Catastrophic Events. Disaster Med Public Health
Prep. 2012;6(3):297-302. PMID: 23077273.
Voigt C, Torzewski S. Direct results. An HIE tests simple information exchange using the direct project. J AHIMA.
2011;82(5):38-41. PMID: 21667863.
Wagner PJ, Dias J, Howard S, et al. Personal health records and hypertension control: a randomized trial. J Am Med
Exclusion: Not HIE
Walker J, Pan E, Johnston D, et al. The value of health care information exchange and interoperability. Health Aff
(Millwood). 2005;Suppl Web Exclusives:W5-10-W5-8. PMID: 15659453.
Walker JM, Carayon P. From tasks to processes: the case for changing health information technology to improve
health care. Health Aff (Millwood). 2009;28(2):467-77. PMID: 19276006.
Exclusion: Not HIE
Walker R, Blacker V, Pandita L, et al. Learning from the implementation of inter-organisational web-based care
planning and coordination. Aust J Prim Health. 2013;19(4):297-302. PMID: 23866768.
Exclusion: Not HIE
Walsh MN, Albert NM, Curtis AB, et al. Lack of association between electronic health record systems and
improvement in use of evidence-based heart failure therapies in outpatient cardiology practices. Clin Cardiol.
2012;35(3):187-96. PMID: 22328100.
Exclusion: Not HIE
Warnekar PP, Bouhaddou O, Parrish F, et al. Use of RxNorm to exchange codified drug allergy information
between Department of Veterans Affairs (VA) and Department of Defense (DoD). AMIA Annu Symp Proc.
2007:781-5. PMID: 18693943.
Exclusion: Not HIE
Weber GM. Federated queries of clinical data repositories: the sum of the parts does not equal the whole. J Am Med
Inform Assoc. 2013;20(e1):e155-61. PMID: 23349080.
Exclusion: Not HIE
Weber SC, Lowe H, Das A, et al. A simple heuristic for blindfolded record linkage. J Am Med Inform Assoc.
2012;19(e1):e157-61. PMID: 22298567.
Exclusion: Not HIE
Weber SC, Seto T, Olson C, et al. Oncoshare: lessons learned from building an integrated multi-institutional
database for comparative effectiveness research. AMIA Annu Symp Proc. 2012;2012:970-8. PMID: 23304372.
Webster PC. Infoway tacks towards "networked" patients. CMAJ. 2011;183(4):E223-4. PMID: 21324865.
Weitzman ER, Kelemen S, Kaci L, et al. Willingness to share personal health record data for care improvement and
public health: a survey of experienced personal health record users. BMC Med Inform Decis Mak. 2012;12:39.
PMID: 22616619.
Exclusion: Not HIE
D-52
Wells S, Hill-Smith I. Bridging the communication gap in diabetes care. Practical Diabetes International.
1996;13(6):174-6.
Exclusion: Not HIE
Wen K-Y, Kreps G, Zhu F, et al. Consumers' perceptions about and use of the internet for personal health records
and health information exchange: analysis of the 2007 Health Information National Trends Survey. J Med Internet
Res. 2010;12(4):e73. PMID: 21169163.
Exclusion: Not HIE
Were MC, Meeks-Johnson J, Overhage JM. Enhanced laboratory reports: using health information exchange data to
provide contextual information to laboratory results for practices without electronic records. AMIA Annu Symp
Proc. 2008:1174. PMID: 18999174.
Exclusion: Not HIE
Westbrook JI, Braithwaite J, Georgiou A, et al. Multimethod evaluation of information and communication
technologies in health in the context of wicked problems and sociotechnical theory. J Am Med Inform Assoc.
2007;14(6):746-55. PMID: 17712083.
Exclusion: Not HIE
Westbrook JI, Braithwaite J, Iedema R, et al. Evaluating the impact of information communication technologies on
complex organizational systems: a multi-disciplinary, multi-method framework. Stud Health Technol Inform.
2004;107(Pt 2):1323-7. PMID: 15361029.
Exclusion: Not HIE
Wilcox AB, Shen S, Dorr DA, et al. Improving access to longitudinal patient health information within an
emergency department. Appl Clin Inform. 2012;3(3):290-300. PMID: 23646076.
Exclusion: Not HIE
Wiljer D, Urowitz S, Apatu E, et al. Patient accessible electronic health records: exploring recommendations for
successful implementation strategies. J Med Internet Res. 2008;10(4):e34. PMID: 18974036.
Exclusion: Not HIE
Willis E. Engagement in online health communities: Expressed attitudes and self-efficacy of arthritis self-
management behaviors. Diss Abstr Int. 2011;74(4-A(E)).
Wilt DH, Muthig BA. Crossing barriers: EMR implementation across a nationwide continuum of care. J Healthc Inf
Manag. 2008;22(2):23-6. PMID: 19266991.
Winthereik B, Vikkelsø S. ICT and Integrated Care: Some Dilemmas of Standardising Inter-Organisational
Communication. Comput Support Coop Work. 2005;14(1):43-67.
Exclusion: Not HIE
Wong HJ, Caesar M, Bandali S, et al. Electronic inpatient whiteboards: improving multidisciplinary communication
and coordination of care. Int J Med Inform. 2009;78(4):239-47. PMID: 18786851.
Exclusion: Not HIE
Woods SE, Coggan JM. Developing a medical informatics education program to support a statewide health
information network. Bull Med Libr Assoc. 1994;82(2):147-52. PMID: 8004015.
Exclusion: Not HIE
Woodside JM. EDI and ERP: a real-time framework for healthcare data exchange. J Med Syst. 2007;31(3):178-84.
PMID: 17622020.
D-53
Wright A, Soran C, Jenter CA, et al. Physician attitudes toward health information exchange: results of a statewide
survey. J Am Med Inform Assoc. 2010;17(1):66-70. PMID: 20064804.
Wu M, Rhyner P. Design of an integrated system for Milwaukee children with developmental disabilities. AMIA
Annu Symp Proc. 2005:1156. PMID: 16779442.
Wynn A, Wise M, Wright MJ, et al. Accuracy of administrative and trauma registry databases. J Trauma.
2001;51(3):464-8. PMID: 11535892.
Exclusion: Not HIE
Yang W-H, Hu J-S, Chou Y-Y. Analysis of network type exchange in the health care system: a stakeholder
approach. J Med Syst. 2012;36(3):1569-81. PMID: 21046205.
Exclusion: Not HIE
Yaraghi N, Du AY, Sharman R, et al. Professional and geographical network effects on healthcare information
exchange growth: does proximity really matter? J Am Med Inform Assoc. 2014;21(4):671-8. PMID: 24287171.
Yasnoff WA, Humphreys BL, Overhage JM, et al. A consensus action agenda for achieving the national health
information infrastructure. J Am Med Inform Assoc. 2004;11(4):332-8. PMID: 15187075.
Exclusion: Not HIE
Yee KC, Miils E, Airey C. Perfect match? Generation Y as change agents for information communication
technology implementation in healthcare. Stud Health Technol Inform. 2008;136:496-501. PMID: 18487780.
Exclusion: Not HIE
Zafar A, Dixon BE. Pulling back the covers: technical lessons of a real-world health information exchange. Stud
Health Technol Inform. 2007;129(Pt 1):488-92. PMID: 17911765.
Zhao J, Zhang Z, Guo H, et al. E-health in China: challenges, initial directions, and experience. Telemed J E Health.
2010;16(3):344-9. PMID: 20406121.
Zimmerman CR, Chaffee BW, Lazarou J, et al. Maintaining the enterprisewide continuity and interoperability of
patient allergy data. Am J Health-Syst Pharm. 2009;66(7):671-9. PMID: 19299376.
Exclusion: Not HIE
Zulman DM, Nazi KM, Turvey CL, et al. Patient interest in sharing personal health record information: a web-based
survey. Ann Intern Med. 2011;155(12):805-10. PMID: 22184687.
Exclusion: Not HIE
Zulman DM, Piette JD, Jenchura EC, et al. Facilitating out-of-home caregiving through health information
technology: survey of informal caregivers' current practices, interests, and perceived barriers. J Med Internet Res.
2013;15(7):e123. PMID: 23841987.
Exclusion: Not HIE
Zvarova J, Lhotska L, Seidl L, et al. Health data collecting and sharing: case studies of Czech e-health applications.
Stud Health Technol Inform. 2012;180:672-6. PMID: 22874276.
Exclusion: Not HIE
D-54
Appendix E. Study Design Terminology
The studies included in this review are described in terms of their design, data source and
analysis approach. Study designs are included in summary tables, while all three characteristics
may be discussed in the text.
1) Study design:
Randomized controlled trial
Cohort (prospective or retrospective)
Case Control (be sure it actually is)
Cross-sectional
Time Series
Multiple site case studies
Case series
2) Data Source:
Database (administrative data, clinical data)
Survey, questionnaire, focus group
Audit logs
Observations
Documents
3) Analysis:
Quantitative
Descriptive statistics
Regression/Other multivariable analysis
Qualitative
Content or Thematic Analysis
Mixed methods
Includes quantitative and qualitative
Narrative description
Ethnographic
E-1
Appendix F. Evidence Table
Table F1. Evidence Table
Study Purpose/Research Geographic Data Source(s)/ Evaluation Time Period of Data
Author, Year Study Design Question Location Setting Data Collection
Abramson, et al., Cross-sectional Measure EHR and HIE adoption New York State Hospital Survey of hospitals May-December 2009
76 in New York State hospitals
2012
Abramson, et al., Cross-sectional Measure EHR and HIE adoption New York State Nursing homes Survey of nursing homes November 2011-
77 in New York State nursing homes March 2012
2014
Abramson, et al., Cross-sectional To determine rates of New York Hospitals Survey responses November 2012 -
96 participation in HIE February 2013
2014
F-1
Date HIE
Author, Year Name of HIE (Intervention) Description of HIE (this will become Types) Implemented Population
Abramson, et al., Various HIEs around New York Type of data exchanged NR NR All 205 hospitals in New York
76 State State
2012
Abramson, et al., Nursing homes around New York Exchange of data (NR) with pharmacies, lab, hospitals, NR All 632 nursing homes in New
77 State physician offices, and RHIO York State
2014
Abramson, et al., NA NA NA Surveyed Hospital IT directors

96 or chief information officer
2014
F-2
Author, Year N Sample description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison
Abramson, et al., Various HIEs All hospitals in New York State NA None
76
2012
Abramson, et al., Various HIEs All nursing homes in New York NA None
77 State
2014
Abramson, et al., Contacted: 210 Hospitals All hospitals in New York state NA Results compared
96 Respondents: 129 (61.4%)
2014
Nonrespondents: 81 (38.6%)
F-3
Author, Year Outcomes Measured Independent Variables Confounding Variables Analysis Methods
Abramson, et al., Participation in HIE Participate in HIE (exchange of data) NA Quantitative
76
2012 Descriptive statistics
Abramson, et al., Participation in HIE Participate in HIE (exchange of data) NA Quantitative

77
Abramson, et al., Use of HIE, if information is sent and/or NA NA Descriptive statistics

96 received by the institution, type of
2014
institution information is shared with,
barriers to implementation
F-4
Risk of
Author, Year Results Bias
Abramson, et al., 23% of respondent hospitals participate and exchange data vs. 37% participate but do not exchange data vs. 40% do not participate Low
76
2012
Abramson, et al., 54.4% participate in HIE, Low

77 OR of participating in HIE: 2.26 more likely when have EHR
2014
Exchange with providers when EHR
59.7% within system vs. 31.3% outside system
HIE highest usage
Pharmacies: 41.8%
Labs: 38.5%
Hospitals: 38 5%
Abramson, et al., -79.1% (n=102) of respondents reported actively exchanging any electronic patient-level clinical data with an entity outside their institution Moderate
96 in 2012 vs. 60% in 2009
2014
Type of institution respondents exchanged data with:
Hospitals outside your system: 70.6% (n=72)
Ambulatory providers outside your system: 68.6% (n=70)
Long term care facilities: 45.1% (n=46)
Home health agencies: 38.2% (n=39)
The most commonly exchanged data were radiology reports, followed by laboratory results. Only 45 respondents (44.1%) exchanged
medication lists and clinical history with hospitals outside their system.
Respondents reporting participation in a regional arrangement for HIE:

Any data exchange: 89.9% (n=116)
Actively sending and receiving data: 50.9% (n=59)
Sending data only: 25.9% (n=30)
Receiving data only: 16.4% (n=19)
Barriers to HIE participation reported by responding hospitals:

Privacy concerns: 54.7% (n=70)
Security concerns: 52.3% (n=67)
Lack of IT staff to support HIE: 38.2% (n=49)
Lack of architecture to support HIE: 35.9% (n-46)
No differences in barriers among hospitals engaging in HIE and those not engaging in HIE were found. When hospitals engaged in
sending and receiving data were compared with hospitals only sending or only receiving data hospitals only engaged in one activity were
more likely to identify lack of architecture p=0.05 and cost of participating p=0.03 as barriers to HIE
F-5
Adjerid and Cross-sectional -Analyze data from compilation of U.S. Any Survey 2009-2010
140 privacy laws and Adler-Milstein
Padman, 2011 Data from compilation of privacy
2009 analysis of RHIOs laws and Adler-Milstein 2009
-Examine association of state analysis of RHIOs
"consent prior to disclosure" laws
with number of operational HIEs
Adler-Milstein and Cross-sectional -Analyze data from annual AHA U.S. Any Survey Late 2012
108 survey of hospital IT
Jha, 2014 Hospital survey database,
-Measure HIE usage among U.S. augmented with market and other
hospitals characteristic data
Adler-Milstein, Cross-sectional Measure number of RHIOs, U.S. Any Survey of RHIOs June 2008-December
Bates, and Jha, participation in them by 2009
79 ambulatory practices and
2011
hospitals, and number financially
viable
Adler-Milstein, Cross-sectional Measurement of types of data U.S. Any Survey of HIE organizations August-November
Bates, and Jha, exchanged, organizations 2012
25 involved, and sources of financial
2013
support
Adler-Milstein, Cross-sectional Measurement of participation in a U.S. Hospital Hospital survey database AHA survey from
DesRoches, and regional HIO and exchange of spring-summer 2009
107 data with hospitals or ambulatory
Jha, 2011
providers of a different system
F-6
Date HIE
Adjerid and All in U.S. All types NA 313 HIE initiatives from 2004-
140 2009
Padman, 2011
Adler-Milstein and All in U.S. All types NA 2,849 U.S. hospitals that
108 responded to AHA IT survey
Jha, 2014
Adler-Milstein, All in U.S. All types provided by a RHIO NA 197 organizations meeting
Bates, and Jha, definition of RHIO
79
2011
Adler-Milstein, All in U.S. All types NA 221 organizations facilitating

Bates, and Jha, HIE
25
2013
Adler-Milstein, All in U.S. All types NA 3,101 acute-care, nonfederal
DesRoches, and hospitals that were U.S. based
107 members of AHA
Jha, 2011
F-7
Adjerid and All 313 HIE initiatives HIE status; state health None None
140 disclosure law status
Padman, 2011
Adler-Milstein and All of population All hospitals responding to None None

108 survey
Jha, 2014
Adler-Milstein, 165 RHIOs All RHIOs Not meeting definition of None

Bates, and Jha, RHIO
79
2011
Adler-Milstein, NA All organizations facilitating HIE Organizations only None

Bates, and Jha, participating in HIE
25
2013
Adler-Milstein, Various HIEs All acute-care, nonfederal Hospitals that were federal None
DesRoches, and hospitals that were U.S. based or nonacute or were not
107 members of AHA members of AHA
Jha, 2011
F-8
Adjerid and Total, operational, and failed HIE -Health disclosure law HIE size not accounted for Quantitative
140 -Population
Padman, 2011 Econometric models
-Per capita GDP
Adler-Milstein and Participating in HIE -Ownership NA Quantitative Multivariate
108 -Market position
Jha, 2014 Analysis
-Size OR of likelihood of participation
-Teaching status
-Cardiac ICU
-System affiliation
-Medicaid admissions
-EHR system
Adler-Milstein, Operational RHIOs, supporting stage 1 Operational RHIOs, supporting stage 1 NA Quantitative
Bates, and Jha, meaningful use, ambulatory practices and meaningful use, ambulatory practices
79 hospitals participating in RHIOs, and and hospitals participating in RHIOs,
2011
number of financially viable and number of financially viable
Adler-Milstein, Operational exchange or data, types of Operational exchange or data, types of NA Quantitative
Bates, and Jha, data exchanged, barriers to exchange data exchanged, barriers to exchange Descriptive statistics; compared
25
2013 with previous reports
Adler-Milstein, Participation in HIE and market -Hospital profit status NA Quantitative

DesRoches, and characteristics -Market share Analysis of database
107
Jha, 2011 -Teaching status Logistic regression models
-Size
-Cardiac ICU
-System affiliation
-Medicaid admissions
-EHR system
F-9
Risk of
Adjerid and States with stronger privacy laws have more operational HIEs, fewer failed HIEs, and take less time to reach operational status. NA
140
Padman, 2011
Adler-Milstein and -30% of hospitals engage in HIE, varying widely by state Low
108 -For-profit hospitals less likely to engage than nonprofit hospitals. Hospitals with larger market share or in less competitive markets more
Jha, 2014
likely to exchange
Adler-Milstein, -75 operational RHIOs, covering 14% of U.S. hospitals and 3% of ambulatory practices Low
Bates, and Jha, -13 supporting meaningful use, covering 3% of hospitals, 0.9% of ambulatory practices; 67% not meeting criteria for financial viability
79
2011
Adler-Milstein, Predominant organization nonprofit; Low

Bates, and Jha, Sources of support
25
2013 Grants and contracts: 52%; participant fees: 28%; operating costs not covered by revenue: 57%
Barriers to development
Sustainability: 74%; lack of funding: 57%; privacy: 60%; mandates: 55%; technical barriers: 61%; competition: 56%; linking; 54%
Adler-Milstein, 10.7% participation in regional HIO; statistically significantly higher for private/nonprofit status, greater market bed share, teaching status, Low
DesRoches, and large size, cardiac ICU presence, and had EHR system
107
Jha, 2011
F-10
Adler-Milstein, et Cross-sectional Measurement of activities and U.S. Any Survey of RHIOs July 2006-March
81 financing of functioning RHIOs 2007
al., 2008
Adler-Milstein, Cross-sectional Measurement of types of data U.S. Any Survey of operational RHIOs 2008, following up of
Bates, and Jha, exchanged, organizations survey from 2007
78 involved, and sources of financial
2009
support
Adler-Milstein, Cross-sectional Measure factors associated with U.S. Any Survey of RHIOs Mid-2008
Landefeld, and becoming operational and
80 achieving financial viability
Jha, 2010
Afilalo, et al., RCT Impact of sending family Montreal, Canada ED and family Survey Not stated but likely
66 physicians electronic vs. mailed physician same as Lang, 2006
2007 Survey of family physician
reports of ED visits for their practices satisfaction
patients
F-11
Date HIE
Adler-Milstein, et All in U.S. All types provided by a RHIO NA 138 organizations meeting
81 definition of RHIO
al., 2008
Adler-Milstein, All in U.S. All types NA 207 organizations defined as

Bates, and Jha, RHIOs
78
2009
Adler-Milstein, All in U.S. All types provided by a RHIO NA 131 organizations meeting
Landefeld, and definition of RHIO
80
Jha, 2010
Afilalo, et al., Adult university teaching hospital in Report of ED visit sent to family physicians NR Patients visiting ED during
66 Montreal 0800-2200
2007
F-12
Adler-Milstein, et 32 RHIOs actively exchanging data 20 RHIOs actively exchanging Not actively exchanging None
81 clinical data for 5000+ patients data
al., 2008
Adler-Milstein, All 44 operational RHIOs exchanging data for ≥5,000 All RHIOs exchanging data for RHIOs not exchanging data None
Bates, and Jha, patients ≥5,000 patients or doing so for <5,000
78 patients
2009
Adler-Milstein, 81 RHIOs currently or planning to exchange data for 81 RHIOs currently or planning Not meeting definition of None
Landefeld, and 5000+ patients to exchange data for 5000+ RHIO
80 patients
Jha, 2010
Afilalo, et al., 2,022 (out of 3,168) patients visiting ED Patients visiting ED Patients in altered mental ED visit summary provided
66 state (129), state of electronically vs. on paper sent
2007
agitation (21), or with by mail
language barrier (29)
F-13
Adler-Milstein, et Proportion of RHIOs sending and receiving -Entity sending data NA Quantitative
81 data to different entities and proportion -Entity receiving data
al., 2008 Descriptive statistics
exchanging specific types of data -Type of data exchanged
Adler-Milstein, RHIO exchanging data for ≥5,000 patients -Types of data NA Quantitative
Bates, and Jha, -Entities exchanging data Descriptive statistics
78
2009 -Sources of financial support
Adler-Milstein, Factors associated with becoming -Participation NA Quantitative

Landefeld, and operational and achieving partial or full -Types of data exchanged, focused on Multivariate logistic regression for
80 financial viability a specific population, history of
Jha, 2010 predictors
collaborating, and sources of revenue
Afilalo, et al., Physician attitudes on aspects of continuity Survey Physicians already are sent carbon Quantitative
66 of care for patients copies of first page of ED note; self-
2007
report of followup data
F-14
Risk of
Adler-Milstein, et Entities providing data Low
81
al., 2008 Hospitals: 83%; ambulatory settings: 67%; labs: 60%; imaging results: 56%
Entities receiving data
Ambulatory settings: 95%; hospitals: 83%; public health departments: 50%; payers: 44%
Type of data exchanged
Test results: 90%; inpatient data test results: 90%; inpatient data: 70%; medication history: 70%; outpatient data: 60%
Adler-Milstein, Source of funding Low

Bates, and Jha, Time or in-kind resources: 64%; recurring fee: 55%; grant: 48%
78
2009 Types of data exchanged
Test results: 84%; inpatient data: 70%; medication history: 66%; outpatient data: 64%
28% of operational RHIOs expected to eventually cover operating costs
Barriers
Lack of funding, concerns about privacy/security, legal/regulatory changes, costs higher than expected, technical/infrastructure challenges
Adler-Milstein, Likelihood of being operational associated with exchanging narrow set of data and involving broad group of stakeholders, likelihood of Low
Landefeld, and financial viability associated with involvement of hospitals and ambulatory physicians and early funding from participants. Financial viability
80 diminished with early grant funding.
Jha, 2010
Afilalo, et al., ED visits followed up by electronic reports led to family physicians having OR of higher rate of information receipt, more useful information, Moderate
66 better knowledge of ED visits, better patient management, and more actions initiated by physicians. There was not perception of higher
2007
rate of followup in family practice offices.
F-15
166 Multiple Case To describe current state HIE National scan, in Multiple Multiple Sources 2005-2006
AHRQ, 2006
Studies environment and analyze state depth case Literature reviews, web-based
HIE activities and initiatives. studies of 8 research, reports, interviews,
States: Arizona,
Florida, Hawaii,
New York, North
Carolina, Rhode
Island,
Tennessee, Utah
Altman, et al., Cross-sectional To assess clinicians’ impressions New York Family practice Survey July 2011-October
57 of an hourly notification of ED clinics 2011
2012 Interviews
visit, hospital admission or
hospital discharge with respect to
the notifications effect on the
continuity and coordination of
patient care
Anand, et al., Cross-sectional Is real-time alerting useful and Indiana Primary care Databases, questionnaire June-November 2012
92 does it lead physicians to take physician offices
2012 Survey of value for real-time
action? alerting for patient ED visit
anywhere in state
Audet, Squires Cross-sectional Measurement of physician U.S. Physician offices Surveys March-July, 2012 (as
109 exchange of data outside of well as comparison
and Doty, 2014
practice or to receive hospital from data with 2009
discharge reports survey, specific dates
not provided)
F-16
Date HIE
166 Varies Varies 2003 to 2005 All HIE projects in US in 2055-
AHRQ, 2006
2006
Altman, et al., New York Clinical Information Hourly electronic notifications sent to family practice clinicians November 2010 Family practice clinicians in
57 Exchange (NYCLIX) when any of 3 patient events occur at a participating hospital: single health system receiving
2012
(1) a new ED visit, (2) a hospital admission, or (3) a hospital HIE notifications
discharge. 86% MDs
50% male
Anand, et al., Indiana HIE (IHIE) Patient data concerning ED visit 1994 Known physicians (538) of
92 patients (1,275) seen in an ED
2012
for asthma
Audet, Squires All in U.S. Physician exchange of data outside of practice or to receive NA 1,012 primary care physicians
109 hospital discharge reports in 2012
and Doty, 2014
F-17
166 101 HIE projects in 35 states for which information was HIE projects that included State HIE projects within a single Comparison of HIE project
AHRQ, 2006
available. and/or Medicaid involvement, hospital or health system or characteristics across states
8 States for in depth case studies targeted patients statewide or in that focused on
large portions of the state, administrative exchange or
involve a RHIO or RHIO like reducing fraud
organization
Altman, et al., 14 of 20 total Clinicians receiving notifications None Changes in practice as

57 perceived by interviewee
2012
Anand, et al., 79 physicians (10%) receiving 126 (15%) notifications Physicians who had ≥1 patient NA Information helpful, resulted
92 seen in ED and faxed notification followup action
2012
letter back to HIE
Audet, Squires, Various HIEs Primary care physicians in U.S. NA None

109
and Doty, 2014
F-18
166 Number of HIE projects NA NA Qualitative
AHRQ, 2006
Similarities and differences among projects
Altman, et al., Usage logs of number of notifications sent NA NA Thematic analysis

57 to each clinician over a period of several
2012 Themes of clinician perceptions
months, questionnaires identified and compared with
recorded usage logs
Anand, et al., Rates of information helpful, resulted in Survey None Quantitative

92 followup action
Audet, Squires, Proportion of physicians exchanging data Proportion of physicians exchanging NA Quantitative
109 outside of practice or receiving hospital data outside of practice or receiving
and Doty, 2014 Descriptive statistics and logistic
discharge reports hospital discharge reports regression
F-19
Risk of
166 States have multiple HIE projects NA
AHRQ, 2006
Project have similar goals but vary widely across other characteristics, particularly infrastructure which makes sharing lessons learned
challenging
Most projects are in early stages and have overly optimistic timelines
Funding varies widely
Sustainability is a long term goal but has not yet been realized. Most have not identified long term sources of funding
While state are critical stakeholders many do not plan to play primary leadership roles indefinitely.
Altman, et al., Notifications from an HIE system can enhance clinicians’ awareness of their patients’ interactions in the medical system. Clinicians Moderate
2012
57 perceived improvements in communication and followup scheduling as a result of notifications. Increase in clinician workload and change
in responsibility may be unintended effects of notifications Workflow issues should be carefully considered. Timely notifications may further
improve clinician-to-clinician communication
Anand, et al., -35% found information helpful vs. 20% not helpful NA
92 -24% made followup call to patient vs. 4% sent attached letter
2012
Audet, Squires, 32% use of HIE, with higher proportion for formal IT support, part of integrated system, receiving financial incentives, larger practice Low
109
and Doty, 2014
F-20
Bailey, et al., Retrospective To determine Memphis, ED Log file August 2007-July
39 cohort whether HIE by ED personnel in Tennessee 2009
2013 Diagnostic neuroimaging,
the evaluation of patients evidence-based guideline
with headache reduces use of adherence
neuroimaging, increases
adherence with guideline
Bailey, et al., Retrospective To determine whether HIE Memphis, ED Log file August 2007-July
40 cohort reduces repeated diagnostic Tennessee 2009
2013 Administrative data for imaging
imaging and costs in ED back log in patient record for HIE
pain evaluation access
Ben-Assuli, Retrospective Probability of single-day Israel ED Log file 2004-2007

Shabtai, and cohort admission and 7-day readmission
72 when HIE viewed
Leshno, 2015
F-21
Date HIE
Bailey, et al., MidSouth e-Health Alliance MSeHA HIE connects 15 major adult hospitals and 2 regional 2007 Patients presenting to
39 (MSeHA). clinic systems in 4 counties of the Memphis Metropolitan participating EDs with principle
2013
Statistical Area. Patient demographic, diagnosis, all hospital diagnosis of headache
radiologic and laboratory reports, most procedure reports,
and discharge summaries are exchanged. ED providers have
read-only access to data.
Bailey, et al., MidSouth e-Health Alliance Secure, password-protected, read-only access to clinical 2007 All patients with an ED visit for
40 (MSeHA), 15 major hospitals and information from participating hospitals and clinics through a back pain in the Alliance
2013
2 regional clinic systems in the 4 Web portal separate from each facility’s electronic health hospitals
most populous counties of the record system.
Memphis Metropolitan Statistical MSeHA HIE connects 15 major adult hospitals and 2 regional
Area. Decentralized, query-based clinic systems in 4 counties of the Memphis Metropolitan
exchange. Statistical Area. Patient demographic, diagnosis, all hospital
Consent was ‘opt-out. radiologic and laboratory reports, most procedure reports,
and discharge summaries are exchanged. ED providers have
read-only access to data.
Ben-Assuli, Clalit HMO, Israel Query 2004 All ED referrals

Shabtai, and
72
Leshno, 2015
F-22
Bailey, et al., 2,101 2nd or subsequent visits for 1,252 patients ≥18years, a second or Primary diagnosis (ICD-9 None
39 subsequent ED visit to a MSeHA codes) of variants of
2013
participating general hospital’s migraine (346.2),
ED between August 1, 2007 and hemiplegic migraine
July 31, 2009 with a primary (346.3), chronic migraine
discharge diagnosis of primary (346.7), other forms of
headache disorder (ICD-9-CM migraine (346.8), and
codes 346.0, 346.1, 346.9 and tension headache (307.81,
784.0); and no discharge 339.1)
diagnosis of stroke (ICD-9-CM 1st visit for headache
430–438), brain cancer (ICD-9-
CM 191.x, 225.0 and V10.85),
traumatic injury, motor vehicle
accident, poisoning, or fall.
Bailey, et al., Patients: 478 ≥18 years, >1 visit to system ED Discharge diagnosis of Repeat visits in which HIE was
40 Visits: 800 for back pain, index (previous trauma or cancer. accessed vs. repeat visits in
2013
visit) with imaging which HIE was not used
Ben-Assuli, 340,804 admitted and 474,310 non-admitted patients Referred to ED and had a None Access HIE information
Shabtai, and creatinine test
72
Leshno, 2015
F-23
Bailey, et al., Use of diagnostic neuroimaging (CT, CT -Any HIE use nonuse of HIE Quantitative
39 angiography, MRI or MRI angiography), -HIE use by physician or nurse
2013 Modeling using the generalized
evidence-based guideline adherence and practitioner estimating equation method to
economic -HIE use by administrative/nursing staff adjust for repeated measures
(since some subjects had >1
visit) and for clustering of
subjects within hospital system
Bailey, et al., -Use of repeated lumbar or thoracic -HIE accessed by any ED staff during -Patient age, sex and race Quantitative
40 imaging repeat ED visit (Yes/No) -Comorbidity 2
2013 Chi
-% cases HIE used -Type of staff accessing HIE (MD or -Hospital Multivariate: generalized
-Cost Nurse Practitioner vs. admin or -Number of previous ED visits estimating equation
nursing)
Ben-Assuli, Same-day admission and 7-day Access HIE information None Quantitative
Shabtai, and readmission Same-day admission and 7-day
72
Leshno, 2015 readmission via logistic
regression
F-24
Risk of
Bailey, et al., OR (95% CI ) of any HIE use Low
39
2013 Neuroimaging: 0.38 (0.29 to 0.50)
Adherence to guideline: 1.33 (1.02 to 1.73)
-Increased odds of neuroimaging by subjects of older age, black race,
and higher comorbidity
-Prior visits lower the odds of imaging 7%, but the effect was reduced to 2% with use of HIE
- No significant change in costs
Secondary analyses
-Administrative/nursing staff neuroimaging: OR 0.25 (95% CI, 0.18 to 0.34)
-Physician/Nurse Practitioner HIE use and interaction terms for previous visits were not significantly associated
-No secondary analyses were significant for guideline adherence
Bailey, et al., Repeated imaging for any HIE: OR 0.36 (95% CI, 0.18 to 0.71), p<0.05 Low
40 Visits with repeated imaging: 22.4% (179/800)
2013
HIE used: 12.5%
-Physician or Nurse practitioner use of HIE lowered OR for repeat imaging OR 0.47 (95% CI, 0.23 to 0.96)
- No cost savings associated with HIE use because of increased CT imaging when health care providers used HIE
Ben-Assuli, When external information viewed, probability of single-day admission decreased 9.5% and of 7-day readmission decreased 6.5% Low
Shabtai, and
72
Leshno, 2015
F-25
Ben-Assuli, Retrospective To determine whether HIE use Main Israeli HMO 7 acute care Log file 2004-2007
Shabtai, and Cohort was associated with reduced network hospitals EDs
41 readmissions and "avoidable" belonging to
Leshno, 2013
admissions largest Israeli
HMO
Bouhaddou, et al., Multiple site case Across 3 large integrated delivery San Diego, Integrated delivery Database and survey NR
82 studies with systems, how many patients can California system Patient identifier and
2011
focus on and will participate; how much demographic data
identification of used
patients eligible,
matching, and
consent; usage
F-26
Date HIE
Ben-Assuli, Largest Israeli HMO network 3.8 Clinical and administrative data from all HMO hospitals, 2004 Adult patients presenting to
Shabtai, and million patients, operates 7 community clinics and thousands of labs, imaging centers etc. Israeli ED with 1 of 5 main
41 hospitals Demographics, prescriptions, allergies, lab, imaging, past diagnosis; gastroenteritis,
Leshno, 2013
medical history, procedures. abdominal pain, chest pain,
pneumonia organism, urinary
tract infection
Bouhaddou, et al., Veterans Lifetime Electronic Query-based, transfer of records between integrated delivery NR Patients of 3 large IDSs who
82 Record (VLER) systems opted in to HIE
2011
F-27
Ben-Assuli, 115,719 ED Visits NR NR HIE vs. local EMR and no EMR
Shabtai, and HIE vs. local EMR use
41
Leshno, 2013
Bouhaddou, et al., 1,144 patients shared between VA and KP Patients identified as getting None None
82 care in VA and KP
2011
Nationwide Health Information Network allows users to
pull in data from other organizations. The VA and DoD
used the VLER systems for eHealth exchange with
private sector. Federated pull (query-based) model
Transfer of records between integrated delivery
systems; National query-based. Patient consent: Opt-
in.
F-28
Ben-Assuli, -OR for 7-day readmission for -MD Viewed EMR -Age Quantitative
Shabtai, and gastroenteritis, abdominal pain, chest pain, -MD Viewed local EMR -Gender -t test for continuous variables
41 pneumonia organism or urinary tract
Leshno, 2013 -MD viewed external information (HIE) -HMO 2
-Chi for dichotomous
infection -HMO to which patient belonged -ED
-OR for 1-day admission for gastroenteritis, -Multi-variate regression analysis
-Differential Diagnosis -Hospital -P<0.05, no adjustment for
abdominal pain, chest pain, pneumonia
organism, or urinary tract infection -ED sub department (Int. med or multiple hypothesis testing
surgical)
-Economic
-Specific Hospital
-Age
-Gender
-Authors list all these variables as
independent but some are more
confounding per se
Bouhaddou, et al., Patients who opted in and provided valid -Patients correlated across KP and VA NA Quantitative
82 authorization, with subsequent measure of -Actual records exchanged
2011 Survey, descriptive statistics
records exchanged between KP and VA 2-
3 per week
F-29
Risk of
Ben-Assuli, OR for all 5 differential diagnosis as composite Low
Shabtai, and Readmission within 7 days: 0.52 for HIE vs. local EMR and no EMR, p<0.001
41
Leshno, 2013 1-day admission: 0.76, p<0.001
Readmission within 7 days: 1.272, p=0.05 for local EMR vs. HIE
1-day admission: 1.13, p=0.005 for local EMR vs. HIE
-Decrease in readmissions within 7 days when HIE used 56.1%

-Decrease in single-day readmissions when HIE used 29.0%
-Viewing external medical history more highly correlated with lower single-day admissions and 7-day readmissions than local medical
history
Bouhaddou, et al., Of 363 patients who opted in and provided valid authorization, 264 could be correlated; exchange of records between KP and VA 2-3 per NA
82 week. Older patients were more likely to consent for HIE.
2011
F-30
Byrne, et al., Multiple site case Describe key findings, lessons, 12 sites across Unrestricted Audit logs, database, survey, December 2009-
116 studies implications from VLER pilot U.S. October 2012
2014 interviews, documents from
project meetings
Veterans authorization
preferences, system dashboard,
VA provider (11/12 site) and
veteran interviews. 73 provider
interviews, 50 veteran interviews
F-31
Date HIE
Byrne, et al., Veterans Lifetime Electronic Query-based HIE between VA, DOD, nonfederal care December 2009 Veterans
116 Record (VLER) organizations. The Nationwide Health Information Network.
2014
The VA and DoD used the VLER systems for eHealth
exchange with private sector. Federated pull model transfer of
records between integrated delivery systems; 12 total sites, 4
did 3 way exchange, 8 did 2 way between VA and private
sector. Federated pull model via eHealth Exchange
F-32
Author, Year N Sample Description (if applicable) Inclusion Criteria Exclusion Criteria Comparator or Comparison
Byrne, et al., 12 pilot sites 12 VLER pilot sites. Veterans None NA
116 N=73 provider and 50 veteran interview included were any who opted in.
2014
F-33
Byrne, et al., -Veterans accept NA NA Mixed Methods
116 -Veteran concerns about participation
2014 Quantitative, descriptive analysis
-Veterans perceived benefit on usage; qualitative, thematic
-Veteran awareness of VLER use during analysis
their care
-Veterans preference of signed
authorizations
-Metrics of exchanged data
F-34
Risk of
Byrne, et al., -64,237 veterans provided authorization and opted in NA
116 -Opted in then out: <0.01%
2014
-Veterans matched with exchange partner: 31,080 (48%), range: 12-88%
-Highest matching rates with exchange partners using social security number in their algorithm
-Inbound discloser's to VA from exchange partners 5,524
-Outbound disclosure to exchange partner 13,913
-Inbound disclosures to VA from exchanged partners per matched patients 18/100
-Unique VA patient with exchange partner data retrieved: 2,724
-Unique VA providers retrieving exchange partner data: 1,764
- Percent of matched veterans for whom there was ≥1 disclosure to VA from exchange partner: 9%
-75% of providers trusted VLER data, 90% trusted privacy and security
-Most frequently cited provider benefits, more data for medical decision making, improved quality of care, reduced repeat testing, timelier
and faster access to information
-23/73 interviewed providers reported using VLER, 79% of users reporting overall satisfaction
-43% reported challenges with system response time, 29% with identifying patients who might have data
-Identified minimizing provider steps in information retrieval, one site Indiana HIE had an automated query resulting in push into their
system to allow providers pushed access anytime a patient was admitted discharged or transferred
-Providers at outside organizations did not having additional sign ones
-Workflow improvements suggested by outside users was to have data pushed in their EMR
-Sustaining HIE requires ongoing resources and oversight, often unanticipated technical issues arose
-Requires national policies and central coordination
-None of the veterans interviewed were aware if their providers were using HIE, the user-interfaces at the sites face the provider not the
patient
-Providers increased usage after training on VLER system
-Providers noted barriers of missing data, additional sign-on and need for better integration with workflow
F-35
Campion, et al., Cross-sectional Determine the extent to which Binghamton, New Hospital/clinic HIE log data 2010 until 23 months
97 automated HIE queries supported York following
2013
patient encounters.
Campion, et al., Cross-sectional What is usage and satisfaction of Buffalo and Health systems, Survey July-December 2010
58 push and pull HIE Rochester, New health
2012 Online survey responses from
York departments, 112/584 invited physicians (19%
practice response rate)
associations,
RHIO
F-36
Date HIE
Campion, et al., Southern Tier HealthLink RHIO in Lawson Cloverleaf HIE, centralized data repository with MPI. 5 2005 ≥18 years, with positive
97 Binghamton, New York part of hospitals, one imaging center and 30 ambulatory care consent to participate in HIE
2013
SHIN-NY. Automated queries practices affiliated with single integrated delivery system.
occurred evening prior to
ambulatory patient appointments to
generate CCRs and for the
hospitals during ED visits, at
inpatient admission, inpatient unit
transfer and provided CCD doc to
providers. Providers could also log
in manually. Auto queries started
month 1 for clinics and month 17
for hospitals.
Campion, et al., HealtheLINK (Buffalo) and Direct exchange (push) of local lab and radiology results; query- 2007-2009 Physicians
58 Rochester RHIO based (pull) searching for lab and radiology results across
2012
greater Buffalo and Rochester area.
Robust RHIOs using HIE platform from Axolotl Corporation
(San Jose, California)
F-37
Campion, et al., 202,365 auto queries ≥18 years, who had automated Lack of known provider or NA
97 HIE query generated, which lack of known facility in
2013
occurred when a care transition auto-queries from HIE
occurred
Campion, et al., 112/584 invited physicians (19% response rate). Only Physicians who completed Respondents who did not Compared various attributes of
58 99 completed. 75% were primary care providers. Most survey and rated overall rate satisfaction with HIE HIE for push vs. pull
2012
practices had 2-19 providers. outcome of satisfaction with HIE
F-38
Campion, et al., Generation of automated HIE queries NA NA Quantitative
97
Campion, et al., Use of push vs. pull HIE. Satisfaction with Type of HIE: push or pull NR Quantitative
58 types of HIE.
F-39
Risk of
Campion, et al., -202,365 automated HIE queries: 54% to hospitals, 46% to clinics NA
97 -After exclusions, duplicates removed: 145,668 unique patient encounters
2013
-81,687 unique patients provided consent for query based HIE during study period, 41% had ≥1 supported encounter
-For the 33,219 patient with ≥1 clinic encounter: median IQR 3
-98% of patients had between 1 and 20 encounters, 71% had ≥2
-530 patients with ≥20 encounters
-52% occurred in hospital, 48% in clinics
Care Transitions
-28% of the 145,668 unique encounters occurred as care transitions
-53% were patients from a clinic to hospital, 36% in reverse, 11% clinic to clinic
Campion, et al., -80% used push HIE and 53% used pull HIE Moderate
58 -A greater proportion of MDs reported using push HIE always or most of the time (68%) vs. pull HIE (19%), p=0.001
2012
-MDs more satisfied with push HIE vs. pull HIE, p<.0.05
-112 physician respondents (19% response), 13 then excluded for 99 participants
->50% of physicians felt HIE improved 8 domains; access to timely, completeness, accurate information, admin efficiency, communication
with colleagues, and quality
-Only 30% felt it improved reducing test redundancy and security of PHI
-Physicians who used push and pull vs. only single type had higher rates of perceived effects of HIE in same 8 domains, (3of 8 domains
p<0.05)
F-40
Campion, et al., Cross-sectional Measure usage patterns of query 3 separate RHIOs Unclear, inpatient/ System log data A, B: January 2009-
98 survey based HIE with respect to encompassing 1 outpatient Demographics of patient, March 2011
2013
practice sites, users, patients, community each provider character (i.e. role, C: September 2010-
and data (~1 million patient location etc.) May 2011
population) in
New York state
(from HEAL-NY)
Caffrey and Park- Cross-sectional To determine use of EHR and U.S. Residential care Survey 2010
93 HIE by residential care communities
Lee, 2013 2010 National Survey of
communities. Residential Care Facilities
70 Case series Does HIE reduce unneeded test Charleston, ED Questionnaire August-December
Carr, et al., 2014
ordering and costs, admissions South Carolina User-initiated survey, with costs 2011
calculated for self-reported
testing not performed
F-41
Date HIE
Campion, et al., NY State HIE consists of 12 RHIOs Axolotl Virtual Health Record-commercial product. Web based 2007, 2007, 2010; All patients
98 (HEAL NY) secure stand alone portal. Federated architecture with MPI, A, B and C,
2013
RLS and user directory. respectively.
Caffrey and Park- NR NR NR Residential care communities

93
Lee, 2013
70 Carolina eHealth Alliance Access to EHRs and ED from all hospitals in region NR Physicians, Nurse
Carr, et al., 2014
Practitioners, Physician
Assistants, and students
F-42
Campion, et al., Combined 2.9 million total patients in 3 RHIO All patients None NA
98 communities
2013
Caffrey and Park- Sampled: 3,605 Residential care communities Communities licensed to NA
93 Interviewed: 2,302 that have been licensed, serve severely mentally ill
Lee, 2013
registered, listed, certified or or intellectually or
otherwise regulated by the states developmentally disabled
with >4 beds, >1 resident populations exclusively.
currently living in the community, Nursing homes were also
and provide room and board with excluded unless they had a
at least 2 meals a day, around unit or wing meeting
the clock onsite supervision, and inclusion criteria where
help with personal care such as residents could be
bathing and dressing or enumerated separately.
health=related services such as
medication management.
70 18,529 patient encounters, with 998 logons (5.39%) by All survey responses from HIE NA None
Carr, et al., 2014
60 clinicians. 138 (13.8%) surveys completed. 105 users
(10.5%) of patients had data in HIE.
F-43
Campion, et al., -% practice sites accessing data NA NA Quantitative
98 -Type of practice accessing HIE
-Number of roles and primary practice of
users accessing HIE
-Characteristics of patients whose data was
accessed
-Consenting of patients related to access
Caffrey and Park- % of residential care communities that NA NA Quantitative

93 used EHR with computerized support for
Lee, 2013 Regression
HIE
70 -Services, costs, and admissions avoided Tests, costs, and admissions avoided NA Quantitative
Carr, et al., 2014
-Perceived time saved Self-reported tests and
admissions avoided, calculation
of costs saved based on local
data.
F-44
Risk of
Campion, et al., A vs. B vs. C NA
98 -Of sites registered to use system: 18% vs. 30% vs. 82% accessed in first 9 months
2013
-After 27 months 60% vs. 59% vs. NR of sites had accessed
-In each community majority of practice sites from which access occurred were out patient
-In A and B majority of sessions were from outpatient sites, C was inpatient
-Registered users in community: 368 vs. 3461 vs. 118
-More than 1/2 users accessing system in A and B were nurses + staff, in C 2/3 were MDs + physician extenders
-Majority of all users practiced in ambulatory setting
-Patients whose data was accessed were older than those whose was not and then the entire population
-For community A&B majority had data accessed on same day as consent
-Majority of patients in A and B had their data accessed in community setting, C was inpatient
-% of patient whose data was accessed from ≥2 sites in first 9 months: 0.1% vs. 1.8% vs. 0.01%; after 27 months: 0.1% vs. 11.6% vs. NR
-System access occurred from 60% to 82% of practice sites registered to use system, depending on community
-Proportions of patients whose data were accessed varied between 5%-60%
-Most frequently accessed data were patient summaries, followed by lab and radiology data
Caffrey and Park- 17% of residential care communities reported using EHR Low
93
Lee, 2013 % of residential care communities using EHR with computerized system to support HIE by provider type:
Any provider: 40
Pharmacy: 23
Other health or long-term care provider: 20
Physician: 17
Corporate office: 17
Other: 17
70 -Reported avoiding: 30.5% lab/micro tests ($462), 47.6% radiology tests ($161,000), 19% consultations ($4,000), 11.4% admissions Moderate
Carr, et al., 2014
($118,000)
-86.7% reported improved quality of care
-81% reported time savings, averaging 120.8 minutes
F-45
Chang, et al., Cross-sectional Development and evaluation of Indiana Physician office, Survey 2 week period in 2007
51 enhanced reporting of lab data outpatient
2010 Survey of physicians who were
based on data available to HIE potential users of reporting
interface
Codagnone and Cross-sectional To measure and explain levels of 31 countries: Varies as this was Survey, interviews, focus October 25, 2012 to
Lupiañez- availability and use (adoption) of EU27 countries an international groups March 6, 2013
94 eHealth applications and services plus Croatia, survey
Villanueva, 2013
Iceland, Norway
and Turkey
Dixon, Miller, and Cross-sectional What are barriers to participation Indiana Small hospitals, Survey and interviews August 2009-March
141 in a mature state HIE? small physician
Overhage, 2013 Initial mixed methods interviews 2010
practices, and with most physician groups given
large physician online survey
practices
Dixon, Jones, and Cross-sectional Awareness and engagement of 6 states with HIE - Case reporting for Survey NR
83 infection preventionists in HIE for 3 funded by CDC public health
Grannis, 2013 Online survey of 63 infection
public health surveillance for explicit HIE- reporting of preventionists
based reporting notifiable
and three with conditions
mature HIEs
F-46
Date HIE
Chang, et al., Indiana Network for Patient Care Collection of all lab data with enhancements (prior results, Not stated, but in Primary care physicians who
51 other historical lab results, prescriptions, encounters), 1990s were users of HIE
2010
pharmacy data, and patient encounter data
Codagnone and Varies as this was an international Varies as this was an international survey Varies as this was an Random sample of general
Lupiañez- survey international survey practitioners who use a
94 computer
Villanueva, 2013
Dixon, Miller, and Indiana HIE (IHIE) Full medical record in HIE 1994 Small hospitals, small
141 physician practices, and large
Overhage, 2013
physician practices in Indiana
who were not participating in
HIE
Dixon, Jones, and 6 states with mature HIEs but 6 states with HIE — 3 funded by CDC for explicit HIE-based Not specific, would Infection preventionists
83 details not explicitly provided public health surveillance reporting for infections, versus three be variable by state
Grannis, 2013
with mature HIEs, but without active surveillance reporting.
63 preventionists.
F-47
Chang, et al., NA Convenience sample of primary NA None
51 care physicians
2010
Codagnone and 9,196 general practitioners General practitioners who use a General practitioners who Comparison of HIE use by
Lupiañez- computer don't use a computer country to prior survey in 2007
94
Villanueva, 2013
Dixon, Miller, and 12 small hospitals, 20 small physician practices, and Small hospitals, small physician Small hospitals, small Barriers of cost, lack of
141 11 large physician practices who were not participating practices, and large physician physician practices, and sufficient technical or human
Overhage, 2013
in HIE practices in Indiana who were large physician practices in resources, or lack of
not participating in HIE Indiana who were awareness regarding value
participating in HIE proposition
Dixon, Jones, and NA Infection preventionists in public NA Comparisons in states with

83 health departments in 6 states active public health
Grannis, 2013
surveillance vs. those without
F-48
Chang, et al., Evaluation of developed report Various factors related to usefulness NA Quantitative
51 and completeness
2010 Satisfaction survey
Codagnone and Use of 15 functions of HIE and 4 functions Country, Types of HIE use Addressed thoroughly in multiple Quantitative multivariate
Lupiañez- of telehealth. Comparison with previous analyses of use and adoption. analysis
94
Villanueva, 2013 survey in 2007. Factor analysis to create 1 overall
composite indicator, and 4
smaller composite indicators
(EHR, HIE, telehealth, PHR).
Comparison with 2007 results.
Dixon, Miller, and Barriers of cost, lack of sufficient technical Survey None Mixed methods
141 or human resources, or lack of awareness
Overhage, 2013 Qualitative content analysis of
regarding value proposition interviews and quantitative
tabulation of surveys
Dixon, Jones, and -EHR use -Organizations with EHR NA Quantitative

83 -EHR involvement in implementation -Involved in implementation of EHR
Grannis, 2013 Descriptive Statistics
-Involvement in HIE -Engaged in HIE
-Method for notifiable case reporting -Reporting methods for notifiable cases
F-49
Risk of
Chang, et al., -9 physicians sampled Moderate
51 -Average 5 point Likert scales reported showed perception was generally favorable. ELRs well organized (4.2±0.97) and easy to interpret
2010
(4.3±0.50). Additional data elements were valuable: relevant test (4.2±0.97), contextual drugs (4±0.89), visit histories (3.25±0.71) and
computer generated clinical reminders (3.25±0.71). Compared with traditional lab results ELRs generally saved time (3.78±0.67), reduce
the need to search for information (3.67±0.71) and improve quality of care (3.78±0.67). Physicians asked whether they would prefer to
use ELRs instead of traditional reports (3.78±0.67).
Codagnone and Substantial increases in HIE use between 2007 and 2013. Qualitative results on barriers to adoption and use. Low
Lupiañez- Countries with National Health Systems have high HIE use that countries with social insurance or transition systems.
94
Villanueva, 2013 Barriers to implementation included lack of interoperability, issues with system resilience, and security concerns. Systems that focused on
administrative rather than clinical applications were used less.
Dixon, Miller, and Barriers (small hospitals, small physician practices, large physician practices) Moderate
141
Overhage, 2013 Cost: 100%, 50%, 55%
Lack of sufficient technical or human resources: 42%, 45%, 36%
Lack of awareness regarding value proposition: 33%, 15%, 36%
Dixon, Jones, and -72% in organizations with EHR; 20% involved in implementation of EHR; 10% engaged in HIE; 49% unaware of organizational Moderate
83 involvement in HIE
Grannis, 2013
-<5% reporting via secure email, web-based entry, through EHR, or through HIE each
F-50
Dixon, McGowan, Retrospective To determine completeness and Indiana Public health Log file November 14, 2010-
and Grannis, cohort quality of data for public health -7.5 lab results reported in HIE December 15, 2010
42 electronic laboratory reporting in
2011 -Statutory public health reporting
an HIE records
Dobalian, et al., Cross-sectional Describe lessons learned from Long Beach, 3 hospitals, 2 Interviews 2008
142 one Nationwide Health California ambulatory
2012 Test data
Information Network practice groups
implementation
Dullabh and Multiple case 1) Assess the experience of Maine, Nebraska, Health Systems, Site visits, interviews, focus November 29, 2011 -
158 studies states in establishing governance Texas, provider March 21, 2012
Hovey, 2013 groups
structures, technical services to Washington, association, state Not clearly stated but suggests:
enable health information Wisconsin health IT lab exchange, e-prescribing and
exchange, and privacy and coordinators, state exchanging clinical care
security frameworks; 2) Assess public health documents.
stakeholder priorities, current agencies
use, and anticipated need for
information exchange; 3) Identify
common enablers, barriers, and
challenges; and 4) Collect and
characterize lessons learned.
Fairbrother, et al., Cross-sectional Describe the Beacon community Greater Primary care, Interviews Fall 2012
143 program experience Cincinnati area, hospitals, federally
2014 Alerts for diabetic and pediatric
Ohio qualified health asthma patients in ED or
centers and admitted sent to primary care.
community
centers insurance
partners
F-51
Date HIE
Dixon, McGowan, Indiana HIE (IHIE)- includes lab Reporting of all lab data NR, but in 1990s All patients having lab tests
and Grannis, reports
42
2011
Dobalian, et al., One site in Nationwide Health Make inpatient and outpatient data available to ED. Were not 2008 ED patients
142 Information Network, another used yet able to exchange data about patient care.
2012
First Gateways exchange
(HealthView). This specific HIE
was called Long Beach Network
for Health
Dullabh and Not described per state States had two models of HIE: “thin layer” model with services NR NR
158 based on light infrastructure (Texas, Washington and
Hovey, 2013
Wisconsin), or a heavy infrastructure model (Nebraska and
Maine) with features such as a central repository"
Fairbrother, et al., 87 primary care, 18 hospital, 7 Data exchange, registries, alerts to PC practices when patient September 1, 2010 - Adult diabetics, pediatric
143 federally qualified health centers in ED or admitted to hospital. March 31, 2013 asthma patients
2014
and community centers, 3
insurance partners
F-52
Dixon, McGowan, 7.6 million lab reports from 168 hospitals and lab All laboratory values NA Proportion of fields in lab
and Grannis, information systems, of which 16,365 from 49 hospitals reports that were complete
42 and lab information systems were enhanced by a
2011
Notifiable Condition Reporter
Dobalian, et al., N=18 to sample NR NR Participants in LBNH vs. not in

142 LBNH
2012
Dullabh and N=105 to sample; no response rate reported. NR NR Comparison of 5 states

158
Hovey, 2013
Fairbrother, et al., N=38 interviews to sample Adult diabetics, pediatric asthma NR NA

143 patients
2014
F-53
Dixon, McGowan, Comparison of completeness of lab test 19 data elements NA Quantitative
and Grannis, results for regular and enhanced systems Completeness of data fields
42
2011
Dobalian, et al., Descriptive narrative only NA NA Qualitative

142
2012
Dullabh and Descriptive narrative only NA NA Qualitative

158
Hovey, 2013
Fairbrother, et al., Descriptive narrative only NA NA Qualitative

143
2014
F-54
Risk of
Dixon, McGowan, -Patient identifiers and test, name, and results were nearly 100% complete for both; most but not all measures more complete for Low
and Grannis, enhanced system
42 -15 of 18 record fields showed improved completeness with enhanced system. Units of measure, normal range and abnormal flag fields
2011
all showed reduced completeness with enhanced system. No tests of statistical significance performed.
Dobalian, et al., "Despite a limited concentration on ED care, virtually all respondents noted concerns regarding the sustainability, or business case, for the NA
142 exchange of health information."
2012
Dullabh and "Results show the last 2 years have seen unprecedented growth in HIE infrastructure. Key factors such as maturity of HIE at baseline and NA
158 healthcare market characteristics have shaped governance models and technical infrastructures." "Given the significant concerns about
Hovey, 2013
sustainability and who will pay for state-offered services in the long term, it may also prove beneficial to ensure that states have
assistance, either from state or national informational resources, in developing both sustainability plans and contingency plans."
Fairbrother, et al., Despite some setbacks and delays, the basic technology infrastructure was built, the alert system was implemented, 19 practices focusing High
143 on diabetes improvement were recognized as patient-centered medical homes, and many participants agreed that the program had helped
2014
transform care.
F-55
Feldman and Retrospective To determine challenges and Virginia SSA, MedVirginia Database, interviews, audit June-November 2009
43 cohort successes of HIE for Social HIE, and Bon
Horan, 2011 logs
Security disability determination Secours Health Semi-structured interviews of 43
System individuals from the 3
participating organizations
159 Multiple Case To understand the effects of the Six US States Multiple Site visits, interviews, meetings 2012-2014
Dullabh, 2014
Studies State HIE Program on HIE Iowa, Mississippi,
progress New Hampshire,
Utah, Vermont
and Wyoming
Feldman, Cross-sectional Obtain insights into technical, Virginia Integrated delivery Interviews, observations, August 2012-June
Schooley, and organizational, and governance system documents 2013
144 issues of a large private health
Bhavsar, 2014 Direct observation, informal
system participating in a state information gathering, document
HIE analysis, and semi-structured
interviews
Finnell and Cross-sectional To describe the underlying Indianapolis, EMS providers Survey, database July 1, 2009-
133 technology, the utilization Indiana using tablets December 31, 2009
Overhage, 2010
statistics, and the survey results
from the medics who used an
integrated emergency medical
service point-of-care system and
RHIE system
F-56
Date HIE
Feldman and Medical Evidence Gathering Data for Social Security disability determination transmitted February 2008 Patients being evaluated for
43 Through Health IT (MEGAHIT) from health system through HIE to SSA via NHIN, push of Social Security disability
Horan, 2011
background, lab, and medication data in a CCD from health determination; interviewed
system to SSA included personnel from the 3
participating organizations
159 Multiple Most projects enabled both directed and query-based HIE. Varies State HIE programs supported
Dullabh, 2014
While services varied they included care summary exchange, by the Office of the National
lab results, public health reporting, and transmission of Coordinator (U.S. Federal
admission/discharge/transfer messages. Government).
Feldman, ConnectVirginia EXCHANGE Query of Continuity of Care Documents August 2012 All patients in Invoa IDS
Schooley, and
144
Bhavsar, 2014
Finnell and 30 hospitals, 5 health systems, EMS providers use a button that links to the Indiana Network Started in 1994 Number of patients who were
133 Marian County Health Department for Patient Care (INPC). Data are stored in a secured, seen by EMS.
Overhage, 2010
and various physician practices. password protected, centralized database. Medics receive a
data abstract (pdf) of patient demographics, lab, ED, inpatient,
chief complaint, coded diagnoses and procedures.
F-57
Feldman and 203 Members of 3 organizations NA None
43
Horan, 2011
159 Programs In 6 states States not included in prior States included in prior Programs were compared
Dullabh, 2014
rounds of case studies. case studies of this across states in terms of
States were selected for program leadership models and other
variation in program factors, characteristics.
state contextual factors, state
HIE progress,
Feldman, 10 individuals from IDS, HIE, and vendors Members of all organizations None None
Schooley, and
144
Bhavsar, 2014
Finnell and 26,754 patient contacts by medics. Also survey of 58 Invited all 180 medics. 58/180 NR Comparison of use over time of
133 medics on use of INPC responded study.
Overhage, 2010
F-58
Feldman and Technical, organizational, and governance Mean Social Security disability case NA Quantitative, Mixed Methods
43 attributes processing time 59 days (vs. average Development of Collaborative
Horan, 2011
of 84) Enactment Model
159 Provider participation Technical model NA Qualitative

Dullabh, 2014
Critical mass of data exchange Leadership model
Variety and type of stakeholders
Feldman, Technical, organizational, and NA NA Qualitative

Schooley, and governmental attributes Themes extracted from data
144
Bhavsar, 2014
Finnell and Number of unique medic users over 6 HIE use, barriers to use NR Quantitative
133 months, number of INPC requests.
Overhage, 2010 Multivariable analysis
F-59
Risk of
Feldman and -Technical challenges of HIE can be overcome but organizational and governance factors are also important Moderate
43
Horan, 2011
30% decrease in mean case processing time from 84 to 59 days from the usual method to HIE supported method, respectively.
159 Local stakeholder needs in the long and short term influenced decisions NA
Dullabh, 2014
Other factors were cost, privacy and security
Tangible intermediate goals supported implementation.
Providing value and meeting Stage 2 meaningful use criteria were related to estimates of sustainability.
Most programs were planning to use subscription fees for long term financial support.
Feldman, Some technical challenges required workarounds, leadership and adequate resources essential, and appropriate decision making NA
Schooley, and authority required
144
Bhavsar, 2014
Finnell and Over a six month study period, requests for patient data via HIE increased from 15% to 26% per patient contact. The majority of medics Moderate
133 surveyed felt the HIE information was an important for delivering quality patient care.
Overhage, 2010
F-60
84 Cross-sectional Description of projects, stages, Wisconsin Any Survey 2006
Foldy, 2007
users, organizational home, Unable to access due to broken
governance, scope, standards, URL link
drivers, challenges,
recommendations
Fontaine, et al., Cross-sectional Examine factors that motivate or Minnesota Primary care Survey and Interviews November 10, 2008-
85 prevent small primary care practices with <20 February 20, 2009
2010
practices from participating in providers in 1 of
EHR and HIE use as mandated the 3 described
by Minnesota e-Health Law from HIE regions
2007
Frisse, et al., Retrospective To examine the financial impact Memphis, ED Log file January 2007-
44 cohort of HIE in EDs Tennessee December 2008
2012 Tennessee Hospital Association
billing database of all ED visit
records
F-61
Date HIE
84 NA HIE defined as projects in which multiple independent NA eHealth board, staff,
Foldy, 2007
organizations routinely send or receive electronic clinical consultants, workgroup
information about patients for purposes other than billing or members and survey
claims payment respondents all nominated the
survey recipients
Fontaine, et al., Various HIEs 9 primary care practices in Minnesota NR 39 participants in discussions
85 3 HIE initiatives in Minnesota 1) a 10 year old HIO that
2010
promotes HIE and coordinates immunization registry, 2)
network of independent metropolitan community clinics that
received MN e-health grant funding to implement EHRs, 3)
initiative to develop PHR with congestive heart failure patients
Frisse, et al., MidSouth e-Health Alliance 11 of 12 hospitals accessed information through a dedicated 2005 All ED visits
44 (MSeHA) secure web portal. 1 hospital printed encounter summaries as
2012
part of triage for the first 10 months of the study.
Patient demographic, diagnosis, all hospital radiologic and
laboratory reports, most procedure reports, and discharge
summaries are exchanged.
F-62
84 30 Organizations contacted, 27 (90%) responded eHealth board, staff, consultants, NR NA
Foldy, 2007
workgroup members and survey
respondents all nominated the
survey recipients
Fontaine, et al., Unclear NA NA NA

85
2010
Frisse, et al., 15,798 visits in which HIE was accessed; matched ED visit to 1 of the participating Patients in both the HIE Encounters with vs. without
44 comparison group of 15,798 cases hospitals. Visit only in HIE or no and no HIE subset (932) HIE
2012
HIE subset. HIE accessed in non ED
setting (3,555)
F-63
84 -Status of projects operation vs. planned NA NA Quantitative
Foldy, 2007
-Stage of development Descriptive Stats
-Description of information users
-Organization, funding, governance
-Scope
-Standards
-Drivers
-Challenges
-Recommendations
Fontaine, et al., -Use of EHR NA NA Qualitative

85 -What data elements are being
sent/received
Frisse, et al., -Financial consequences based on ED- HIE accessed during ED visit -Admission type Quantitative Multivariate
44 originated hospital admissions -Length of stay
2012 Analysis
-Admissions for observation, lab tests, -Charlson comorbidity index Generalized estimating equation
head or body CT, ankle or chest -Patients matched on age, gender, logistic regression
radiographs, echocardiograms race, site of ED, diagnosis and
payer
F-64
Risk of
84 -27 responded, 21 judged to be HIE organizations, 21 respondents had 16 operational projects, 11 planned projects Moderate
Foldy, 2007
-Rating of most advanced HIE project had 40% of respondents in implementation and 40% in operational
-44% deliver data only to central registries, 50% deliver to providers and registries and only 1 to providers only
-62.5% are based in government organizations
-73% started with only public funds, 20% exclusively private, 75 used both
-For continued operations 57% rely entirely on public funds, 21% only on private and 21% a combo
-Governance all have multiple stakeholders
-14 are statewide, 7 southeast Wisconsin, 2 south, central and north and west.
-Standards 46% of projects have specific vocabulary or data standards
Fontaine, et al., -8/9 practices uses EHR Moderate
85 -Only 1 practice was able to transmit/receive patient health records
2010
-All 9 practices shared information with department of health immunization registry though not through any of the EHRs in the practices
-Labs were next most common Several practices were receiving data directly into EHRs
-None were sharing data with nonaffiliated practices
-HIE motivations themes: External - government mandates, payer mandates, quality reporting; Internal - cost savings, quality/patient
safety, efficiency
-HIE barriers: lack of interoperability, lack of buy-in, competition, security, costs, creating business model, limited success and large time
investment, limited technical support
-No practice was fully involved in a regional HIE; HIE was not part of most practices’ short-term strategic plans.
Frisse, et al., HIE accessed: 6.8% of ED visits (in 12 EDs) Moderate
44 Admissions when HIE used
2012
Adjusted OR 0.27; 95% CI, 0.210 to 0.351, p<0.0001
191 fewer admissions with HIE vs. without HIE
-In 11 EDs directly accessing HIE data only through a secure Web browser, access was associated with a decrease in hospital
admissions (adjusted OR 0.27; p<0001)
-In 12th ED relying on print summaries, HIE access was associated with a decrease in hospital admissions (OR 0.48; p<0001) and
statistically significant decreases in head CT use, body CT use, and laboratory test ordering
-HIE access associated with annual cost savings of
$1.9 million, with hospital admission reductions accounting for 97.6% of total cost reductions
F-65
Furukawa, et al., Time Series Describe extent of HIE in U.S. All 50 states and Hospital Survey 2008-2012
111 hospitals the District of
2013 Health IT supplements to the
Columbia American Hospital Association
Annual survey of hospitals, 2008-
2012. 63% response rates.
2,805 hospitals in 2008, 2,836
hospitals in 2012. nonfederal
acute care hospitals
Furukawa, et al., Cross-sectional NAMCS Survey, How have rates U.S. U.S. ambulatory Surveys 2009-2013
110 of EHR changed since HITECH? providers
2014
What % of MDs are engaged in
HIE in 2013? What % are using
PHR in 2013? How did these
things vary by physician and
practice characteristics?
Gadd, et al., Cross-sectional To assess the usability of an HIE 3 counties around ED and outpatient Survey June-November 2009
86 in a densely populated Memphis, clinics
2011 Email survey responses from
metropolitan region Tennessee 165/ 237 health care
professionals (70% response
rate)
Genes, et al., Cross-sectional What are perceptions of ED New York City ED Interviews NR
145 users of HIE?
2011 Semi-structured interviews of
users and nonusers
F-66
Date HIE
Furukawa, et al., NA NA NA U.S. acute care nonfederal
111 hospitals
2013
Furukawa, et al., NA NA NA Ambulatory physicians not

110 radiologists, pathology, or
2014
anesthesia
Gadd, et al., MidSouth e-Health Alliance Consolidated data from multiple hospital EDs and community- 2004 in 3 counties Medical staff (Physicians,
86 (MSeHA) based ambulatory clinics. Decentralized, query-based Nurse Practitioners,
2011
A rapid deployment HIE that exchange. Physicians assistants, nurses,
consolidated data from several Consent was opt-out. and other) at organizations
sources participating in the HIE
Genes, et al., New York Clinical Information All data from 10 academic medical centers 2009 ED physicians
145 Exchange (NYCLIX)
2011
F-67
Furukawa, et al., 2,805 hospitals in 2008 and 2,836 in 2012 NA NA NA
111 Various HIEs
2013
Furukawa, et al., NR NA NA NA
110
2014
Gadd, et al., 162 responses analyzed NR other than list of roles People who were no longer The impact of usability on use
86 Details on sample: 345 people identified; 269 valid included employed by the system of HIE
2011
contacts; 237 surveys distributed; 165 responses were not contacted
(69.6%); 3 excluded for missing responses on
satisfaction items.
Genes, et al., 18 users of NYCLIX ED pilot All users NA -For users, was HIE data
145 useful?
2011
-For nonusers, why not using?
F-68
Furukawa, et al., Any exchange activity with outside NA -Provider type Qualitative
111 providers outside the organizations -Organizational affiliation
-Type of clinical information
-Hospital characteristics
-Area characteristics
Furukawa, et al., Descriptive statistics NA NA Quantitative

110
2014 Descriptive statistics and logistic
regression
Gadd, et al., -Use None None Quantitative, multivariable

86 -Questionnaire for User Interaction
2011 analysis
Satisfaction (QUIS 7.0) -Wilcoxon rank sum test
-Trust -Descriptive statistics
-Ordinal logistic regression
Genes, et al., -For users, was HIE data useful? Semi-structured interviews None Qualitative
145 -For nonusers, why not using?
2011
F-69
Risk of
Furukawa, et al., -58% of hospitals exchanging in 2012, 41% increase of 2008, p<0.01 Low
111 -2012 51% hospitals exchanged with unaffiliated ambulatory providers, 36% with other hospitals outside their organization
2013
-2012 52%, 53%, 35% and 33% exchanging radiology reports, labs, care summaries and prescription lists with outside providers,
respectively. That is a 39%, 51%, 40%, 55% increase, respectively.
-After adjusting for hospital and area characteristics hospitals with basic EHR and participation in Health information organizations had
highest rates of exchange activity in 2012, 80% of hospital with EHR and HIO were exchanging, 71% with HIO but no EHR were
exchanging 60% of hospitals with EHR but no HIO were exchanging, all consistent across different providers types and clinical information
types
-Hospital characteristics associated with lower exchange rates, rural, for-profit, locations with greater Medicare part A spending
Furukawa, et al., -Broad HIE definition (39% of office-based physicians reported having an HIE with other providers or hospitals). Increased odds of HIE Low
110 both within and outside of their organization with larger practice, health-system owned practice and multispecialty practice. Very few
2014
characteristics associated with HIE outside of the practice, significantly lower outside HIE with community health centers and practice
outside of metropolitan statistical centers
-35 % HIE inside, and 13% HIE outside
Gadd, et al., 151 users (93%), 11 non users Low

86 Average usage per week
2011
<1 hour: 65 (43%)
1 hour to <4 hours: 58 (39%)
≥4 hours: 27 (18%)
Mean usability scale: 6.5 SD 1.4 (>5 is favorable, out of 9)
Association of Scales with higher use (ORs)
Overall reactions: 1.50, p<0.01
Learning: 1.32, p<0.05
System functionality: 1.34, p<0.01
Trust not predictive of usage. Users commented that HIE needs more tech support and could use more types of data
Genes, et al., -Half of users reported usage affecting patient care on ≥1 occasion Low
145 -nonusers reporting forgotten login credentials
2011
F-70
Goldwater, et al., Cross-sectional Evaluate the progress of the HIE, Washington, 6 acute care Interviews, focus groups, July 1, 2013-January
146 how many providers and District of hospitals 6, 2014. Survey of
2014 survey
hospitals were participating in the Columbia Demographic, inpatient, 148 individuals and
program, and what benefits were encounter notifications, lab stakeholders released
being realized through the use of testing, electronic prescribing October 1, 2013 and
the HIE. services, integration with public closed November 4,
health and Medicaid providers. 2013.
Greenhalgh, et Mixed-method; 1) What is usability, use, 3 districts within ED and Qualitative data: 2009-2010?
121 multi-level case functionality, and impact of SCR; the English unscheduled care 140 interviews of policy makers, Not quite clear
al., 2010
study of 2) What explains variation in its National Health managers, clinicians, software
England's adoption and use; Service suppliers;
Summary Care 3) How has the programme been 2,000 pages of ethnographic field
Record (SCR) constrained by influences at the notes;
macro, meso, micro level; Observation of 214 clinical
4) What are the transferable consultations;
lessons for practice and policy? 3,000 pages of documents.
Quantitative Data:
416,325 encounters in 3
participating clinics
Grossman, Multiple case Compare differences in success Indiana, Any Interviews of stakeholders February-August
Kushner, and studies and barriers for HIEs Cincinnati, 2007
November, Northeast
160 Tennessee,
2008
Tampa Bay
F-71
Date HIE
Goldwater, et al., The 6 acute care hospitals chose Demographic, inpatient, encounter notifications, lab testing, Launched February Survey sent to 148, 30
146 the Chesapeake Regional electronic prescribing services, Integration with public health 2012 completed 20% response rate
2014
Information System for our Patients and Medicaid providers.
Greenhalgh, et SCR, which was comprised of 3 Not specified 2007-2010 2007-two early adopter clinics;
121 data fields - medications, allergies 2010 - 113 of 152 primary care
al., 2010
and adverse reactions trusts in England had
committed to participating; by
2010, 16 had begun to create
SCRs;
By 2010, 1.5 million records
had been created.
Grossman, IHIE, HealthBridge, CareSpark, All types Varying Stakeholders in 4 HIEs

Kushner, and Tampa Bay RHIO
November,
160
2008
F-72
Goldwater, et al., NR NR NR NA
146
2014
Greenhalgh, et 1.5 million records in 2010 3 districts who were Not specified None
121 implementing SCRs
al., 2010
Grossman, 2 mature and 2 newer NA None None

Kushner, and
November,
160
2008
F-73
Goldwater, et al., Descriptive narrative only NA NA Mixed Methods
146
2014
Greenhalgh, et What is usability, use, functionality and None None Qualitative

121 impact of the SCR;
al., 2010 Interpreted and themed
What explains variation in adoption and Quantitative
use;
Descriptive statistics and logistic
How does context play in;
regression
What are the lessons to practice and policy
Grossman, Success, barriers, sustainability NA NA Qualitative

Kushner, and
November,
160
2008
F-74
Risk of
Goldwater, et al., "HIE is used to electronically capture and report immunization data; and in requiring electronic lab reporting and results as part of the Moderate
146 Meaningful Use Requirement—which can assist in detecting HIV/AIDS and providing better care for the district’s high population of
2014
individuals with HIV/AIDS. Electronic lab reporting and electronic prescribing within the HIE can assist the Department of Health and
providers in identifying specific diseases, such as tuberculosis and viral hepatitis, before they affect a significant part of the population. '
Greenhalgh, et Adoption was complex, technically challenging, labour intensive; Low

121 Went more slowly than planned;
al., 2010
SCR accessed in 4% of all encounters;
SCR accessed in 21% of encounters where an SCR was available;
Main determinant of success was clinician characteristics (which were not specified);
When available, clinicians accessed SCR 0% to 84% of time;
SCR supported better quality care and increased clinician confidence;
No direct evidence of improved safety;
SCR not associated with shorter clinical consultations;
Successful implementation hinged on successful interactions among multiple stakeholders (clinical, technical, political)
Grossman, Stakeholder buy-in essential for success, offering hospitals value to reduce costs important, hospitals concerned about controlling access NA
Kushner, and to data, employers and health plans not buying in
November,
160
2008
F-75
Gutteridge, et al., Cross-sectional To describe the development and New York ED, hospital, and Subscription lists and reports March 11, 2013-
112 use of a CEN system based on metropolitan are outpatient March 2, 2014
2014 generated
an HIE.
Hamann and Secondary To examine ownership U.S. Nursing homes Surveys Nursing home:
Bezboruah, analysis of cross- differences (for-profit; nonprofit) and residential 2004 National Nursing Home August 2004-January
113 sectional survey in the use of technology in long care 2005
2013 Survey; 2010 National Survey of
term care facilities Residential Care Facilities Residential care:
2010
Herwehe, et al., Cross-sectional To conduct a formative evaluation Louisiana Health Interviews, focus groups, log February 1, 2009 and
124 of an HIE for HIV that integrates department, January 31, 2011
2012 data
public health and clinical hospital,
information outpatient
F-76
Date HIE
Gutteridge, et al., Healthix A federated architecture for data sharing. Log in is via a 2004 was initial Geriatric patients seen in ED
112 standa lone web portal funding and admitted to hospitals
2014
-Healthix included a total of 107 organizations with 383 CEN system March
facilities, 9.2 million patients, and >6,500 users performing 2013
>10,000 patient searches per month as of January 2 014
Hamann and Varies, NR Varies, NR Varies Long term care Facilities

Bezboruah, Nursing home is U.S.
113 Residential Care (aka Assisted
2013
Living in U.S.)
Herwehe, et al., The Louisiana Public Health A secure bi-directional public health informatics application (an Started February Patients with HIV seen for non
2012
124 Information Exchange HIE in a broad sense, as defined by Dixon et al.), linking 2009 and in all HIV services at 7 Louisiana
(LaPHIE) statewide public health surveillance data with patient-level EMR participating hospitals Hospitals; 442 clinicians (206
data. by September 2009 physicians and 236 nurses)
trained on system to serve as
peer trainers
F-77
Gutteridge, et al., These patient who are enrolled in the system NA NA None
112
2014
Hamann and Nursing home Sample: 1,174 response rate 81% NR NR Nonprofit vs. for profit use of
Bezboruah, Residential care Sample: 2,302 response rate 81% health IT including HIE
113 Various HIEs
2013
Herwehe, et al., 16 focus groups n=149; and 23 key informant NA NA NA

124 interviews with patients
2012
F-78
Gutteridge, et al., -Enrollment of patients NA NA Counts
112 -Number of notifications sent
2014
Hamann and Whether facility shares information Nonprofit or for-profit ownership -Chain ownership Quantitative
Bezboruah, electronically with other care partners and -Size of facility and type of 2
-Chi
113 the extent of HIE defined as the number of residents
2013 -Ordered Logit regression
entities with which the facility shares -Use of volunteers
information -% revenue from Medicaid and
Medicare
Herwehe, et al., Patients identified and matched providers NA NA Mixed methods

124 responses to alerts
2012 -Description
-Counts of alerts and responses
F-79
Risk of
Gutteridge, et al., -5,722 patients enrolled (612 notifications sent) NA
112 -Without duplications 497 event notifications about 206 unique patients
2014
-Notifications originated from 23 separate institutions, ED visits comprised 44% (219 of the 497 notifications), 98 notifications were for
inpatient admissions
-121 of 497 (55%) during normal business hours
-Hospital admissions resulted from 45% of ED visits; 17.8% of these lasted <48 hours, suggesting they were avoidable
-70% of notifications were received within 1 hour of the event, during the study year; in following year 71% were received within 15
minutes
Hamann and For Profit/Nonprofit (corrected F) Low

Bezboruah, % Residential care using HIE: 0.14/0.21 (10.29), p=0.00
113
2013 Number of partners in HIE: 0.32/0.42 (2.56), p=0.02
Regression results: for profits less likely to participate in HIE OR 0.663, p<0.001
Supports hypothesis and proposed framework for why nonprofits are more likely to use health IT
NOTE: NH survey did not have HIE question
Herwehe, et al., In the 2 year period 2/1/2009 to 1/31/2011: NA

124
2012 -488 registrations of patient (345 unique patients) with HIV identified
-Clinicians responded to 73% of alerts and documented actions on note that was shared with public health
-Results include statement that 'no negative feedback has been received from providers' with no detail
-Summary of patient interviews found general acceptance of data sharing as long as there was patient benefit and a preference for care in
the healthcare verses the public health system
-Challenges: concerns about data ownership and ethics and disparate data systems, but these are reported as challenges they were able
to address
F-80
Hessler, et al., Cross-sectional To understand assessment of U.S. RHIOs and State Survey late February 2007-
87 HIE by RHIO and state and local and Local Health March 25, 2007
2009 Online survey created by
public health department Departments researchers
representatives
Hincapie, et al., Cross-sectional Assess perceptions of physicians Arizona All physician use Focus group meetings of 29 NR
132 users of HIE physicians on HIE quality of care,
2011
workflow and cost
Hyppönen, et al., Cross-sectional To compare usability of different Finland Varies as this Survey 2010
133 regional health information includes sites with
2014
exchange system (RHIE) types RHIE
as well as the factors related to
the experienced level of success
Jha, et al., Cross-sectional, To assess health IT, including U.S., U.K., Physicians and Literature review, available Literature review:
117 mixed modes HIE adoption in 7 countries Canada, hospitals surveys, (Medline and Google) 2000 -2006
2008
Germany, and interviews with governmental
Netherlands, and nongovernmental experts
Australia, New
Zealand
F-81
Date HIE
Hessler, et al., Varies, NR Varies, NR Varies 164 RHIOs
87 540 health agencies
2009
Hincapie, et al., Arizona Medical Information Medication history, lab test results, and discharge summaries October 2008 Physicians who agreed to
132 Exchange (AMIE) participate in focus groups
2011
Hyppönen, et al., Regional Health Information Varies depending on type of RHIE system. Type 1: master Before 2010 Inpatients and outpatients of
133 Exchange patient index required separate login to centralized database. physicians working in public
2014
Type 2: web distribution model. Limited group of referring sector in 13 regions of Finland
physicians could see hospital info. Type 3: regional virtual where RHIE systems were in
model. If patient grants permission, clinician uses integrated use.
system that includes all inpatient and outpatient information.
Jha, et al., Varies, NR Varies, NR Varies Developed countries

117
2008
F-82
Hessler, et al., N=44 RHIOs (27% response); 20 non-governmental RHIOs: listed in 1 of 7 sources Missing or invalid email RHIOs vs. state vs. local health
87 N=138 Health agencies (26% response); 41 state and Public Health: on list from addresses or an exchange officials
2009
97 local public health agencies national associations specific to 1 disease
Hincapie, et al., 29 physicians Physicians who agreed to use None None

132 system and participate in focus
2011
groups
Hyppönen, et al., 1,693 physician respondents aged less than 65 years. Physicians working in public Physicians in the private Comparison of HIE usability by
133 1,079 specialize care; 614 primary care sector in 13 regions of Finland sector or in regions where type of RHIE and EHR
2014
where RHIE systems were in RHIE not in use whole
use. region or was unavailable
Jha, et al., 7 selected for data availability NA NA HIE use across countries
117
2008
F-83
Hessler, et al., -Sharing of data Type of respondent Characteristic reported but not Mixed Methods
87 -Challenges used in analysis
2009 -Descriptive statistics, no
-Unique resources significance tests
-Minimal requirements -Qualitative assessment of open-
ended responses
Hincapie, et al., Benefits and disadvantages of HIE Transcripts NA Qualitative

132
2011 Thematic analysis from
transcripts
Hyppönen, et al., Levels of agreement to 11 statements RHIE type used, local EHR system Managed multi-collinearity Quantitative, multivariable
133 about HIE success used, working sector and primary
2014 analysis
means of HIE Models to predict successful HIE,
stratified by type of clinician user
(specialized or primary care).
Results were broken out by
function of HIE.
Jha, et al., -HIE existence Country NR Descriptive, qualitative

117 -Use
2008
-Policies promoting development
F-84
Risk of
Hessler, et al., Public Health: 50 (36%) no RHIO in jurisdiction; 16 (12%) no relationship with RHIO; 26 (40% responding to item) are exchanging High
87 information
2009
RHIOs: 12 (60%) are exchanging info; 7 (35% with public health); lab data shared most frequently (86% of the time)
Challenges (RHIO/Local/State % endorsing)
Lack of standards: 33/12/15
Limited resources: 17/67/45
Unique resources Public Health brings
Perspective: 41/45/30
Data: 35/16/39
Minimum Public Health must bring
Commitment: 50/31/23
Funding/sweat equity: 33/43/47
More dialogue about needs and expectations could increase HIE; early successes with lab data could encourage future use.
Hincapie, et al., Benefits included identification of "doctor shopping", avoiding duplicate testing, and increased efficacy for gathering information; Moderate
132 disadvantage was limited availability of data
2011
Hyppönen, et al., Users of three local EHR systems preferred electronic HIE to paper to a larger extend than users of other EHR systems. Experiences with Low
133 an integrated RHIE system (type 3) were more positive than those with other types or RHIE systems.
2014
Jha, et al., Australia: early pilots, but no major investment. Lack of unified patient identification an issue High
117
2008 Canada: province-wide efforts, particularly Alberta; national--early development of Health Infoway but little info exchanged
Germany: most computers with records not connected; Germans have smart cards, but only admin data now
The Netherlands: National SwithPoint pilot with 20% of population, plan full implementation in 2008
New Zealand: planning stage, have unified patient Id, focus of discharge, lab and path reports to GPs
U.K.: National Program, but mostly small amount of data exchanged in more minor programs
U.S.: RHIOs, but <12% of organizations exchanging data and <1% of population involved
F-85
Johnson, et al., Multiple site case To assess first year of MidSouth Memphis, EDs Audit logs, database Implied 1 year after
99 studies eHealth Alliance Tennessee May 2006; but data
2008 (administrative), comments by
users on use in January
2008
Johnson, et al., Multiple site case To explore characteristics of use Memphis, EDs, ambulatory Audit logs, database Interviews 1 month, 1
118 studies and uses of a regional HIE Tennessee groups year after system in
2011 administrative data,
observations, comment cards, use in all sites
feedback in system, interviews, Audit data and ED
observations visits January 2008-
June 2008
Jones, Friedberg, Cross-sectional To evaluate the association U.S. Hospitals Database June 2005-June 2008
and Schneider, et between hospitals’ HIE and 2007 AHA Survey for Hospital Compare
68 health IT use and 30-day risk
al., 2011 2009 September Hospital
adjusted readmission Compare
Kaelber, et al., Cross-sectional What is use and perceived value Northeast Ohio Public healthcare Usage logs, survey of users November 2010-
120 of HIE? system December 2011
2013
F-86
Date HIE
Johnson, et al., MidSouth eHealth Alliance Multiple hospital emergency departments and community- May 2006 ED staff in 5 participating sites
99 (MSeHA) based ambulatory clinics. Decentralized, query-based
2008
exchange.
Data Exchanged: demographics, ICD-9 discharge
codes, lab results, encounter data, and dictated
reports.
These are in a vault controlled by the hospital, but accessed
when a query is made, unless patient opts out.
Johnson, et al., MidSouth eHealth Alliance Data Exchanged: demographics, ICD-9 discharge May 2006 in EDs 6 ED sites and 9 clinics for
118 (MSeHA) codes, lab results, encounter data, and dictated later in clinics (NR) interviews
2011
reports. All visits records and usage
Multiple hospital emergency departments and community- logs
based ambulatory clinics. Decentralized.
These are in a vault controlled by the hospital, but accessed
when a query is made, unless patient opts out.
Jones, Friedberg, Varied. As defined by hospital Varied. As defined by hospital Varied. As defined Hospitals in U.S.
and Schneider, et by hospital
68
al., 2011
Kaelber, et al., HIE in Northeast Ohio 10 hospitals and affiliated practices using Care Everywhere November 2010 Not stated for patient
120 population, 412 physician
2013
users
F-87
Johnson, et al., 5 sites; number of users varies by site NR NR HIE use across sites and
99 overall
2008
Johnson, et al., Number of people interviewed NR NA NA NA

118 369 comments (12% of all visits)
2011
Jones, Friedberg, 2,406 hospitals (58% of eligible hospitals responded to General acute care non federally Not specified. Specialty Hospitals that self report
and Schneider, et AHA survey) owned U.S. hospitals and federal implied by exchanging any information
68 inclusion criteria with ambulatory providers
al., 2011
outside their system vs.,
hospitals who say they do not
participate in this type of HIE
Kaelber, et al., 74 (18%) of physicians who replied to survey All users NA -Measurement of usage
120 -Perceptions of users
2013
F-88
Johnson, et al., -% of ED visits with HIE use NA Role (Nurse, MD, registrar, unit Quantitative, descriptive
99 -% of users who logged in clerk)
2008 statistics
-Theme from comments: perception that Counts and percentages
HIE reduces redundant testing was most
common
Johnson, et al., -HIE Access NA -Profession (Doctors or Mixed Methods

118 -Type of data accessed nurse/clerk)
2011 -quantitative, descriptive data
-Provider log on rates -Type of visit -qualitative analysis
-Counts and percentages
Jones, Friedberg, All- cause 30-day risk-standardized HIE Participation Hospital characteristics Quantitative
and Schneider, et readmission rates for patients initially (also use of health IT) (ownership, critical access status, -Unadjusted mean differences
68 admitted with acute myocardial infarction, trauma status, number of beds,
al., 2011 -Propensity score matching
heart failure, or pneumonia. teaching status, system
-Linear regression
membership, core-based statistical
area type, U.S. census division,
long term care unit, critical care
unit)
Kaelber, et al., -Measurement of usage -Usage of HIE None Quantitative
120 -Perceptions of users -Survey of users
2013 Descriptive and Multivariate
F-89
Risk of
Johnson, et al., HIE viewed in 2.6% of all visits and 9.5% of visits where patient had visit to other site in past 30 days. NA
99
2008
% of total users who logged on ranged from 0 in one site where the high was 12% to 75% by unit clerks in a site that had high use by other
professions
-MSeHA was used for 3% of all visits

-The site with the highest usage had registrars looking up HIE data when patient arrived at the ED
-The site that mostly serves pediatric patients used MSeHA the least vs. other sites
Johnson, et al., HIE access NA

118
2011 Patient encounters increased over 24 months: 4% to 6.5% (range: 1 to 16 % across sites)
14.6% for return ED visits and 18.7% for return clinic visits (p<0.001)
Higher where nurses and clerks involved and lowest where MD only access
Patient opt out rates: 1% to 3%
Primary user reported consequence of HIE: provided additional history (29%), prevented repeat test or procedure (19.8%)
Jones, Friedberg, Unadjusted readmission rates (no HIE vs. HIE) Low
and Schneider, et Acute myocardial infarction: 20.0 vs. 19.8, p=0.14
68
al., 2011 Heart failure: 24.6 vs. 24.3, p=0.003
Pneumonia: 18.2 vs. 18.1, p=0.68
Hospitals did not participate in HIE: 58.7%
Adjusted readmission rates (no HIE vs. HIE)
Acute myocardial infarction: 19.9 vs. 19.8, p=0.18
Heart failure: 24.4 vs. 24.2, p=0.11
Pneumonia: 18.2 vs. 18.1, p=0.68
Kaelber, et al., Usage of HIE NA

120 ED: 31% to 35%
2013
Primary care: 18% to 22%
Specialty care: 9% to 11%
-Usage highest among patients who were older, with more comorbid illness, Medicare/Medicaid insured, and black
-Self-reported impact was more efficient care (93%), time savings (85%), prevented admissions (15%), decreased tests ordered (84%),
decreased imaging ordered (74%), and improved care in other ways (82%)
F-90
Kaushal, et al., Cross-sectional To assess users experiences with Massachusetts 5 Massachusetts Survey December 2005
60 an HIE project that provided Emergency
2010 Semi-structured interview
medications information to EDs. Rooms covering need for intervention,
history, personal use, induction,
current us, completeness and
accuracy, value added, rollout to
other hospitals and evaluation
Pharmacy benefit claims data
Kern, et al., Prospective To determine predictors of New York Varies (setting Survey and administrative data Phone Interviews
171 cohort sustainability among community- was part of January-February
2011 Baseline assessment and New
based organizations analysis) York State Department of Health 2007 (same as
implementing health IT including information on awarded grants baseline for Kern,
Same as Kern, et
173 HIE in a state with significant 2009).
al., 2009 funding of such organizations. New York State
Department of Health
data: March 2008
Kern, et al., Time series To identify lessons for state- New York NR Organizational assessment Baseline: January-
173 based initiatives that can be February 2007
2009 Baseline and followup
learned from HEAL NY assessments Followup: July-August
2008
Kern, et al., Retrospective To determine the effect of HIE on Hudson Valley Physician small Log file January 2005-June
45 cohort ambulatory quality region, New York group practices 2006 (split into 3 6-
2012 From Portal for usage, MVP
Health Care Quality Reports month periods)
including HEDIS measures and
satisfaction
F-91
Date HIE
Kaushal, et al., MedsInfo-ED, a project Claims data from pharmacy benefit managers (PBMs) were 2004 Staff at participating sites
2010
60 Massachusetts Health Data made available at the point of care to clinicians in the EDs
Consortium (MHDC)
Kern, et al., Varies NR Varies HEAL 1 Grantees given

171 awarded funds for health IT
2011
Same as Kern, et
173
al., 2009
Kern, et al., Varies NR Varies HEAL Grantees given

173 awarded funds for health IT
2009
Kern, et al., MedAllies Portal Internet-based with secure log-in from any computer. 2001 Taconic Independent Practice
45 covers 2 counties, 5 hospitals, and Providers can view tests and results order by themselves or Association MDs
2012
2 labs others.
F-92
Kaushal, et al., N=12 interviewed of 15 contacted 3 EDs that were pilot sites; 2 Patients not covered by Comparisons across the 3
60 more added in expansion. participating plans initial pilot sites
2010
Agreement to participate from
MassHealth and 5 health plans.
Kern, et al., 26 Phase I grantees (100%) HEAL 1 Grantee NA Organizations that received
171 further funding vs. those that
2011
did not
Same as Kern, et
173
al., 2009
Kern, et al., 26 HEAL grantees NA NA NA

173
2009
Kern, et al., 138 MDs with quality information (out of 168, 82%) 79 ≥150 patients with MVP Health No quality of care data Physicians who used portal vs.
45 nonusers and 59 users of the HIE portal Care those who did not
2012
F-93
Kaushal, et al., Descriptive narrative only NA NA Thematic analysis
60
2010 Coding of interview transcripts by
tow investigators
Kern, et al., Receipt of HEAL 5 funds -Responses to 26 questions covering 9 NA Quantitative multivariate
171 areas
2011 analysis
-Type of organization that was the lead -Bivariate and multivariate logistic
application (health care or health regression
Same as Kern, et
173 information) -Backward stepwise elimination
al., 2009
Kern, et al., -Grantee still in operation NA None reported Quantitative

173 -Exchanging data or implementing other IT
2009 -Counts and proportions
-Met definition of RHIO -McNemar 2-sample test for
binomial proportions for matched-
pair data for comparison between
baseline and followup
Kern, et al., -Rate of portal use Any portal use -Physician characteristics Quantitative
45 -Quality of care -Case mix 2
2012 -Chi
-t-tests
-Fischer exact tests
-Generalized estimating equation
regression
F-94
Risk of
Kaushal, et al., Need: respondents believed gaps in medical information are an important problem and this system could help High
60
2010 Information was perceived as accurate, range of estimate of patients with information 15% to 80%
Perception: system improved knowledge but did not decrease time and did not improve care enough to justify hospital paying for system
Barriers: need for patient consent, difficulty matching patients
Suggestions: increasing the types of information included (e.g., psychiatric, HIV, and mail order medications) and improving the format of
the output
Kern, et al., Predictors of funding from bivariate (OR, 95%CI) High

171
2011 Lead by health information organization: 11.4, 1.7 to 78.4, p=0.01
Performed community-based needs assessment: 5.1, 0.8 to 32.3, p=0.08
Same as Kern, et Targeting long term care settings: 0.14, 0.02 to 0.79, p=0.03
173 Predictors of funding from multivariate (OR, 95%CI)
al., 2009
Lead by health information organization: 6.4, 0.8 to 52.6, p=-.08
Kern, et al., -All grantees still existed at followup Moderate

173 -Half decreased number of planned projects (3 possible: HIE EHR, electronic prescriptions)
2009
-HIE all grantees planning at baseline, 85% at followup (22 of 26)
-9 (35%) had users ranging from 5 to 1600. HIE was most common project.
-13 baseline/20 followup met definition of RHIO
-Expected interventions (not just HIE) to save money: 65% baseline, 35% followup p=0.02
-Concern about financial and technical barriers increased by followup
Kern, et al., -% of MDs using portal: 33% months 1-6 vs. 42% months 7-12 vs. 43% months 13-18 Low
45 -Mean days logged in per month by MD: 8 (SD 6)
2012
-Quality score at followup: 49 for nonusers vs. 64 for users, p<0.0001
-OR for higher quality use of portal: 1.42 (95% CI, 1.04 to 1.95)
-Average ambulatory quality of care for composite of 15 measures, stratified by time and use of HIE showed difference between non-users
vs. users (49% vs. 64%, p<0.0001) at followup and among users between baseline vs. follow-up (57% vs. 64%, p<0.001)
F-95
Kern, et al., Cross-sectional To understand which components NA Ambulatory, Literature and expert April 2007 (expert
147 of EHRs and HIE are most likely inpatient, and ED review)
2012 consensus
to drive financial savings in the settings. Literature search results, input of
ambulatory, inpatient, and ED 28 national experts, analysis of
settings. Stage 1 of Meaningful Use
88 Prospective To describe the use of an HIE for Indianapolis, Hospital and Survey, log data June 2007-June 2010
Kho, et al., 2013
cohort tracking patients with Indiana associated clinics
antimicrobial resistance
Kierkegaard, Multiple site case To investigate how HIE can better 3 communities ED and Observations, interviews May-June 2013
Kaushal, and studies meet the needs of care (RHIOs) in New outpatients 2 day site visits, onsite and
127 practitioners York State
Vest, 2014 telephone interviews with HIE
users and nonusers,
observations of workflow
Lammers, Adler- Cross-sectional To evaluate whether HIE is California and EDs Database 2007-2010
Milstein, and associated with decreases in Florida State ED databases, Health
69 repeat imaging in EDs
Kocher, 2014 Information Management
Systems Society data, AHA
annual survey
F-96
Date HIE
Kern, et al., NA NA NA HIE functions by settings
147
2012
88 Indiana Network for Patient Care 5 hospital systems (17 hospitals) May 2007 for this Infection preventionists at all
Kho, et al., 2013
(INPC) tracking function hospitals; patients with MRSA
or VRE
Kierkegaard, NA 2 federated model, 1 centralized model. NR 11 RHIOs in NY and users

Kaushal, and All required login to standalone web portal and non users of HIE
127 2 provided automated delivery of imaging and lab results
Vest, 2014
1 included patient portal and iPhone app
1 included secure messaging and event notification.
Query- based but also provided direct exchange of CCD
Lammers, Adler- Varies, not a single HIE Varies Varies ED visits in California and
Milstein, and Florida
69
Kocher, 2014
F-97
Kern, et al., Top 10 functions based on researcher ratings In top 10 for function based on: Rating below top 10 High rated functions across
147 1) probability of achieving a setting and between HIE and
2012
benefit, 2) time to benefit, 3) EHRs
probability of measuring a
benefit for initial framework.
Experts added 3 additional
criteria
4) complexity, 5) likelihood of
usage, and 6) expected
magnitude of impact
88 NR NA NA NA
Kho, et al., 2013
Kierkegaard, N= 38 interviews Received HEAL NY funding and NA Themes across sites

Kaushal, and 3 sites (13, 15, 10) been in existence for ≥7 years,
127 and distinct.
Vest, 2014 3 EDs, 7 outpatient
3 types of respondents: MDs, other clinical users,
administrative users
Lammers, Adler- Patients at HIE adopters: 33,084 (11%) ED visits with data in State and ED visits that resulted in 37 EDs that participated in HIE
Milstein, and Patients at non adopters: 274,640 HIMSS, patient had another ED admissions vs. 410 that did not
69 visit in prior 30 days in different
Kocher, 2014
EDs, or selected imaging in
index visit
F-98
Kern, et al., Rating of function Setting type (HIE, EHRs) NA Quantitative
147
2012 ANOVA for scores across
settings
t-tests for HIE, EHRs
comparisons
88 -Number of alerts generated NA NA Counts

Kho, et al., 2013
-Number of patients admitted to multiple
hospitals
-User satisfaction/ burden
-Coordinated antibiotic-resistant infection
tracking, alerting and prevention
Kierkegaard, Themes related to use of HIE Site and type of setting NA Qualitative
Kaushal, and -Thematic analysis from
127
Vest, 2014 transcripts
-Dual coding of interviews
-Iterative coding, grouping of
themes in categories continued
until saturation
Lammers, Adler- Repeat CT, ultrasound or chest x-ray in HIE participation in each year -Patient demographics Quantitative
Milstein, and same body region within 30 days at -Number of days between ED visits Regression with fixed effects and
69 unaffiliated EDs
Kocher, 2014 -comorbidities trends
-Total annual ED discharges
-ED characteristics
F-99
Risk of
Kern, et al., -73 setting-HIE function pairs were identified
147 -Mean function score (range 6 to 18): 13.0 EHR vs. 11.3 HIE, p<0.0001
2012
-No difference in scores across setting (p=0.33)
-High scoring HIE functions: transferring imaging reports (all settings), receiving lab results (outpatient and ED), enabling structured
medication reconciliation
-HIE functions were considered more difficult to implement (complexity and time) vs. EHRs
-HIE is most likely to generate a positive financial effect through its ability to coordinate care among providers. Based on assessment for
EHRs adding decision support to HIE could potentially yield even greater financial returns
88 Over 3 years Low
Kho, et al., 2013
-12,748 email alerts on 6,270 unique patients
-23% (MSRA) and 22% (VRE) had previous history identified at a different hospital system
10 Infection Preventionists surveyed
-All reported email alerts were useful
-Estimated receiving 5 alerts per day; half already known; alerts used to identify patients requiring intervention
-3 said system added time, 1 saved time, 6 neutral
-Most comment recommendation was to add automate capture of lab data
Kierkegaard, Availability of information varied based on patient consent (required in New York State) and healthcare organization participation. Moderate
Kaushal, and USE
127
Vest, 2014 -MDs had low tolerance for search failures.
-Practice staff are important to obtaining patient consent. Where clerks were not trained or supported, fewer patients consented.
-Patients saw providers covered by other exchanges, suggesting need for larger areas
-Physician use HIE less than other clinical users; MDs often delegate the task.
USABILTY
-Login process perceived as a burden
-Slow system response times
Lammers, Adler- Probability of repeat ED imaging (percentage points [95% CI]), relative reduction Low
Milstein, and CT: -8.7 (-14.7 to -2.7), 59%
69
Kocher, 2014 Ultrasound: -9.1 (-17.2 to -1.1), 44%
Chest x-ray: -13.0 (-18.3 to -7.7), 67%
-Repeat tests more likely in large EDs
F-100
Lang, et al., RCT Impact of sending family Montreal, Canada ED and family Database June 2001-April 2002
65 physicians electronic vs. mailed physician
2006 Surveys and determination of
reports of ED visits for their practices patient outcomes
patients
89 Pre-post To understand MD perception South Korea Hospital and Survey, audit logs June 2008 Week 1
Lee, et al., 2012
implementation prior to HIE implementation and ambulatory clinics and 2 (pre survey)
survey post implementation use and Post: NR
evaluation
Lobach, et al., Cross-sectional To describe use of an HIE for Durham County, Outpatient Audit logs September 2006-
100 population health management North Carolina February 2007
2007
F-101
Date HIE
Lang, et al., Adult university teaching hospital in Report of ED visit sent to family physicians NR Patients visiting ED during
65 Montreal 0800-2200
2006
89 Seoul National University Bundag Federated architecture model with ebXML RS and ebSML RIM June 2008 with MDs in hospital (50) and
Lee, et al., 2012
Hospital and 35 clinics standards updates October clinics (147) for pre; MDs
Included demographics, diagnoses, medications, lab results, 2009 using the HIE for post
imaging, treatment, care plans, vital signs, history and
summaries.
Lobach, et al., Northern Piedmont Community The 4 types of data collected by the system include*: 1) 2001 Patients in program
100 Care Network set up a system administrative (demographics and identifiers, services used,
2007
called COACH (Community- provider associations, audit trails); 2) care management (care
Oriented Approach to Coordinated management encounters, health risk and environment
Healthcare) includes 32 private assessment, socio-economic data, special needs, and care
practices, 3 federally qualified management plans); 3) clinical (encounters,
health centers, 4 community problems/procedures, missed appointments, medications,
hospitals, 9 government agencies allergies, laboratory results, disease-specific care plans); and
(county health departments and 4) communication (messages and alerts, referrals, notices of
departments of social services), 1 new information).
academic medical center, and 2
care management teams: Durham
County, North Carolina, Medicaid
F-102
Lang, et al., 2,022 (out of 3,168) patients visiting ED Patients visiting ED Patients in altered mental ED visit summary provided
65 state (129), state of electronically vs. on paper sent
2006
agitation (21), or with by mail
language barrier (29)
89 23 from hospital and 48 from 20 clinics (46% and 33% MD at pilot site <50% of items completed Hospital vs. clinic based MDs
Lee, et al., 2012
response) for pre; 15 from hospital and 25 from clinics
for post out of all MDs using the system
Lobach, et al., 11,899 patients in Durham County in Medicaid NA NA NA

100
2007
F-103
Lang, et al., -Physician satisfaction -Physician satisfaction Physicians already are sent carbon Quantitative
65 -Return visits at 14 and 28 days -Return visits at 14 and 28 days copies of first page of ED note; self- Survey, analysis of followup care
2006
-Duplication of requests for diagnostic tests -Duplication of requests for diagnostic report of followup data
-Duplication of specialty consult requests tests
- Economic -Duplication of specialty consult
requests
89 -Pre: Perceptions Setting (hospital vs. clinic based) -Gender Quantitative

Lee, et al., 2012
-Post: Information transmission rate -Age Fischer exact tests
Information utilization rate -Specialty
Lobach, et al., Sentinel events: resource utilization by None None Quantitative

100 patients (events of commission) that were
2007 Counts, observation
considered excessive (e.g., 3 ED visits in
90 days) or potentially avoidable (e.g., ED
visit for asthma) and that could potentially
be modified by the involvement of care
managers and other providers
F-104
Risk of
Lang, et al., -Reports found to be received, especially in timely manner, and were more likely to be legible, comprehensive, and useful. Moderate
65 -No difference in return visits within 14 and 28 days, although near significance for fewer visits for patients >65 years within 28 days.
2006
-No difference in duplicate test ordering but greater subspecialty consult requests in intervention group.
89 Pre HIE High

Lee, et al., 2012
-Mean Likert scale that HIE is needed (5 strongly agree): 4.2, p=0.8888 for all and by setting. Similar responses about the need for HIE
for specific items (e.g., lab reports) and perceived benefits of HIE.
-Hospital based MDs had higher levels of agreement about concerns related to HIE than clinic based MDs
Post HIE
Most commonly transmitted information differed by setting
From hospital was working diagnosis: 99.5% vs . 70.5% for clinic, p<0.0001
From clinic it was clinical findings: 79.8%, but this did not differ from hospital
The most useful was lab or imaging in both settings but it was more frequently rated as useful by hospitals (88.2% and 72.9% of cased
p<0.0001)
Lobach, et al., In an analysis of 11,899 continuously enrolled patients from a single county over a six-month period 19.3% (2,285 unique patients) had Low
100 7,226 sentinel health events
2007
Frequency of types of events
Hospital admit asthma: 43
Hospital admit diabetes: 76
Low-severity ED: 2, 546
≥2 missed appointments in 60 days: 1,728
Implementation lessons
-Political issues are more challenging than technical issues
-Perceived value of notices was dependent on timeliness and completeness of underlying HIE dataset.
-Difficult to determine who should be notified of these events, how many notices should be resent and how to prioritize them.
F-105
Maass, et al., Cross-sectional Ascertain benefits of HIE when Finland Regional Survey NR
61 they occurred information
2008 Time-motion study of diabetic
system for patients in a health center
exchange of
clinical data
between hospital
and primary care
offices
Machan, Cross-sectional Assess value of different aspects Tyrol region of Regional Survey, interviews May-August 2004
Ammenwerth, and of regional network of hospitals Austria information Initial qualitative development of
Schabetsberger, and physician practices system for survey followed by quantitative
62 exchange of
2006 evaluation of responses
clinical data
between hospital
and primary care
offices
Mäenpää, et al., Retrospective What is impact of a regional Tampere, Finland Hospital district Log file Data collected 2004-
46 cohort health information system on test that includes 1 2008
2011 Usage of HIE and ordering of
ordering and referrals? hospital district laboratory and radiology tests as
and its community well as specialty referrals
health system.
Outpatient
F-106
Date HIE
Maass, et al., Regional information system in Transmission of patient data into physician EHR NR Physicians in health centers in
61 Finland Finland
2008
Machan, Tiroler Landeskrankenanstaleten Transmission of discharge letters and clinical findings from June 2003 General practitioners in Tyrol,
Ammenwerth, and (TILAK) hospitals to general practitioners. Direct exchange via Austria
Schabetsberger, email.
62
2006
Mäenpää, et al., Regional information system in Full medical record in regional information system 2004 About 234,000 inhabitants in
46 Finland hospital district and associated
2011
clinics
F-107
Maass, et al., 20 visits by patients with diabetes NR NR Use of information system and
61 description of benefits
2008
Machan, 4 providers followed by cross-sectional survey of 104 All general practitioners in Tyrol None None
Ammenwerth, and of 242 (43%) providers.
Schabetsberger,
62
2006
Mäenpää, et al., NR NA NA Appointments, ED visits,

46 laboratory and radiology tests
2011
for primary and specialty care
F-108
Maass, et al., Use of information system and description System used and benefits described NA Thematic analysis
61 of benefits
2008 Time-motion study
Machan, -Measurement of overall satisfaction Survey NA Mixed methods

Ammenwerth, and -Desirability for receiving reports -Quantitative, descriptive data
Schabetsberger, electronically -Qualitative, content analysis
62 -Reduced work for filing and archiving
2006
-Leading to improved quality of care
Mäenpää, et al., -Rates of laboratory and radiology test None Use of HIE not correlated Quantitative
46 ordering specifically with outcomes
2011 Log analysis
-ED visits and primary care referrals
F-109
Risk of
Maass, et al., 20 visits, 4 involved use of information system, with 1 allowing faster treatment decision and 3 providing access to latest test results High
61
2008
Machan, Satisfaction with HIE Low

Ammenwerth, and Positive: 66.4%
Schabetsberger, Agreeing desirable for receiving all reports electronically: 83.7%
62
2006 Reporting less work for filing and archiving: 82.7%
Agreeing it led to improved quality of care: 78.8%
Mäenpää, et al., Change in rates of ordering over time (primary vs. specialty care) Low
46
2011 Laboratory tests per appointment: 19.0% vs. 7.0%
Laboratory tests per inhabitant: 19.0%, 17.9%
Clinical chemistry ordering per appointment: 6.6% overall
Clinical chemistry ordering per inhabitant: 17.5% overall
Radiology exams per appointment: -16.4% vs. -11.0%
Radiology exams per inhabitant: -18.9% vs. -1.9%
ED visits: -1%, -16.2%
Primary care referral to specialist per appointment: 43.6%
Primary care referral to specialist per inhabitant: 35.2%
F-110
Mäenpää, et al., Retrospective What is usage of a regional Tampere, Finland Hospital district Audit logs Data collected 2004-
115 cohort health information system for that includes 1 2008
2012 Usage of HIE and ordering of
different amounts of test ordering hospital district laboratory and radiology tests as
and referrals? and its community well as specialty referrals
health system
Magnus, et al., Retrospective To describe patients identified by Louisiana HIV specialty, Log file February 1, 2009-July
47 cohort the LaPHIE system and HIV- inpatient and 31, 2011
2012 ; Herwehe, Alerts for HIV patients that
124 related outcomes associated with outpatient care continue to appear until patients
et al., 2012
LaPHIE over 2 years. within Louisiana receive CD4 or VL testing;
State University actions taken by the provider are
Health Care documented within the structured
Division system. EMR
Includes 7 safety
net hospitals
F-111
Date HIE
Mäenpää, et al., Regional information system in Full medical record in regional information system 2004 10 municipalities;
115 Finland About 234,000 inhabitants in
2012
hospital district and associated
clinics
Magnus, et al., Seven safety-net hospitals; LaPHIE is a secure bi-directional public health informatics February-September HIV patients coming to
47 application linking statewide public health surveillance data 2009 (Herewhe, Louisiana State University
2012 ; Herwehe,
124 with patient-level EMR data. The exchange functions in real- 2012) Health Care Services division
et al., 2012
time throughout the integrated data networks emergency clinics or ED.
departments, primary care and specialty ambulatory clinics,
and inpatient units.
F-112
Mäenpää, et al., NR NA NA Usage of HIE by physicians,
115 nurses, and department
2012
secretaries, and number of
appointments, ED visits, and
laboratory and radiology tests
Magnus, et al., 419 patients in 60 clinics; alerts to 223 clinicians HIV persons identified by HIV patients who had been Time-matched random sample
47 LaPHIE with no CD4 or VL seen within past year and of HIV-infected persons who
2012 ; Herwehe,
124 monitoring in >1 year, were had no break in care of >1 had been seen for HIV care
et al., 2012
followed in 6-month intervals for year since diagnosis within the Louisiana State
retention in HIV specialty care, University Health Care
inpatient and outpatient Services Division integrated
healthcare utilization data network ≥1 within the past
5 years at the time of
comparison.
F-113
Mäenpää, et al., -Rates of laboratory and radiology test Usage of HIE Use of HIE not correlated Quantitative
115 ordering specifically with outcomes
2012 Descriptive statistics and
-ED visits and primary care referrals negative binomial regression
Magnus, et al., 3 Use of LaPHIE Adjusted for demographic and Quantitative

-CD4 <200 cells/mm
47 clinical characteristics 2
2012 ; Herwehe, -VL >10,000 RNA copies/mL -Chi tests, unadjusted logistic
124 and timing of entry into the cohort regression, and adjusted logistic
et al., 2012 -Having been prescribed antiretroviral
treatment during each 6-month interval regression
-Generalized estimating
equations using an
exchangeable correlation matrix
F-114
Risk of
Mäenpää, et al., Usage of HIE (views per year) NA
115
2012 Physicians: 1,333
Nurses: 758
Department secretaries: 497
-No associations detected between use of HIE and test ordering outcomes
References (means one view of the HIE) viewed in primary health care in 2004–2008:
By physicians from n=486 to n=3581
By nurses from n=59 to n=2,3535
By department secretaries from n=26 to n=13,542
References viewed in special care in 2004–2008:
By physicians from n=1,496 to n=25,051
By nurses from n=284 to n=20,587
By department secretaries from n=1,156 to n=6,958
-The HIE utilization rates increased annually in all 10 federations of municipalities, and the viewing of reference information increased
steadily in each professional group over the 5-year study period. In these federations, a significant connection was found to the number of
laboratory tests and radiology examinations, with a statistically significant increase in the number of viewed references and use of HIE.
The higher the numbers of emergency visits and appointments, the higher the numbers of emergency referrals to specialized care, viewed
references, and HIE usage among the groups of different health care professionals.
Magnus, et al., "After adjustment for demographic and clinical characteristics and timing of entry into the cohort, the LaPHIE-identified group remained Low
47 3
2012 ; Herwehe, significantly more likely to be immunocompromised (CD4 < 200 cells/mm ) than their counterparts (OR 3.22, 95% CI 1.72 to 6.04,
124 3
et al., 2012 p<0.001). However, there was improvement over time, with a decrease in odds of having a CD4 < 200 cells/mm at each successive six-
month interval (OR 0.91, 95% CI 0.83 to 0.99, p<0.05). VL proved more responsive to changes in treatment and care; LaPHIE-identified
persons rapidly became similar to their in-care counterparts, with no significant differences between VL, and again, decreased odds of
having a VL > 10,000 copies/mL at each successive interval (OR 0.83, 95% CI 0.73 to 0.93, p<0.01)."
24% of those identified had not had a CD4 count or VL since initial diagnosis. Of remaining 76% who had been in care previously, 55%
had been out of care for ≥18 months. Following LaPHIE identification, 42% had CD4 counts < 200 cells/mm and 62% had VL >10,000
3
RNA copies/mL. Of 344 patients with at least 6 months of followup, 85% had ≥1 CD4 and/or VL after being identified.
F-115
Massy-Westropp, Cross-sectional Pilot the effectiveness of Adelaide, South Link patient health Survey, focus group Piloted over 6 months
134 electronic data linking tools to Australia information 2002-2003
et al., 2005 Email alert to community; remote
assist in the transfer of between the access to hospital reports; flag
information between an acute hospital and community patients; web access
care hospital and the main community to community reports.
regional provider of home-based services sector
care.
McCarthy, et al., Multiple case Factors influencing technical Regions within Any Interviews NR
161 studies architecture, clinical outcomes, Maine, Indiana,
2014 Written and telephone interviews
and challenges for Beacon- Ohio, of implementers of 7 HIEs
funded HIEs Washington,
Pennsylvania,
Oklahoma, New
York
McCullough, et Cross-sectional To assess barriers and benefits San Gabriel Outpatient small Interviews of clinicians, NR
135 to HIE participation in 2 Valley, California practices administrators and office staff
al., 2014
underserved settings and Minneapolis (California) and users
St. Paul, federally qualified
Minnesota health centers
(Minnesota)
McGowan, et al., Cross-sectional To ascertain lessons learned in Vermont NR Interviews and documents and NR
148 the development of Vermont's presentations about the
2007
RHIO development of VTMEDNET
F-116
Date HIE
Massy-Westropp, Public teaching hospital, ED and Email alert to community; remote access to hospital reports; Piloted over 6 months Medical, nursing, and allied-
134 aged home-based care community flag community patients; web access to community reports. 2002-2003 health staff across the
et al., 2005
services organization. organizations
McCarthy, et al., Beacon Communities within Maine, Varied from hybrid-federated to centralized 1994-2009, Operational, technical, and
161 Indiana, Ohio, Washington, depending on HIE clinical leaders of each HIE
2014
Pennsylvania, Oklahoma, New
York
McCullough, et Citrus Valley Health Partners California: Collaborate system. a web-based tool enabling all NR Independent practices serving
135 Federally Qualified Health Center providers to view data exchanged from 3 hospitals, an predominately Hispanic
al., 2014
Urban Health Network (FUHN) anticipated 90 providers, and laboratories in the community patients and federally qualified
and to securely message other providers. health centers developing an
Data are available to be viewed by all participating providers, accountable care organization
regardless of whether a physician is contributing data to the
system.
Minnesota: CentraHealth aimed at enabling electronic
exchange between FQHCs and the hospitals serving
their Accountable Care Organization patients. This system was
in implementation at time of study
McGowan, et al., VTMEDNET (early HIE) and more Federally funded (NLM and AHRQ) initiated by hospitals, but NR NA
148 recent statewide RHIO developed by a coalition. No other detail provided
2007
F-117
Massy-Westropp, 82 medical, nursing and allied-health staff. HIE NR NR 82 respondents of HIE project
134 included up to 4,000 patients. vs. 50 care providers outside of
et al., 2005
Satisfaction survey responses from 55 or 132 nurses, the HIE project
clinicians and allied health staff.
McCarthy, et al., 7 HIEs funded by Beacon Community grants NA None Compared various factors
161 across hybrid-federated vs.
2014
centralized HIEs
McCullough, et N=24 providers, administrators, and office staff in 16 Individuals who would be None None
al., 2014
135 sites involved in adoption decisions
and integration of HIE into
workflows at each organization
McGowan, et al., 5 interviews: 2 CIO of hospitals and 3 key leaders NA NA Description of 2 efforts. Some
148 limited comparison of the 2
2007
F-118
Massy-Westropp, Satisfaction with electronic data linking NA NA Mixed methods
134
et al., 2005 -Quantitative, descriptive
statistics
-Qualitative, content analysis
McCarthy, et al., -Trust Qualitative NA Qualitative

161 -EHR context
2014 Interviews
-Clinical transformation
-Clinical research
McCullough, et Benefits and barriers to HIE use NA NA Qualitative

135
al., 2014 Thematic analysis from
transcripts
McGowan, et al., Facilitators and barriers to creation and NA NA Qualitative

148 implementation
2007 Simple summary of interviews
F-119
Risk of
Massy-Westropp, Provided bar graphs (figures 2 and 3) but not specific quantitative results except for a statement about use and satisfaction. Those who High
134 had embraced the use of the Integration tools were significantly more likely to rate integration higher than those who were not using it as
et al., 2005
often (p<0.001). In the discussion they estimated a 20% savings in staff time.
McCarthy, et al., Hybrid-federated models maintain autonomy, accommodate disparate EHRs, and build incrementally, while centralized models require Moderate
161 trust fabric, leverage common EHRs, and while providing long-run cost-efficiency may require larger upfront investment. Hybrid-federated
2014
models provide most functionality at individual organization level while centralized models leverage value of communitywide data and
usage.
McCullough, et Barriers Low

135 -Lack of well-functioning area-level exchange
al., 2014
-Market characteristics
-Relationships or previous experiences with exchange partners
-Challenge achieving a critical mass of users
-Health IT used
-Data ownership and provider liability concerns
Benefits
-Improved productivity at initial visit
-Improved completeness of records
-Avoidance of duplicative services/patient financial risk
-Improved nonvisit consults
McGowan, et al., Major facilitators for success High
148
2007 -Public awareness
-Provider buy-in
-Benefits understood in terms of patient safety and quality of care
Barriers
-Perceived public perception of privacy issues
-Providers lack working knowledge of HIE concepts
-Need for a sustainable business model is recognized but not solved
-Need for health information to cross state lines
F-120
Merrill, et al., Time series Evaluate the complex dynamics New York State health Interviews, documents 2010-2011
174 involved in implementing department, 3
2013 Lab results and other information
electronic HIE for public health RHIOs for rapid and efficient
reporting at a state health identification, monitoring,
department, and to identify policy investigation, and treatment of
implications to inform similar communicable and emerging
implementations diseases
Messer, et al., Before-after (1) Assess and North Carolina Ambulatory HIV Interviews 2010
138 enhance organizational readiness providers and
2012 -Pre-post survey
to adopt information technology, ancillary care -HIV patient data and lab results
(2) develop a RHIO to share providers
electronic data between
medical and ancillary care
providers, (3) implement the
RHIO
and begin active information
exchange and (4) evaluate the
effect of the intervention on
provider-related attitudes and
satisfaction with information
exchange
F-121
Date HIE
Merrill, et al., 3 RHIOs and New York State Lab results and other information for rapid and efficient August 2007-August Not described but patients
174 Department of Health. identification, monitoring, investigation, and treatment of 2011 who would be reported to the
2013
communicable and emerging diseases health department for risk and
disease.
Messer, et al., Carolina HIV information 1 large academic medical center and 5 AIDS service 2008 organization HIV care providers and
138 cooperative regional health organizations. Used CAREWare from HRSA. Federated, begun ancillary service providers
2012
information organization (CHIC query-based exchange.
RHIO)
F-122
Merrill, et al., NR NR NR NA
174
2013
Messer, et al., 1 large academic medical center and 5 AIDS service Leaders of the individual NA NA
138 organizations mostly providing case management. organizations, HIV providers
2012
Interviews and assessment with 39 stakeholders; pre
and post survey of 29 providers' satisfaction with
HIE, relationships with other providers, barriers.
F-123
Merrill, et al., Descriptive narrative only NA NA Qualitative
174
2013
Messer, et al., -Organization readiness for Charge NA NA Mixed Methods

138 measure
2012 -Quantitative, descriptive data
-Qualitative process summary -Qualitative, theme analysis from
-Provider surveys of effectiveness transcripts.
F-124
Risk of
Merrill, et al., Three casual loop diagrams captured well recognized system dynamics: Sliding Goals, Project Rework, and Maturity of Resources. The Low
174 findings were associated with specific policies that address funding, leadership, ensuring expertise, planning for rework, communication,
2013
and timeline management.
Messer, et al., -Organizational readiness assessment found organizations were well prepared to adopt new technology, in the 4 domains (motivation, Moderate
138 adequacy of resources, staff attributes, and org climate) only motivation was slightly below nationally determined levels. Results were
2012
consistent by agency type and respondent type
-Largely positive response to quality process. Improved sense of mission, more contact with other agencies, better awareness of other
agency roles.
-Providers found increased case manager knowledge of medical care
-Concerns: Initial concerns about confidentiality dismissed over time as trust was built; Respondents noted it is important to manage
expectations upfront; Clinic staff must use 2 systems the EHR and CAREWare which takes effort and increases errors; There was an
unmet need for training for report generation
-Quantitative provider survey: AIDS service organizations and medical providers generally both felt increased ease of data exchanged and
that patient care improved. For AIDS service organizations 7/8 satisfaction related questions improved statistically from pre-post, in clinic
survey 4/8 improved statistically
F-125
162 Multiple case Assessed how well 5 diverse California A captivated Interviews August 2010-April
Miller, 2012
studies California health care entities’ integrated delivery EHR, Patient portal, HIE, 2011
HIE capabilities, policies, and system (Kaiser); a administrative, inpatient,
procedures satisfied the patient physician outpatient. Patients’
and consumer principles as of management medications, allergies, chronic
early 2011. service disease diagnoses, history, and
organization lab results. Providers could also
(Nautilus); a large view hospital radiology reports.
public hospital; a
large Medicaid
HMO; a regional
HIE organization
Miller and Tucker Cross-sectional How does size of user (hospital U.S. Health systems Survey 2007-2009
149 health system or network) affect and networks
2014 Hospital Electronic Health
HIE usage? Record Adoption Database (AHA,
funded by ONC and is intended
to be the most comprehensive
and representative survey of the
state of healthcare IT)
F-126
Date HIE
162 1 capitated integrated delivery Each of the 5 systems had their own HIE. Some used EPIC, NR NR
Miller, 2012
system (Kaiser); a physician Next Gen, Siemen's NetAccess, Axoloti's Elysium HIE software
management service organization
(Nautilus); a large public hospital; a
large Medicaid HMO; a regional
health information exchange
organization
Miller and Tucker Various Various, within-system and out-of-system HIE Various U.S.
149
2014
F-127
162 N=5 organizations; 23 interviews with 18 people NR NR They compared against 9
Miller, 2012
principles e.g., important
benefits for individual health;
important benefits for
population health; inclusivity
and equality; etc.
Miller and Tucker 430 hospital systems, 4,060 hospitals; average system NR None NA
149 contains 6 hospitals and operates in just under 4
2014
regional markets
F-128
162 Discussed each principle and how well it NA NA Qualitative
Miller, 2012
was met Descriptive
Miller and Tucker Self reported internal or external exchange System's size, defined as the number Patient flow, insurance status Quantitative multivariate
149 of data by hospitals of hospitals owned, leased, sponsored (Medicaid, Medicare fractions) per
2014 analysis
or contract-managed by a central capita payroll, physician Unit of analysis is hospital,
organization relationship (independent practice logistic regression p (exchange) =
association, group practice, system size, etc.
integrated salary model);
profit/nonprofit status; specialty vs.
general; IT vendor (HIE capability),
EMR age
F-129
Risk of
162 Discussed each principle. Also discussed challenges and barriers. Moderate
Miller, 2012
Miller and Tucker 68% do internal exchange: HIE increases with system size; each additional hospital in system increases likelihood by 2 percentage Moderate
149
2014 points; increase if nonprofits, decrease w/ more Medicaid, Medicare, unaffected by location in U.S., age of technology, vendor
17% do external exchange: larger hospital systems are less likely to exchange information externally. Each additional hospital in a
system lowers the chance of external data exchange from hospitals in that system by 0.7 percentage points. Not affected by relative
number of outside hospitals; more sharing with number of beds, number of doctors, % Medicare, per capita payroll; regardless of age of
system or size of vendor
-Robust to type of data (demographic or clinical);
-No relation to HMO, PPO, etc.;
-Same effects stronger with higher per capita salaries, suggesting some strategic benefit
F-130
Moore, et al., Cross-sectional To describe the status and New York City Hospital, ED, and System logs November 1, 2010-
106 lessons learned from the out patient April 30, 2011 (6
2012
development and establishment months)
of an HIE based system to alert
ambulatory providers when their
patients are admitted or
discharged from the hospital or
ED.
Myers, et al., Multiple site case Describe how members of HIV Urban settings Hospital specialty Survey and interviews during site July 2008-December
128 studies patients’ care teams perceived and 1 suburban clinics, support visits. 2010
2012
usefulness and ease of use of setting in New services, primary Laboratory, diagnostic, medical,
newly implemented, innovative York, New care clinics, and service utilization; referrals;
HIEs in diverse HIV treatment Jersey, testing sites, ED, and ancillary care support, such
settings. Louisiana, outpatient and as case management, counseling
California, North inpatient clinics, and testing, transportation, and
Carolina Office of Public substance use and mental health
Health, insurers, services
laboratory and
pharmacy services
F-131
Date HIE
Moore, et al., New York Clinical Information -An event detection and notification system based on a RHIO November 2009 63,305 patients enrolled from
106 Exchange (NYCLIX) including major medical centers, primary care 3 hospitals
2012
physicians, a home health care agency, long-term care
facilities and a Medicaid managed care plan
-NYCLIX uses a federated architecture in which the clinical
repository is spread over a collection of “edge servers” that
reside in each of the members’ data centers.
-Alerts are considered 1-to-1 communication between
providers and are limited to name, date and location of service,
so patient consent was not required
Myers, et al., † 5 HIEs, each site designed, tailored, and implemented NR Members of HIV patient care
5 HIEs that were part of the
128
2012 Information Technology Networks enhancements to existing HIEs according to local needs teams
of Care Initiative that included
Bronx-Lebanon Hospital Center,
Duke university; hospitals, the city
of Paterson, Louisiana State
University Health Care Services
Division, NY Presbyterian Hospital,
St. Mary Medical Center
Foundation. Query-based
F-132
Moore, et al., NR NA NA NA
106
2012
Myers, et al., 60 case workers, medical providers, nonclinical staff. Medical providers, case NR Comparison by type of
128 62 of 102 responded (62%) managers and nonclinical responder
2012
members of the participating HIE
organizations
F-133
Moore, et al., Number of events detected overall and per NA NA Quantitative
106 patient
Myers, et al., -10-item perceived ease of use Role NR Mixed methods

128 -10-item perceived usefulness
2012 Quantitative: Descriptive
statistics stratified by role and
analysis of variance comparison
by role
Qualitative: Thematic analysis
of the qualitative data interviews
were organized
F-134
Risk of
Moore, et al., -42,818 events detected, on average 238 per day Moderate
106 -≥1 event: 6,913 patients
2012
-1 event: 1,879 patients
-≥10 events: 623 patients
-Mean events of inpatients who had an event: 7.7 events
-Mean events of all patients: 0.7 events
Myers, et al., Quantitative: vs. medical providers (57%) and case managers (39%) nonclinical staff members (12%) were significantly less likely to report Low
128
2012 that they provided input into the design of the HIE (p <0.008). Mean composite for ease of use was high (3.9/5.0) and no difference by role.
Mean composite for usefulness was also high (4.0/5.0) and no differences by role.
Qualitative: adoption of the HIEs and perceptions of its use and usefulness varied by occupational role of the patient-care team. Also
noticed that case workers outside the clinic used the HIE routinely. Those within clinics used HIE sporadically.
F-135
Nagykaldi, et al., Retrospective Describe a pilot study on a more Central 30 primary care Log file March 2010-June
48 cohort sophisticated architecture that Oklahoma practices, several 2012
2014 Specialty referrals, hospital
may provide a preliminary specialty admissions, prescriptions,
roadmap for building HIE with practices, and the laboratory imaging results, and
intelligence. Norman Physician emergency care
Hospital
Organization
including an
academic hospital
and 11 other
major hospitals.
Morris, et al., Multiple Case To understand the lessons U.S. States Multiple Interviews and Surveys Not reported
163 Studies learned from HIE organizations
2012
and projects that have succeed
and those that have failed.
F-136
Date HIE
Nagykaldi, et al., exHUB Comprehensive patient registry and clinical decision support NR 346 patients from 6 primary
48 SMRTnet is a statewide network tool and reminder system for preventive care and chronic practices. Average age 66.3
2014
that includes 120 healthcare disease management. Preventive Services Reminder System years, 67.1% female, 20%
organizations. ethnic minority
Morris, et al., Closed HIOs include CareSpark. All query based Varies Query based HIE project in
163 Consolidated HIOs include U.S.
2012
Minnesota HIE (MN HIE) and
Galveston County HIE. Additional
HIOs were studied but declined to
be included in the public report.
Successful HIOs include:
Chesapeake Regional Information
System for Our Patients (CRISP),
Delaware Health Information
Network (DHIN), HealthInfoNet,
Indiana Health Information
Exchange (IHIE), Michiana Health
Information Exchange, and
Rochester RHIO.
F-137
Nagykaldi, et al., 346 patients NR NR Before and after HIE
48
2014
Morris, et al., 9 HIEs provided data that they permitted to be reported HIE organizations that ceased HIE organizations that Successful to failed HIE
163 publicly. operations, merged or continued refused to have their organizations
2012
to operate at the time of the information made public
study
F-138
Nagykaldi, et al., -Time-motion studies Before and after SMARTnet employed NR Quantitative
48 -Complete documentation on preventive
2014 Descriptive
screenings and flu vaccinations
-Medication reconciliation
Morris, et al., Whether the HIE organization continued to Ability to make changes to technology NA Qualitative
163 operate Ambulatory practices participation
2012
Payers participation
Months to deployment
Months to live data
Months to live clinical data
F-139
Risk of
Nagykaldi, et al., All increased significantly (p<0.001 from pre to post) Moderate
48
2014 Completed mammograms: 22.1% to 57.1%
Recommended colonoscopies: 31.7% to 53.8%
Pneumococcal immunization: 39.1% to 50.6%
Influenza immunization: 22.7% to 41.7%
Medication reconciliation (defined as the ratio of matching practice records and patient reports before and after the HIE implementation):
35.3% (370 of 1047) to 44.9% (468 of 1043)
Barriers included: delays and difficulties in collaborating with commercial technology vendors who gave innovation a low priority
Facilitators included: strategic planning, shared goals, and establishing communication methods
Morris, et al., Facilitators: NA

163 Key to successful implementation is abilities to move beyond pilot to have volume and breadth of data: id early adopters who find value
2012
and get to a high number of queries, records returned.
Successful HIE projects seem to be those that have some level of control over the technology they use.
Sustainability is related to the ability of HIE organizations to innovate and react quickly to changes in markets. This requires a
combination of leadership and technology.
F-140
Nøhr, et al., Before-after Compare expectations with Denmark Hospitals and Survey, interviews 1999
139 experiences after HIE launched primary care
2001
Nykänen and Cross-sectional Factors of success and failure for Finland Regional Interviews and documents NR
150 a regional IS network of hospital information Study of HIE documents and
Karimaa, 2006
and physician offices system for processes; interviews of users in
exchange of pilot phase
clinical data
between hospital
and primary care
offices
Onyile, et al., Cross-sectional Determine the geographic New York Multiple settings Database and Audit logs Cumulative: 2009-
125 distribution of patients using the Ambulatory physician groups, 2011 (patients
2013
New York metro RHIO long-term care facilities, a entered by time of
Medicaid managed care plan, the study, 2011)
nation’s largest home health-
care provider and academic
medical centers that serve as
major referral centers with a total
of 7,503 inpatient beds, 341,065
annual inpatient discharge and
540,854 annual ED visits
Overhage, Evans, Cross-sectional Community readiness for HIE. U.S. Various Survey 2004
and Marchibroda, Web based survey for
151
2005 Connecting Communities for
Better Health
F-141
Date HIE
Nøhr, et al., Varies as this was a national effort Four types were described: 1998 to 1999 Not reported
139 in Denmark Common database
2001
EDI: copies of data are transferred between systems
Middle ware: software between application and database
Internet technology: data communicated via browser
Nykänen and Regional information system in Not well-described NR Pilot users of system
150 Finland
Karimaa, 2006
Onyile, et al., New York Clinical Information NYCLIX - Manhattan based RHIO, ambulatory groups, long March 2009 Patients who visited a NYCLIX
125 Exchange (NYCLIX) - Manhattan term care, home health care, academic health centers, facility
2013
based RHIO Medicaid managed care plan
Overhage, Evans, Various Various NA Organizations and individuals

and Marchibroda, who might be interested: 839
151 (national associations: 110,
2005
government agencies: 57,
individuals: 117, national
organizations: 354, state-
focused organizations: 201)
F-142
Nøhr, et al., Survey respondents: Expected benefits in 1998 Seven persons involved in each NR Expectation vs. Experience.
139 (n=102); Experiences in benefits in 1999 (n=57); HIE project. Also comparison to paper
2001
Expected barriers in 1998 (n=101); Experiences in systems at times.
barriers in 99 (n=99)
Nykänen and Unspecified number NA None None

150
Karimaa, 2006
Onyile, et al., 3,980,016 patients (after excluding 26,589 with invalid In RHIO master patient index Invalid zip code NA
125 zip code)
2013
Overhage, Evans, 134 NR NR NA

and Marchibroda,
151
2005
F-143
Nøhr, et al., Expected benefits and barriers. NA NA Mixed Methods
139 Experienced benefits and barriers.
2001 -Quantitative, descriptive data
-Qualitative analysis
Nykänen and Perform work tasks and how the HIE Qualitative NA Qualitative
150 changes them
Karimaa, 2006 Interviews, observations,
usability, and analysis
Onyile, et al., Visited RHIO facility (in master patient Calculated distance from Times NR Quantitative
125 index) Square
2013 Mapped the most current zip
code for each unique patient to
the appropriate U.S. county,
calculated the distance from each
zip code to Times Square,
mapped with EpiInfo v3.5.3,
spatial regressions with SatScan
v9.1.1 and RR of visit by spatial
cluster
Overhage, Evans, None NA NA Quantitative

and Marchibroda, Descriptive - provide only
151
2005 percentages
F-144
Risk of
Nøhr, et al., "What was expected, but not found, was resistance to EPR, as a result of changes in skills and power. The most obvious benefits are Moderate
139 increased data accessibility and improved decision making. The most considerable disadvantage is an enormous growth in discontent with
2001
the systems performance and the fact, that all the projects are delayed. Many different types of integration solutions are chosen, because
of a lack of a common model for integration. Generally the projects find, that EPJ yields increased security, but logistical problems arise in
having the systems running 24 hours 7 days a week"
Nykänen and Quality of design process deemed a success factor. General statement that users experienced better planning of patient care and access Moderate
150 to data, but no details given.
Karimaa, 2006
Onyile, et al., NYCLIX has representation in all 50 U.S. states, 4 U.S. territories and 57 International standards organization countries. 12.1 visits/ 100 Low
125 within 30 miles; 0.4 visits/ 100 at 100 miles; 87.7% live within 30 miles of Times Square; "inflection point" where visits are less than 1 per
2013
100 is 80 miles from Times Square; for cluster counties, RR for visit is 14.4; 77.7% of entire U.S. counties represented; more patients from
outer boroughs than from Manhattan
Overhage, Evans, -22% in beta stage, 28% in pilot, 28% operational, 22% conceptual; of 64 self-reported operational, only 9 could be verified NA
and Marchibroda, -5% no organizational structure; 28% "loose affiliation"; 29% had corporate structure; of these 23% hospitals, 16% provider organizations,
151 10% academic medical centers, 9% dedicated community HIE, 2% public health
2005
-Long lists of organizations to be involved, without actual details of roles; clinicians heavily involved in all, leading the way in 24%;
architectures 2% PHR, 20% peer to peer, 3% federated, 54% centralized database; 18% not decided; most planned centralized; broad
functionality and data inclusion proposed by participants, without specifics about implementation
-Standards proposed: 82% ICD-9, 73% CPT4, 38% LOINC, 41% SNOMED, 48% NDC
-One third had identified funding; planned funding over 60% external, 45% subscribers
F-145
Overhage, Retrospective Compare the completeness and Marion County, Marion County, Log file First quarter of 2001
Grannis, and cohort timeliness of laboratory reporting Indiana Indiana (public Indiana Network for Patient Care:
49 for public health in manual and health system)
McDonald, 2008 9 of 13 hospitals in county,
electronic systems physician practices, laboratories,
radiology centers, public health
departments
Ozkaynak and Multiple site case To describe sociotechnical Madison, 3 EDs in different Observations, interviews 2008-2010
129 studies system in terms of social Wisconsin systems in same 210 hours direct observations,
Brennan, 2013
structure determination of metropolitan area varied across shifts, in 5 rounds,
technical forms: "how social by 1 or 2 observers (industrial/
systems define technology and its systems engineers, nurses,), with
usefulness." informal conversations to enquire
and followup, plus 13 open
ended HIE interviews
F-146
Date HIE
Overhage, Indiana Network for Patient Care Indiana Network for Patient Care: 24 hospitals, physician NR County wide public health
Grannis, and (INPC) automated public health practices, laboratories, radiology centers, public health
49 reporting based on LOINC codes departments in Indiana
McDonald, 2008
Ozkaynak and NR Clinicians choose when to use HIE, which is always available NR ED clinicians
129
Brennan, 2013
F-147
Overhage, Marion county population Notifiable condition in eHIE No match of identifiers Manual public health reporting
Grannis, and system or in manual system(s) by physician offices,
49 laboratories (in and out of
McDonald, 2008
Indiana) to state and local
public health departments,
case finding
Ozkaynak and 184 patient care episodes NR NR NA

129
Brennan, 2013
F-148
Overhage, -Completeness Electronic or manual reporting system NR Quantitative
Grannis, and -Timeliness of public health laboratory Number identified in eHIE vs.
49 reporting
McDonald, 2008 number identified by manual
reporting, time to reporting
Ozkaynak and -Use of HIE NA NA Mixed methods

129 -Views of clinician-users
Brennan, 2013 -Quantitative descriptive
-Qualitative analysis
-Inductive iterative analysis,
systems engineers, nurses,
physician
F-149
Risk of
Overhage, Overwhelming positive effect: 4,635 found by eHIE, 944 by manual; for 818 identified by both, eHIE reported 7.9 days earlier on average, Low
Grannis, and across 53 conditions, eHIE found more for all but 3 conditions; 5/18 data items more often present in manual, 10/18 more often present in
49 eHIE; but false matches (4 Ebola); nondisease positives (rubella screen); repeat testing known positives; delayed report till confirmed or
McDonald, 2008
typed (Shigella)
Ozkaynak and -184 patient care episodes (10 use the HIE system, about 5%) Moderate
129 -2 unexpected uses of the HIE: (1) The HIE was being used mostly for patients only with specific characteristics. (2) The information from
Brennan, 2013
the HIE could be used to confront with the patients.
-System used mainly for patients with chronic pain to check previous visits (and prescribing); workflow issues interfered; extra time and
effort expended when needed,
-When the observers asked the reason of use of the system, the reason mentioned by the majority of the interviewed clinicians was to
detect drug-seeking behavior
F-150
Pagliari, Gilmour, Multiple case To explore the processes and Scotland Primary and Survey responses from users November 2001 - May
and Sullivan, studies outcomes of implementation, Secondary Care and project managers, 2003;
122 barriers and facilitators to system (August 2002-May
2004 interviews, and document
adoption and benefits and review 2003 for minimum
drawbacks for professional users. dataset)
F-151
Date HIE
Pagliari, Gilmour, Electronic Clinical Communication The ECCI is a program initiated as part of the Scottish National 2000 16 Scottish Health Board
and Sullivan, Implementation Program (ECCI) health Service Information Management and Technology areas included in minimum
122 strategy. It targets six electronic deliverables relating to direct dataset;
2004
hospital outpatient appointment booking from primary care, Survey - in-depth studies of 7
referral from primary to secondary care, results reporting from regional sites, chosen to
secondary care laboratories to primary care, transfer of hospital represent the others in terms
discharge and clinic letters to primary care and clinical email. of geographic and
demographic spread and initial
IM & T maturity.
F-152
Pagliari, Gilmour, 16 Scottish Health Board areas included in minimum Minimum dataset: all 16 areas; see inclusion criteria None
and Sullivan, dataset; surveys - limited to 7 regions
122 Survey - in-depth studies of 7 regional sites, chosen to
2004
represent the others in terms of geographic and
demographic spread and initial IM & T maturity;
64% survey response rate for primary care; 34% for
specialty care.
Survey sample represents 17% of Scottish practices;
therefore respondents represent 11%.
F-153
Pagliari, Gilmour, 6 electronic deliverables: NA NA Qualitative
and Sullivan, 1) direct hospital outpatient appointment Minimum dataset: descriptive
122 booking from primary care; statistics
2004
2) referral from primary to secondary care; Surveys: mailed or email
3) results reporting from secondary care
labs to primary care;
4) transfer of hospital discharge and clinic
letters to primary care;
clinical email (second opinion
correspondence)
F-154
Risk of
Pagliari, Gilmour, From the minimum dataset: Moderate
and Sullivan, GP practices with access to e-results reporting software: 37%
122 GP practices using e-RR: 36%;
2004
GP practices with access to e-OP appointment booking system: 3%;
GP practices using e-OP system: 2%;
GP practices with access to e-referral system: 47%;
Referral letters e-transmitted: 18%;
GP practices using clinical email: 9%;
Consultant led departments using clinical email: 5%;
Hospital wards able to send e-discharges: 10%;
Wards generating and sending e-discharges: 7%;
Specialties able to generate e-clinic letters: 11%;
Specialties generating and sending e-clinic letters: 3%.
Surveys - of responding practices:
93% used e-Lab results;
58% e-referrals;
42% e-discharges;
16% e-OP booking;
Percent reporting daily or weekly use:
90% e-results; 96% e-discharges; 92% e-referrals; 28% e-OP booking.
Clinicians most common users of e-reporting/e-referrals; admin/clerical staff most common users of e-discharge/e-OP booking.
Implementation was facilitated by successful engagement of stakeholders that focused on proactive methods. Other facilitators were ease
of use, good training, communication and commitment from staff. Barriers included differences in IT and system bugs or problems and
slow system development.
F-155
Park, et al., Cross-sectional To assess patients’ perception of South Korea Tertiary care and Survey 2008-2009
63 an HIE which includes patients’ affiliated clinics
2013 interview pre-, telephone post-
preferences regarding information
exchange operations,
endorsement of the technology,
and expected and perceived
benefits and concerns about the
technology, and to examine the
influence of demographic
characteristics and HIE
experience on patients’
perceptions.
Patel, et al., Cross-sectional To provide national estimates of U.S. Out patient Survey 2011
91 physician capability to
2013 -2011 National Ambulatory
electronically share clinical Medical Care Survey
information with other providers -Electronic medical record
and to describe variation in supplement
exchange capability across
states and EHR vendor.
Phillips, et al., Multiple case Study 3 RHIOs implementing a New York Any, but this study Interviews and documents NR
164 studies public health use case focused on public Semi-structured interviews and
2014
health reporting review of documentation of RHIO
and querying
F-156
Date HIE
Park, et al., Korean HIE pilot Federated architecture, stores and transfers HL7 CDAs June 2008 All patients visiting tertiary
63 CDA exchanges between referring providers and SUNBH hospital and affiliated clinics
2013
Patel, et al., Several Varies Varies Nonfederal office–based

91 physicians
2013
who provide direct patient care
Phillips, et al., 3 RHIOs in New York state All types Varying Interviews with leaders of the
164 3 HIEs
2014
F-157
Park, et al., Pre: 322 hospital + 408 clinic; Post: 306 of 536 HIE Not explicitly stated (visited Not explicitly stated 1) paper based, offline (USB
63 participants, 180 offline information exchange, 208 hospital or clinic) stick) and online (HIE); 2)
2013
referral letter only participants and non
participants,3) before and after
implementation
Patel, et al., 4,326 respondents (61% weighted response rate) Out patient MDs Federal physicians NA
91
2013
Phillips, et al., NA NA None None

164
2014
F-158
Park, et al., -Need for HIE HIE exposure status (pre, post, offline, Demographics Quantitative
63 -Experience with HIE letter)
2013 Descriptive, MANOVA
-Preferences
-Endorsement
-Perceived benefits and concerns
-Satisfaction
Patel, et al., Reported capacity for exchange of -State NA Quantitative

91 pharmacy, lab and clinical summary -Physician demographics
2013 '-t-tests
information -Physician use of EHR -Profit regression models
-Practice characteristics
-EHR vendor
Phillips, et al., Certification and becoming operational for Qualitative NA Qualitative

164 public health use case
2014
F-159
Risk of
Park, et al., -Group A (offline 'HIE') older, more likely to have operation, inpatient care; 14% used USB, etc., 10% paper HIE; only 23% concerned MD Low
63 do not know about prior care; all preferred consent based HIE, 80% in HIE, 55-59 in non-HIE;
2013
-Post: satisfied, would recommend: 92% of HIE, 88% of non HIE; HIE and offline 'HIE' equally cited convenience, expedited care; all
endorsed HIE, HIE group most strongly; all cited convenience, expedited care, HIE group most strongly; HIE group less concerned about
privacy, complexity, inconvenience
- A higher percentage of HIE patients (80%) compared with A(55%) & B(59%) reported their preferred method of information exchange
was HIE
-In general those who experienced HIE had statistically higher rates of agreement with survey questions regarding need for HIE
Patel, et al., Overall: 31% could share clinical summaries, of these 76% could both send and receive, 64% of these exchanges were through an EHR Low
91 vendor and 28% through a hospital-based system. 55% could e- prescribe, 67% could view lab results, 42% could incorporate lab results
2013
into EHR.
State differences: the capacity to electronically exchange clinical summaries with patients varied from 55% (Minnesota) to 18%
(Louisiana). The proportion of physicians who exchange clinical summaries with other providers varied from 61% (Wisconsin) to 15%
(Alabama).
-Adoption of EHR is strongest practice characteristic associated with exchange capacity, p<.001
-EHR vendors have a wide range of capacities for exchange: 24% to 77% of MDs report exchange capacity by vendor
-Primary care providers were more likely to exchange vs. specialists, age of MD was NS
Phillips, et al., 2 common factors influenced risk management and implementation success: leadership capable of agile decision-making and Low
164 commitment to a strong organizational vision
2014
F-160
Pirnejad, Bal, Cross-sectional How are data integration and Almere, the Community - Interviews, observations, 2005-2006
and Berg, data integrity attained in a Netherlands hospital interface documents
152 communication network?
2008 Interviews (pharmacist focus);
documents, observations of
pharmacist work after
implementation
165 Multiple Case To review the lessons learned in United Kingdom National Health In depth interviews used to Post 1996, but not
Poulidi, 1999
Study the context of HIE related to Care system wide create a stakeholder analysis; reported
collaboration among stakeholders comparison to an analysis
complete in the U.S.
F-161
Date HIE
Pirnejad, Bal, Trans-mural exchange of Medication information exchange community GP/pharmacist 2005 Hospitalized people in Almere,
and Berg, medication data in Almere (TUMA) with hospital pharmacy; same vendor, different systems, Netherlands
152 shared server
2008
165 NHSnet Wide area networking was set up to facilitate the exchange of 1993 UK, sub areas not specified
Poulidi, 1999
administrative, purchasing and clinical data.
F-162
Pirnejad, Bal, 0 of 115 GPs, 2 of 17 community pharmacists, 4 None given None given Pre-post
and Berg, hospital pharmacists in 1 hospital pharmacy; project
152 lead and 2 managers
2008
165 NR NR NR Greater Dayton Area

Poulidi, 1999
Community Patient health
Information Network in the U.S.
F-163
Pirnejad, Bal, Second stage: changes in work, First stage: study context, medication NA Qualitative
and Berg, improvement, problems; after network data communication, information gaps -Grounded theory
152 tested, reasons for problems in test results
2008 -Semi quantitative, formative
165 Stakeholder perceptions and attitudes NA NA Qualitative

Poulidi, 1999
F-164
Risk of
Pirnejad, Bal, -Pitfalls and information gaps in the old medication data communication: missing medication information on admission, delay in Moderate
and Berg, information at discharge, dependence on patients for prescription information
152 -TUMA effect on bridging the information gaps and improving the communication, focusing on the test results and their analysis.
2008
-Important unforeseen problems: (a) technical challenges in system interface (though same vendor); (b) data integrity problems (59 errors
in 32/100 records before fix, 55 items in 14/100 records after fix); (c) problems with coding system and its application, with software and its
application, (d) and conflicts related to the articulation work and responsibility distribution between the involved parties - e.g. coding
differences by GPs and pharmacists
-Aim was to replace patient as weakest link - learned that instead "contribution of patients in saving the integrity of data and in integrating
medication data is valuable"
165 Confidentiality was a major concern for physicians and a barrier that slowed implementation. NA
Poulidi, 1999
The NHS case is more complex than the regional US case in that more types of stakeholders are involved, more settings are involved in
the NHS implementation and the scope of the data exchanged is greater.
F-165
Ross, et al., Multiple case Elucidate perspectives of clinical Colorado SNOCAP-USA Interviews November 2008-April
167 studies and administrative leaders in Practice-based 2009
2010 -Topic guide created based on
smaller ambulatory practices Research literature
regarding desired HIE functions, Networks; -Telephone and on-site guided
key motivators, barriers to and small to medium- discussions
potential incentives for adoption. sized practices
(<20 providers) in
primary care
practices
Ross, et al., Retrospective Does HIE affect laboratory and Mesa County, Physician offices - Log file April 2005-December
50 cohort radiology test ordering Colorado outpatient 2010
2013 Claims data
Rudin, et al., Cross-sectional What are providers' decision- Massachusetts Physician offices Interviews Summer-Fall 2007
153 making processes in
2009 Semi-structured interviews
implementing HIE?
F-166
Date HIE
Ross, et al., 1) Community-wide HIE - currently 2 types of community-HIE: 1) traditional RHIO that provides NR Family practice sites
167 exchanged information, but could limited EMR functionality that includes storage and retrieval of participating in SNOCAP-USA
2010
use paper or electronic medical tests, dictations, meds, allergies, e-prescribing (2 urban (1 practice based research
records; indigent clinic; 1 private clinic), 1 rural site (private clinic); 22 network
2) Paper charts only - No use of providers total).
community-wide HIE; 2) nontraditional HIE-one EMR across multiples sites in an
3) EMR only - No use of community- independent practice association (still met investigators
wide HIE. definition of HIE); (1 suburban site; private; 16 providers).
Patterns included: 1) bulk of info exchanged was related to
ordering tests and studies and receiving results from hospitals
and independent labs; 2) vital to exchange info with hospitals
and specialty practices (consultation reports and discharge
summaries).
Ross, et al., Quality Health Network Query-based and directed 2005 Claims for 34,818 patients
50 served by 306 providers in 69
2013
practices who had access to
the HIE
Rudin, et al., Massachusetts eHealth Hybrid HIE NR Members of MAeHC

153 Collaborative (MAeHC) collaborative and physician
2009
users
F-167
Ross, et al., Purposeful sampling Family practice sites None listed Paper chart only practices and
167 participating in SNOCAP-USA EMR only practices vs.
2010
practice based research network community HIE practices
Ross, et al., Claims for 34,818 patients All having access to HIE None Rates of laboratory and
50 radiology testing for primary
2013
care and specialist care
physicians
Rudin, et al., 14 key informants All interviewed NA Technical HIE architecture

153 chosen
2009
F-168
Ross, et al., -Desired HIE functions Practice group None listed Qualitative
167 -Key motivators
2010 Qualitative analysis was iterative,
-Barriers to and potential incentives for allowing for investigator
adoption corroboration, triangulation, and
checking; then coding and
theming, creation of briefing
sheet, then use of modified
Delphi method to finalize
analysis. Sites also reviewed and
corrected reports prior to final
report creation.
Ross, et al., -Rates of laboratory and radiology testing Rates of laboratory and radiology None Quantitative
50 -Economic testing
2013 Mixed effects regression model
Rudin, et al., Technical HIE architecture chosen NA None Qualitative

153
2009
F-169
Risk of
Ross, et al., Desired functions of HIE: Universally valued was improved ability to receive and review clinical info from outside the practice; this much Moderate
167 more so than improved ability to send or make available info from inside the practice. Paper- and EMR-only anticipated little value in
2010
sharing their data with others, but HIE practices realized the value of having their data available anytime/from anywhere. There was
consensus that community hospitals and independent lab info would be essential. Also highly desirable to include exchange with
specialists. Test results considered most important; followed by discharge summaries.
Mean ranking of potential HIE functions (1=highest; 5=lowest rank): looking up info 1.9; delivering results 2.2; e-prescribing 2.5 (lack of
computers in exam rooms was a barrier for this one); placing nonprescription orders 3.8; creating reports 4.7; secure email was a lower
priority.
Essential attributes of HIE: solid reliability and responsive service; live and direct technical support; comprehensive policies and systems
for privacy, security and data use
Motivations for adopting HIE: motivated to gain uniformity in workflow; improved efficiency (even though did not anticipate monetary
benefit; improved quality of care through better coordination and information;
Barriers and facilitators:
1) Barrier: technical-need to interface with existing systems
2) Barrier: workflow issues-most sites did not want to re-engineer workflow
3) Best facilitator: technical assistance for implementation & maintenance; and training
4) Barrier: financial issues; secondary, but important; capital costs were barrier; not concerned with loss of revenue
5) Facilitators: solidarity & trust were important (easier in smaller cities); wanted involvement by practice leaders, NOT health plans;
neutral about government, foundations
6) Practices thought they could education patients to have trust
Ross, et al., For PCPs, rate of laboratory testing increased over the time span (baseline 1041 tests/1000 patients/quarter, increasing by 13.9 each Low
50 quarter) and shifted downward with HIE adoption (downward shift of 83, p<0.01). For specialist providers (baseline 718 tests/1000
2013
patients/quarter, increasing by 19.1 each quarter, with HIE adoption associated with a downward shift of 119, p<0.01). Imputed charges for
laboratory tests did not shift downward significantly in either provider group. For radiology testing, HIE adoption was not associated with
significant changes in rates or imputed charges in either provider group.
Rudin, et al., To become established, HIE efforts must foster trust, appeal to strategic interests of the medical community as a whole, and meet Moderate
153 stakeholder expectations of benefits from quality measurements and population health interventions.
2009
F-170
Rudin, et al., Cross-sectional What affects clinician use of HIE Massachusetts Hospitals and Interviews of clinician users October 2009-
136 physician offices February 2010
2011 and HIE staff
Saff, et al., Cross-sectional Description of motivation, San Francisco 5 health Database Each medical center
154 implementation and use of San Bay Area organizations; joined the HIE at a
2010 Varying types of clinical and
Francisco Bay Area HIE 2,800 MDs; administrative data - varies by different time, dating
900,000 patients; site from 2002
numerous labs;
several IT vendors
Schabetsberger, Prospective Describe evolution and use of Tyrol, Austria Tiroler Audit logs June 2003 and
172 cohort system, problems. Landeskrankenan October 2004
et al., 2006
stalten, 6 hospital,
6,000 staff, 1,000
physician, 300,000
outpatient, 70,000
inpatient, 400
medical student
health system
F-171
Date HIE
Rudin, et al., Massachusetts eHealth All nontext portions of medical record. Could link directly from Mid-2007 Clinician users and staff who
136 Collaborative (MAeHC) the EHR to existing HIE. Query-based exchange. Consent was implemented HIE
2011
'opt-in'.
Saff, et al., NR Each medical center valued the HIE for different reasons; NR 900,000 patients in the San
154 descriptions are provided Francisco and the East Bay
2010
Schabetsberger, Various (1) Discharge summaries push to GP EHRs as text documents, May 2002-October Tyrol, Austria physicians
172 92+% electronically 2004
et al., 2006
(2) Standalone web-based archive of hospital documents for
nonaffiliated physician access
F-172
Rudin, et al., 15 clinicians and 2 HIE staff and 3 administrators NA None None
136
2011
Saff, et al., 900,000 patients in San Francisco and the East Bay None specifically stated; all None specifically stated; None
154 patients included all patients included
2010
Schabetsberger, NR NR NR NA
172
et al., 2006
F-173
Rudin, et al., Motivators and moderators of use Qualitative NA Qualitative
136
2011 Content analysis
Saff, et al., Lessons learned Characteristics of each health system; NA Quantitative

154 this is a descriptive case study
2010 Descriptive
Schabetsberger, System use NA NA Quantitative

172
et al., 2006 Descriptive
F-174
Risk of
Rudin, et al., -Motivators were belief in improved quality of care, time savings, and reduced need to answer questions. Cost of care was not listed as a Low
136 motivator.
2011
-Motivation was moderated by missing data, workflow issues, and usability issues (too many clicks required to get to information).
-Missing data was attributed contributing providers not "locking their notes" on their EHR.
-Patient-related moderators were those who had trouble communicating, multiple comorbid illnesses, and who received care at multiple
sites within but not outside HIE.
-Clinician-related moderators varied by specialty, use of paper and fax, and integration into workflow.
-HIE-related moderators were gaps in data from local nonparticipants, poor usability, and downtimes.
-Clinicians varied in how quickly they "locked" data for transfer into HIE.
Saff, et al., Lessons learned High

154
2010 -Moved from a competitive to collaborative model
-EMR/PHR integration
-Extensive testing required to ensure quality of data fit for use
-Physician education and engagement required/important
Schabetsberger, -6% to 8% of approximately 40,200 discharge letters were sent out electronically NA
172 -Problems: corrupt data in physician database; differing implementations of standards (EDIFACT standard); independent, nonfederated
et al., 2006
patient index; 4 GPs and the psych ward had security concerns
F-175
Schoen, et al., Cross-sectional To explore the experiences of Australia, Primary Care Survey responses March - July 2012
95 physicians in primary care with Canada, France, Practices
2012
health reform policies. Germany the
Netherlands, New
Zealand, Norway,
Switzerland, The
United Kingdom
and the U.S.
Shapiro, et al., Retrospective Measure incremental increase in New York City 10 hospitals that Log file June 1, 2010-May 31,
51 cohort number of frequent ED users participated in 2011
2013 NYCLIX data (which also
identified when data from all EDs NYCLIX included data from site-specific
(using HIE) were compared with EMRs)
use of site-specific data only
F-176
Date HIE
Schoen, et al., NR Electronic exchange of patient summaries and test results with NR General practice and family
95 doctors outside their practice. practice physicians in all
2012
countries, as well as general
internists and pediatricians in
Germany and the U.S.
Shapiro, et al., 10 hospitals that participated in New York Clinical Information Exchange (NYCLIX) NR All patients with ≥1 instance of
2013
51 New York Clinical Information ≥4 ED visits within 30 days
Exchange (NYCLIX); NYCLIX is a during study period
RHIO in NY City; data sent to
NYCLIX by each participant
organizations; master patient index
links each patient across sites;
NYCLIX staff was 'honest broker'
and provided data.
F-177
Schoen, et al., Primary Care Physicians Surveyed Practicing physicians were NR NR
95 randomly selected from public
2012 Australia: 500
Canada: 2,124 and private lists typically used in
each country
France: 501
Germany: 909
The Netherlands: 522
New Zealand: 500
Norway: 869
Switzerland: 1,025
United Kingdom: 500
U.S.: 1,012
Overall: 8,462
Shapiro, et al., 924,675 ED visits by 591,632; All patients with ≥1 instance of 4,168 visits because they EMR use without accessing
2013
51 920,507 ED visits by 591,632 patients ≥4 ED visits within 30 days occurred within 6 hours of HIE
during study period a previous ED visit, which
investigators decided a
priori might represent
clerical errors
F-178
Schoen, et al., Ability to electronically exchange patient NA NA Quantitative
95 summaries and test results with doctors 2
2012 Survey, Chi tests
outside their practice
Shapiro, et al., -Number ED visits -Gender Cross-over visits (different EDs) Quantitative
51 -Number of patients experiencing these -Age 2
2013 -Chi
visits
-Wilcoxon sign rank test
-Average number ED visits per patient
during 12 months
-Number patients frequent ED users (per
definition)
-Number of ED visits accounted for by
frequent users
-Average number visits per frequent user
-Increase in number of frequent users
when estimated across HIE (vs. within
each site)
F-179
Risk of
Schoen, et al., % of primary care physicians reporting HIE capabilities: High
95
2012 Australia: 27
Canada: 14
France: 39
Germany: 22
The Netherlands: 49
New Zealand: 55
Norway: 45
Switzerland: 49
United Kingdom: 38
U.S.: 31
In the U.S. capacity for electronic exchange of patient information was concentrated in larger practices and those in integrated health
systems (50% of physicians reported HIE vs. 23% of physicians not part of integrated practices p<0.05)
Shapiro, et al., Total visits: 924,675 (591,632 unique patients) Moderate

51 After exclusion: 920,507 visits by 591,632 patients
2013
Mean ED visits/year: 1.6
When used only site-specific data only: 4,786 patients met criteria of frequent user (represented 0.8% of all users)
Number of ED visits: 45,771
Mean visits/years: 9.6 (accounted for 5% of ED visits)
HIE-wide results
5,756 frequent ED users
20% increase in number of frequent user events identified
53,031 visits (6% of all ED visits)
Thus HIE data produced 16% increase in number ED visits that could be identified
Frequent users more likely to be male: 51% vs. 45%, p<0.0001
Mean age higher: 40.7 vs. 37.9 years, p<0.0001
More had cross-over visits: 28.8% vs. 3%, p<0.0001
F-180
Sicotte and Paré, Multiple case Describe the implementation and Quebec, Canada Case 1: 3 pediatric Interviews, observations, January 2001 + 42
168 studies deployment of 2 large HIE hospitals. months (Case 1); May
2010 documents
projects. Case 2: Primary 52 interviews (27 for Case 1, 25 2001 + 32 months
care network for Case 2); all documents from (Case 2)
linking a public the HIE project team, HIE
hospital to 10 organizations and vendors; and
private clinics. observations at HIE project
meetings
Silvester and Carr, Before and after Description of implementation - Brisbane & 239 GPs from 66 Database April 30, 2007-July
114
2009 use of system. Northern practices, Registration, communication, and 2008
Territories of 2 major public clinical database.
Australia hospitals, 3 large Clinical database contains
private hospitals, socioeconomic status,
11 allied health/ medications, diagnosis, allergies,
community based medical history, diagnostic
partners results, care team members,
unstructured documents
Soderberg, Time Series To monitor progress toward Minnesota Clinics Survey February 15-March
Laventure, and meeting the legislative 72 survey questions 15, 2013
90 requirement that all health care
Minnesota, 2013
providers have an interoperable
EHR by 1/2015.
F-181
Date HIE
Sicotte and Paré, Case 1: 3 pediatric hospitals. Case 1: large pediatric hospital, 2 community pediatric hospital, Specific date unclear Key informants description
168 Case 2: Primary care network 4 pediatric clinics. limited to HIE project staff and
2010
linking a public hospital to 10 Case 2: public hospital, over 100 physicians at 10 private HIE users
private clinics. clinics.
Access to laboratory and imaging results.
Silvester and Carr, Name NR Software developed by HealthConnect; web services, HL-7 Prior to April 30, Registered patients with
114 239 GPs from 66 practices, messaging, extracts data from clinician's software package, 2008; implemented chronic conditions, cared for at
2009
2 major public hospitals, interfaces seamlessly with clinician's software, uses Medicare iteratively to ensure these sites
3 large private hospitals, Australia's public key infrastructure security certificates for success
11 allied health/community based authentication; patients 'opt-in'.
partners
Soderberg, Varies Varies Varies 1,623 ambulatory

Laventure, and clinics
90
Minnesota, 2013
F-182
Sicotte and Paré, 52 interviews (27 for Case 1, 25 for Case 2) NR NR NA
168
2010
Silvester and Carr, 1,108 patients in population None, other than stated in None, other than stated in Before implementation
114 population and sample population and sample
2009
Soderberg, The response rate was 79%, with 1,286 clinics Any location where primary or NR None
Laventure, and responding specialty care ambulatory
Minnesota, 2013
90 services are provided for a fee
by ≥1 physician
F-183
Sicotte and Paré, Descriptive narrative only NA NA Qualitative
168
2010 Empirical observations were
organized into narrative using a
risk analysis framework
Silvester and Carr, -Frequency of use (number of events None None Mixed methods
114 uploaded per patient)
2009 -Descriptive summaries
-User access logs and patient registration -Qualitative analysis
growth rates and connection metrics
-User surveys
-Patient case studies
Soderberg, Exchanges with affiliated and unaffiliated NA NA Quantitative

Laventure, and hospitals Descriptive statistics
90
Minnesota, 2013
F-184
Risk of
Sicotte and Paré, Case 1: 4 stages described: project planning with small part-time team; technical system with risks evolving; testing requiring de-scoping; Low
168
2010 piloting with user and technical challenges. Overall deliverable not reached, users discouraged and usage was low.
Case 2: 4 stages described: project planning with full-time staff, system integrator consultant and clinical champions; solicitation of user
views and realistic understanding of context, participant contracts signed; system customization and testing, leveraging super-users;
piloting, troubleshooting system performance issues. Overall view was successful with high usage.
Silvester and Carr, -Mean events uploaded for each patient record during 12 months: 9.7 High
114 -Increased HIE use by nurses
2009
-Number of patients registered increased: 474 (July 2007) to 1,320 (June 2008)
-Increased commitment to use
-Case studies demonstrated use prevented unplanned inpatient admissions
-Interest to adopt by others
Improved staff perceptions in answers to 3 pre-post questions on 5-point Likert scale
Improved understanding of system: 2 to 3
Improved sharing of information: 2 to 2.3
Impact on care delivery: 3 to 3.6
-2 patient-specific case studies showed improved use, communication, satisfaction
-Lessons learned included connectivity, interoperability, change management, clinical leadership, targeted patient involvement,
information at point-of-care, and governance
Soderberg, -54% exchange data with affiliated hospitals Moderate

Laventure, and -36% with unaffiliated hospitals
90
Minnesota, 2013 -Common challenges for HIE: limited capacity of others to exchange, lack of technical support or expertise, competing priorities, cost and
privacy concerns
F-185
Steward, et al., Multiple case Understand the dynamic New York, New Hospital specialty Interviews NR explicitly but at 2
169 studies capabilities that enabled the 6 Jersey, clinics, support points in time: as the
2012 Laboratory, diagnostic, medical,
demonstration projects of the California, services, primary and service utilization; referrals; HIE were being
Information Technology Networks Louisiana, New care clinics, and ancillary care support, such developed and 1-2
of Care Initiative to implement York testing sites, ED, as case management, counseling years after the HIE
HIE. outpatient and and testing, transportation, and became operational.
inpatient clinics, substance use and mental health
Office of Public services.
Health, insurers,
laboratory and
pharmacy services
Swain, et al., ONC Data Brief Summarize trends in HIE use in NA NA Data are from the American 2014 update
26 non-federal acute care hospitals Hospital Association (AHA)
2015
from 2008-2014 Information Technology (IT)
Supplement to the AHA Annual
Survey. Since 2008, ONC has
partnered with the AHA to
measure the adoption and use of
health IT in U.S. hospitals.
F-186
Date HIE
Steward, et al., 6 HIEs that were part of the Each of 6 projects implemented a different HIE. NR 111 project staff and IT
169 Information Technology Networks specialists; staff from
2012
of Care Initiative that included community-based
Bronx-Lebanon Hospital Center, organizations and public
Duke university; hospitals, the city health organizations; users of
of Paterson, Louisiana State HIE.
University Health Care Services
Division, NY Presbyterian Hospital,
St. Mary Medical Center
Foundation
Swain, et al., Varies, as these data are from the Varies, as these data are from the AHA survey NA The survey was administered
26 AHA survey to 4,451 non-federal acute
2015
care hospitals, with a
response rate of 60%.
F-187
Steward, et al., NR NR NR Cross-site evaluation
169
2012
Swain, et al., The survey was administered to 4,451 non-federal The survey was administered to Federal and non-acute prior years
26 acute care hospitals, with a response rate of 60%. 4,451 non-federal acute care care hospitals
2015
hospitals, with a response rate of
60%.
F-188
Steward, et al., Implementation outcomes NA NA Qualitative
169
2012 -Developed 16 coding topics
-Convergent and divergent
perspectives examined within and
across sites
Swain, et al., HIE use between hospitals and hospitals; NA A logistic regression model was Estimates considered unreliable
26 HIE use between hospitals and outside used to predict the propensity of had a relative standard error
2015
providers; survey response as a function of adjusted for finite populations
Types of data exchanged (Labs, radiology, hospital characteristics, including greater than 0.49. Responses
meds, clinical care summaries) size, ownership, teaching status, with missing values were
system membership, availability of assigned zero values. Significant
a cardiac intensive care unit, urban differences were tested using p <
status, and region. Hospital-level 0.05 as the threshold.
weights were derived by the
inverse of the predicted propensity.
F-189
Risk of
Steward, et al., Found evidence for importance of 3 dynamic capabilities: information systems, reconfiguration capacity, and organization size and human Moderate
169 resources. Reconfiguration capacity was most important.
2012
Swain, et al., Hospitals’ electronic health information exchange with hospitals or ambulatory care providers outside their organization increased by 85% NA
26 from 2008 to 2014, and increased by 23% since last year (2013).
2015
In 2014, 47 states and the District of Columbia had at least 60% or more of their hospitals electronically exchange key clinical data with
outside providers. In contrast, in 2010, 10 states had 60% or more of their hospitals electronically exchange key clinical data with outside
providers.
In 2014, state rates of hospitals’ electronic exchange of key clinical data with outside providers ranged from 42% to 100%; whereas in
2010, hospitals’ health information exchange with outside providers ranged from 24% to 67%
Approximately two-thirds of hospitals electronically exchanged laboratory results (69%), radiology reports (65%) and clinical care
summaries (64%) with outside providers in 2014.
Close to six in ten (58%) hospitals exchanged medication history with outside providers. This is an increase of 176% since 2008 and an
increase of 57% since 2013.
Summary:
More than three-quarters (76%) of non-federal acute care hospitals electronically exchanged laboratory results, radiology reports, clinical
care summaries, and/or medication lists with any outside providers. This represents an 85% increase since 2008 and a 23% increase
since last year. Close to seven in ten hospitals (69%) electronically exchanged health information with ambulatory providers outside of
their organization, representing a 92% increase since 2008 and a 21% increase since 2013.
F-190
Thorn, Carter, and Multiple site case To explore views of emergency NR ED in 4 hospitals, Interviews NR
130 studies physicians having access to HIE, private and public
Bailey, 2014 Individual unstructured
about their access of and use of settings interviews, audio recorded and
HIE data transcribed
Tripathi, et al., Multiple case Description of initiative, Massachusetts 3 communities Focus groups, documents Began in 2005
123 studies collaborative design and lessons chosen to pilot Community steering committees, Duration not clear
2009
learned; HIE, Brockton MAeHC, stakeholders; consumer
also includes opt in data by (diverse focus groups
consumer community),
Newburyport
(affluent), North
Adams (rural)
Tzeel, Lawnicki, Retrospective Assess the association of HIE S.E. Wisconsin EDs in 5 health Log file December 2008-
and Pemble, cohort use on health care costs (Milwaukee systems in a WHIE data - health plan member March 2010
52 County) county
2011 with ED encounter when HIE
access occurred.
Humana claims data - costs and
utilization of ED encounter.
F-191
Date HIE
Thorn, Carter, and HIE name NR but may be MSeHA Data in HIE NR NR ED physicians in 3 urban
130 Regional HIE operational for 4 Decentralized, query-based exchange. Consent was 'opt-out' settings
Bailey, 2014
years, linking over 450 providers in
15 clinics and 9 major hospitals
serving a population of 1 million
Tripathi, et al., Massachusetts eHealth NR NR Number of participants in

123 Collaborative (MAeHC) committees and stakeholders
2009
involved not stated
Tzeel, Lawnicki, Wisconsin Health Information Links 5 health systems in the county. Access to patient December 2008 Commercial, fully insured
and Pemble, Exchange (WHIE) demographics, chief complaint, allergy, primary care provider, members of Humana health
52 diagnosis, meds, procedures, encounter date & location. plan (denominator); members
2011
in the WHIE database having
≥2 ED visits
F-192
Thorn, Carter, and N=15 physicians from 4 urban hospital systems having Full or part-time physicians NR NA
130 <10% usage of HIE. Cross section of public and private working regularly scheduled ED
Bailey, 2014
hospitals. 1 Level I Trauma center. 2 of 4 settings had shifts. Purposeful selection of 2
not implemented EHRs because of a 4-year history of
HIE use. Rest recruited with
"theoretical sampling"
Tripathi, et al., NA NA NA NA
123
2009
Tzeel, Lawnicki, Test group: 428 members with ED visits having an HIE ≥1 year continuous insurance <6 months coverage before Pairs matched for age, gender,
and Pemble, query coverage with health plan program started or <3 and costs for net care per
52 Control group: 1,054 members with ED visits with no months after start of participant per month
2011
HIE query. program prescriptions, inpatient,
Propensity score matching for test group (N=326) with outpatient, ED, and physician.
HIE database query in all ED visits vs. control group
(N=325) with HIE database not queried in any ED visit.
F-193
Thorn, Carter, and Descriptive narrative only NA NA Qualitative
130
Bailey, 2014 Thematic, constant comparative
analysis of narrative
Tripathi, et al., -Descriptive narrative only NA NA Qualitative

123 -Type of patient consent
2009
-Type of data to share
Tzeel, Lawnicki, -Comparison of net costs and ED costs per Pairs matched for age, gender, and NR Quantitative
and Pemble, participant costs for net care per participant per Matched pairs t-tests
52 -Comparison of top 5 ED procedures in test month prescriptions, inpatient,
2011
group vs. matched control 1 year before outpatient, ED, and physician
and 1 year after the first ED visit
F-194
Risk of
Thorn, Carter, and Themes Low
130
Bailey, 2014 -Users varied in their HIE use. Stated influencers including trouble accessing system, acuity of patient or history not available, team
members' ability to access.
-HIE use affected decisions sometimes, for specific cases (e.g. drug seekers); often HIE use did not affect decisions
-Use was negatively affected by access challenges, separate login, variability in data being pertinent, absence of data types or data on
specific patients, user design flaws, and lack of technical support.
-Benefits with usage included reducing redundant testing, more accurate history, reducing faxing, knowledge of primary care provider
name
-Barriers to usage included continued practice of defensive medicine, desire for autonomy, changing the culture, belief HIE does not alter
decisions, health system competition, and reduced revenue, workflow disruption.
Tripathi, et al., Discussion of experience/lessons learned NA

123
2009 -Decision on consent: opt in chosen due to state law stricter than federal HIPAA law; use of a centralized data repository; and consumer
feedback.
-Data shared: 3 communities agreed on what to share - all EHR except text notes, consult letters and scanned reports.
-Consumer focus groups identified themes to drive HIE/opt in: promote convenience and costs, promote with providers, say benefits up
front, confront risks, use professional marketing
-Consumer opt In across 2 smaller communities: 88% and 92%
Tzeel, Lawnicki, Unadjusted: ED costs in test group changed $1,068 to $999 from 1st to subsequent visit vs. control group changed $1,043 to $1,157 Low
and Pemble, Adjusted for propensity matching: Net costs (per participant per month) in test patients with higher net costs overall in and
52
2011 subcategories
ED costs: $29 less in test patients from first visit vs. subsequent visits.
Top ED procedures: 4 of 5 were reduced in test group (lab, radiology, CT, EKG)
F-195
Tzeel, Lawnicki, Retrospective Assess the association of HIE S.E. Wisconsin EDs in 5 health Log file December 2008-
and Pemble, cohort use on hospital admissions (Milwaukee systems in a WHIE data - health plan member March 2010
53 County) county
2012 with ED encounter when HIE
access occurred.
Humana claims data - costs and
utilization of ED encounter.
Unertl, et al., Multiple case To investigate how technology Memphis, 6 EDs and 8 Interviews, observations January-August 2009
170 studies and health system coevolve to Tennessee ambulatory clinics Direct observation at 14 sites,
2013
reduce information fragmentation region informal interviews at sites, 9
and improve care coordination semi structured telephone
(Extension of Unertl 2012 study) interviews
Unertl, Johnson, Multiple site case To understand the interaction Memphis, 6 EDs and 8 Observations, interviews January-August 2009
and Lorenzi, studies between HIE and workflow. Tennessee ambulatory clinics Direct observation (180 hours) at
119 How have sites integrated HIE region 14 sites, informal interviews at
2012
into existing approaches? sites, 9 semi structured
Are there common HIE workflow telephone interviews with
patterns across sites? physicians, nurses and IT
How do providers incorporate HIE management
into clinical practice?
F-196
Date HIE
Tzeel, Lawnicki, Wisconsin Health Information Links 5 health systems in the county. Access to patient December 2008 Commercial, fully insured
and Pemble, Exchange (WHIE) demographics, chief complaint, allergy, primary care provider, members of Humana health
53 diagnosis, meds, procedures, encounter date & location. plan (denominator);
2012
Members in the WHIE
database having at least 2
Emergency Dept. (numerator)
was the study population.
Unertl, et al., MidSouth eHealth Alliance HIE structure from Vanderbilt University. Data on >1 million 2004 NR
170 (MSeHA), regional HIE around patients includes test results, imaging, discharge summaries,
2013
Memphis includes majority of large diagnosis codes and claims data. Opt out model.
hospitals and 2 safety net clinic
systems.
Unertl, Johnson, MidSouth eHealth Alliance HIE structure from Vanderbilt University. Consolidated data 2004 NR
and Lorenzi, (MSeHA), regional HIE around from multiple hospital emergency departments and community-
119 Memphis includes majority of large based ambulatory clinics. Decentralized, query-based
2012
hospitals and 2 safety net clinic exchange. Data on >1 million patients includes test results,
systems. imaging, discharge summaries, diagnosis codes and claims
data. Opt out model.
F-197
Tzeel, Lawnicki, Test group: 428 members with ED visits having an HIE ≥1 year continuous insurance <6 months coverage before Pairs matched for age, gender,
and Pemble, query coverage with health plan program started or <3 and costs for net care per
53 Control group: 1,054 members with ED visits with no months after start of patient per month,
2012
HIE query program prescriptions, inpatient,
Matched pairs: 325 outpatient, ED, and physician.
Unertl, et al., NA NR NR NA
170
2013
Unertl, Johnson, NA NR NR NA
and Lorenzi,
119
2012
F-198
Tzeel, Lawnicki, -Admissions per 1,000 members, at time of Pairs matched for age, gender, and NR Quantitative
and Pemble, ED visit (1st, 2nd visit) costs for net care per patient per Chi
2
53 -Conditional probability of admission at ED month, prescriptions, inpatient,
2012
visit (1st, 2nd) outpatient, ED, and physician
-Bed days per 1,000 members
-Average length of stay
Unertl, et al., Descriptive narrative only NA NA Qualitative

170
2013 Open-ended grounded theory
analysis, followed by the
application of the Information
Ecology Framework to structure
additional analysis
Unertl, Johnson, Descriptive narrative only NA NA Qualitative

and Lorenzi, Grounded method using open
119
2012 coding, and framework-focused
axial coding.
F-199
Risk of
Tzeel, Lawnicki, Adjusted for propensity matching Low
and Pemble, Admission/1,000 members (1st to 2nd ED visit): 269 to 664 for test group vs. 321 to 555 for control group
53
2012 Probability of admission higher at 1st ED visit in control group, and higher at 2nd ED visit in test group
Test group had 771 fewer bed days/1,000 members and lower length of stay than control group
Post–propensity matching analysis showed that test group had 199 more admissions per 1000 members than control group, these
admissions might have been more appropriate. Test group admissions resulted in less time spent as inpatients and by average length of
stay (4.27 days per admission for all admissions and 0.95 days per admission when catastrophic cases removed).
Unertl, et al., -All sites had coexisting use of HIE and manual processes to access information Low
170 -Observations were used to map 5 Info Ecology Framework components to a newly developed "Regional Health Information Ecology": 1.
2013
system - HIE to reduce information silos; 2. locality - sites had distinct local context; 3. diversity - staff had varied roles with varied HIE
processes; 4. keystone species - info consumers, who used data for varied reasons; info reservoirs, people who played formal and
informal roles; exchange facilitators, who assisted others and bridged gap between consumers and reservoirs.
-Paradox observed: providers describe HIE useful, regardless of use frequency ('when we use it, it's great"); but, provider belief that HIE
not being used to full potential.
-Examples of impact were identified using their model: a. reduce fragmentation of information; b. reduce time to obtain information; c.
increase provider awareness of patient-health system interactions (e.g., drug seeking)
Unertl, Johnson, Cross organizational patterns; 2 models identified Moderate

and Lorenzi, 1. Nurse workflow: prompted by patient reporting recent hospitalization event during intake, HIE access by nurse or assistant, printed
119
2012 discharge summary, added to chart
2. Physician workflow: HIE accessed by provider (doctor or nurse practitioner) for greater reasons beyond hospitalization; HIE access
occurred at various points of care; HIE review of more information including history
-Other observations: clerks tracked biopsy results; workflow patterns evolved over time, due to factors such as access policies or staffing
changes; residents logged into other EMR due to lack of HIE access
-Reasons to access HIE: visit to another hospital; issues of patient trust; communication challenges; referrals
F-200
Vest and Case control How does HIE access vary by job Austin, Texas Indigent patients Log files from clinical data January 2006-June
Jasperson, type and organization in an and facilities that repository (Indigent Care 2009
103 indigent care HIE in central care for them Collaboration of Austin, Texas
2012
Texas? safety network providers founded
1997); 18 hospitals, public and
private clinics, government
agencies (federally qualified
health centers)
54 Retrospective Test the hypotheses that HIE Central Texas 18 members in Log file January 1, 2005-
Vest, 2009
cohort information access reduced ED HIE (I-Care): Demographic, clinical June 30, 2007
visits and inpatient hospital systems, information, diagnoses,
hospitalizations for ambulatory public and private medication orders, prior visits,
care sensitive conditions among clinics, and payer sources for uninsured
medically indigent adults. governmental patients.
agencies
operating federally
qualified health
centers
F-201
Date HIE
Vest and Integrated Care Collaboration (ICC) Clinical data repository (Indigent Care Collaboration of Austin, HIE 1997; I-Care Indigent people, not Medicare
Jasperson, Texas safety network providers founded 1997); 18 hospitals, database 2002, 3.1
103 clinics, government agencies (federally qualified health million encounters,
2012
centers) 600,000 individuals
54 18 members in HIE: hospital Each site contributes patient electronic data to I-Care through HIE 1997; I-Care Uninsured 18 to 64 years old
Vest, 2009
systems, public and private clinics, secure electronic interfaces. In turn, each location may access database 2002, 3.1 and excluded encounters at
and governmental agencies data from I-Care at a secured website. million encounters, the public mental health
operating federally qualified health 600 thousand provider and Planned
centers individuals Parenthood
F-202
Vest and 105,705 unique user sessions User session as all system Could not classify 35 user None
Jasperson, viewing activity (i.e., screens sessions (0.03%) and
103 accessed) by a given user for a excluded them as too few
2012
given patient on a given date. for meaningful analysis.
54 3463 HIE access, 2651 No access; 6,114 included out Uninsured 18 to 64 years old Encounters at the public Persons with no information
Vest, 2009
of 600,000 individuals, 3.1 million encounters mental health provider and accessed in the HIE vs. those
Planned Parenthood. Also with accessed information
excluded encounters
related to accidents,
pregnancy, labor and
delivery.
F-203
Vest and Administrative vs. clinical vs. repetitive vs. -User types and unique job titles Same day, within a week, within a Cross tabulation to compare
Jasperson, mixed use -Workplaces month, within a year, longer than a usage categories with A) job
103 year, or no encounter categories, B) workplace
2012
categories, and C) timing of
usage categories. Associations
evaluated between types of
usage and these variables using
2
the Pearson chi test of
independence
54 -ED visits and inpatient hospitalizations -Predictors of HIE use (e.g., -Clinical, demographic, Quantitative
Vest, 2009
due to ambulatory care sensitive demographics, number of chronic comorbidity, service measures -Frequencies and percent
hospitalizations conditions, prior ED visits or -Created a chronic condition index -Multiple logistic regression
-Logs document the user’s location, the hospitalizations) by summing chronic conditions adjusting for confounders
patient viewed, the date accessed, and -HIE for predicting ED and (diabetes, hypertension, asthma,
information screen viewed hospitalizations ischemic heart disease,
hypercholesterolemia and stroke)
F-204
Risk of
Vest and ->6/10 sessions users accessed the system in a minimal fashion Low
Jasperson, -Average pattern length: 2.89 screens
103
2012 -Shortest pattern length included only 1 screen and the longest pattern involved 83 screens
-65.7% of all user sessions had a pattern length of only 2 screens
-Use was overwhelmingly (93.9%) administrative, roughly evenly distributed across workplaces but for dominance of hospital accesses
(37.6%) and about half same day, a fifth first week, a fifth over the year, 1/10 unassociated with encounter; usage type associated with job
category: admin, nurse, pharmacy, physician, public/mental health, social services; most clinical access in ED, and public/mental health
-297 users, 113 unique job titles, collapsed into administration (59% of users), nurse (~6% of users), pharmacy (~1% of users), physician
(~12% of users), public health (~6% of users), and social services (~15% of users)
-Workplaces: ambulatory care (~9% of users), ED (~18% of users), children’s ED (3% of users), hospital (53% of users), public health
agency (8% of users), or mental health agency (8% of users).
-In more than 6 out of 10 sessions, users accessed the system in a minimal fashion.
-Average pattern length was 2.89 screens (range 1-83 screens); 66% of all user sessions had a pattern length of only two screens.
54 Adjusted OR of HIE information access Low

Vest, 2009
Increasing age: 1.03; number of chronic conditions: 1.13; ≥1 prior year clinic visit: 1.63; a prior year ED visit: 1.96; and being hospitalized
in 2004: 2.02
All levels of HIE information access were associated with increased expected ED visits and ambulatory care sensitive hospitalizations vs.
no information access
-HIE was used more for those that used the system more, or were sicker.
-HIE was not accessed for 43% of individuals
-Ultimately, these results imply that HIE information access did not transform care in the ways many would expect. Expectations in
utilization reductions, however logical, may have to be reevaluated or postponed.
-Patients with HIE information accessed one time had an 83% higher expected count of ED visits.
F-205
155 Cross-sectional Which nontechnological and U.S. U.S. Hospitals Surveys After 2009
Vest, 2010
technological factors may still 2008-2009 HIMSS Analytic
hamper the existence of effective Database; AHA Annual Survey
HIE even in light of the 2007
substantial financial incentives
offered via the HITECH Act?
Vest and Miller Retrospective Do hospitals using HIE have U.S. Hospitals Log file After 2009
64 cohort higher reported communication
2011 -2008-2009 HIMSS Analytic
among health professionals Database
and/or higher patient -AHA Annual Survey 2007
satisfaction? -Review of all HIE facilitating
efforts in U.S., linked to HCAHPS
survey
Vest, et al., Case control Do hospitalizations, ED visits, Austin, Texas Indigent patients Log files from clinical data January 2006-June
105 and other factors predict HIE use and facilities that repository (Indigent Care 2009
2011
for indigent adults? care for them Collaboration of Austin, Texas
safety network providers founded
1997); 18 hospitals, clinics,
government agencies (federally
qualified health centers)
F-206
Date HIE
155 Various Various Various U.S.
Vest, 2010
Vest and Miller Various Various Various U.S

64
2011
Vest, et al., Integrated Care Collaboration (ICC) Clinical data repository (Indigent Care Collaboration of Austin, HIE 1997; I-Care Indigent people, not Medicare
105 Texas safety network providers founded 1997); 18 hospitals, database 2002, 3.1
2011
public and private clinics, government agencies (federally million encounters,
qualified health centers) 600,000 individuals
F-207
155 4,830 hospitals in AHA and HIMSS-AD In AHA or HIMSS survey NR Operational vs. adopted not
Vest, 2010
operational vs. not adopted
Vest and Miller 3,278 hospitals, 340 adopted, 351 implemented HIE Participated in AHA or HIMSS Too few observations Adopted vs. implemented vs.
64 survey (HCAHPS survey none
2011
responses <100)
Vest, et al., 271,305 encounters (111,482 unique patients) from 10 All ED encounters among Excluded any ED None
105 facilities; (Vest 2009 was 3,463 HIE access, 2,651 no patients ages 18 to 64 that encounters occurring at
2011
access; 6,114 included out of 600,000 individuals, 3.1 occurred between January 1, facilities before the hospital
million encounters) 2006 and June 30, 2009 had an authorized user of
the I-Care system.
F-208
155 HIE adoption (operational, implementing, Technological readiness (number of -Classic markers of innovation Quantitative multivariate
Vest, 2010
nonadapter) live applications, CCHIT EMR), vertical adoption considered covariates analysis
integration, horizontal integration, -Total number of beds (size) -Begins with, or assumes, TOE
high/low information needs, inpatient -Average days cash on hand from framework: technological,
admissions, market competition, all sources organizational, and
uncompensated care burden, primary -Nonmetropolitan location environmental; missing values
care rate, health system/network size -General innovativeness was imputed from earlier versions of
measured both as academic AHA Guide and HIMSS-AD
affiliation and specialization, the -Logistic regression on adoption,
standardized total number of logistic regression on operational
professional job categories
Vest and Miller -Percentage of patients who reported their Level of HIE participation: implemented Organizational variables Quantitative
64 doctors and their nurses always (active sharing); adopted (participating associated with HCAHPS
2011 -Least squares regression
communicated well but not yet sharing); or none outcomes; other AHA -Propensity score adjustment
-Percentage of patients who would organizational characteristics,
definitely recommend the hospital overall level of automation in
-Percentage of patients who gave the hospital, external factors such as
hospital a high global rating (≥9 on a 10- state regulations
point scale)
Vest, et al., No usage vs. basic usage vs. novel usage -Familiarity Assessed with multivariate Quantitative multivariate
105 (more screens) -Complexity analysis, otherwise NR Logistic regression with
2011
-Mental/substance use adjustment for by-patient
-Frequency of prior utilization clustering
elsewhere
-Time constraints
F-209
Risk of
155 -59 operational and 123 nonoperational exchanges Low
Vest, 2010
-453 hospitals operational HIE, 446 adopted HIE, and 3,931 had not adopted HIE; sample includes more general service type and fewer
for-profit hospitals than the more nationally representative AHA survey
-Overall, 81.4% of hospitals had not adopted or implemented HIE
-Adjusted regression OR of adoption for not for profit: 8.57; public: 9.53; number operational application: 1.02; physician portals: 1.38;
network membership: 1.33; ED visit: 1.01' primary care MD in HRR: 1.03
-Adjusted regression OR of implementation: network membership: 1.96; hi competition: 0.15; primary care MD: NS
Vest and Miller -10.4% had adopted Low

64 -10.7% had implemented HIE
2011
-Implemented hospitals, but not adopted hospitals, had higher nurse communication (0.75 increase [95% CI, 0.13 to 1.38]), global
satisfaction (0.82 [95% CI, 0.01 to 1.64]), and would recommend scores (1.34 [95% CI, 0.41 to 2.27]), and a trend toward higher doctor
communication scores (NS after controlling for confounders); results attenuated in propensity score analysis
-Communication: higher for smaller hospitals, rural hospitals, fewer Medicaid patients, higher nurse/patient ratios
-Satisfaction: higher for nonprofit, smaller, Midwest or south, fewer Medicaid patients, higher nursing ratios
Vest, et al., -No access of system for 97.7% of encounters Low

105 -Users accessed the I-Care system for 2.3% of the 271,305 encounters
2011
-Basic usage (2,527) 41.1% of instances
-Sample was predominately Hispanic, younger, and a higher proportion of charity care recipients
-Adjusted OR of access for African American and Hispanic: 0.76 to 0.89; higher for unknown or charity care; but mainly for unknown
payer: 4.7 vs. 2.6; access higher for more ED visits; hospitalizations: ~1.25-1.5 (from graph)
-Access lower for alcohol use, injury, poisoning, unfamiliar patient, busier than average day
F-210
Vest, et al., Case control Do hospitalizations, ED visits, Austin, Texas Indigent patients Log files from clinical data January 2006-June
104 and other factors predict HIE use and facilities that repository (Indigent Care 2009
2011
for indigent children? care for them Collaboration of Austin, Texas
safety net providers founded
1997); 18 hospitals, clinics,
government agencies (federally
qualified health centers)
Vest, et al., Case control Use of HIE in 2 ambulatory Austin, Texas 2 ambulatory Log files from clinical data January 2006-June
103 indigent clinics without EHRs, clinics serving repository (Indigent Care 2009
2012
and patient factors associated indigent people, Collaboration of Austin, Texas
with this use. part of nonprofit safety network providers founded
hospital system, 1997); 18 hospitals, public and
10,550-12,250 private clinics, government
encounters/year agencies (federally qualified
health centers)
Vest, Campion Jr., Cross-sectional Identify the strengths and New York State HEAL-NY (HIE Interviews March - June 2010
and Kaushal, weaknesses of organizational promotion Semi structured interviews with
156 models to achieve exchange, and legislation), HITEC selected experts
2013
what can be done to ensure the (academic
sustainability and effectiveness o collaborative
performs
evaluations)
F-211
Date HIE
2011
clinics, government agencies (federally qualified health million encounters,
2012
clinics, government agencies (federally qualified health million encounters,
Vest, Campion Jr., Various Various Various New York State

and Kaushal,
156
2013
F-212
Vest, et al., 179,445 encounters All ED encounters among Excluded any ED None
104 patients <18 years occurred encounters occurring at
2011
between January 1, 2006 and facilities before the hospital
June 30, 2009 and had parental had an authorized user of
consent the I-Care system.
Vest, et al., 39,447 encounters 6,393 patients Age 19-64 years Austin metro Children (different None
2012
103 area, consent to inclusion utilization) or ≥65 years
(Medicare)
Vest, Campion Jr., 17 of 21 invited HIE experts Selected to represent public, None stated NA
and Kaushal, private, leaders, participators,
156 policymakers
2013
F-213
Vest, et al., No usage vs. basic usage vs. novel usage 3 factors as indicative of uncertainty NR Quantitative multivariate
104 (more screens) that creates an information need: Logistic regression with
2011
comorbidity, prior utilization, and adjustment for by-patient
unfamiliarity with the patient clustering
Vest, et al., Encounter level or retrospective usage -Age Assessed with multivariate Quantitative multivariate
103 -Gender analysis, otherwise NR Primary care encounter: unit of
2012
-Race analysis; multinomial regression,
-ED visits over 3 months clustered to account of unit of
analysis, adjusted for
-Hospitalization over 12 months
confounders
-Fragmentation (N of clinics -1)
-Payer (Medicaid or not)
-Charlson comorbidity
-Independent mental health/substance
abuse comorbidity
-AHRQ chronic conditions indicator
definitions
Vest, Campion Jr., NA NA NA Qualitative

and Kaushal, Semistructured interview
156
2013 exploring issues from literature,
open independent coding and
comparison by 2 investigators,
consensus; [no triangulation of
data or analysis, no member
check]
F-214
Risk of
Vest, et al., -System accessed: 15,586 of 179,445 encounters (8.7%) Low
2011
104 -OR of basic HIE access for >1 year old vs. ≤1 year old: ~1.5 (from graph); lower for race unknown; higher for payer unknown; PC visits
within 12 months: ~1.5 (from graph); ED visits within 12 months: 1.5-2 (from graph); hospitalized: 1.3; number of diagnoses: 1.05;
unfamiliar: 0.46; busier than average: 0.65
-OR of novel HIE access for >1 year old vs. ≤1 year old: ~1.3; NS for race unknown; higher for payer unknown; PC visits within 12 months:
~2 (from graph); NS for ED visits within 12 months; hospitalized: 1.15; number of diagnoses: 1.05; unfamiliar: 0.19; NS busier than
average
Vest, et al., -Access for 21% of encounters Low

103 -7,101 encounter based, 1,227 retrospective
2012
-Adjusted OR for association with access for female: 1.12; >40 years: 1.16; chronic disease: 1.19; ED visit last 3 months: 1.13;
-Retrospective access, same 4 factors plus hospitalized last 4 months OR 1.33 and fragmentation OR 1.52
Vest, Campion Jr., Themes: (A) HIE is a public good; (B) challenges (1) financial challenges include upfront costs, discordance between investors and NA
and Kaushal, beneficiaries of technology "how to make that savings accrue to us and not to the payers.”; opportunity cost of lost revenue and lack of
156
2013 ROI "from a business perspective, HIE is kind of a bad idea. Why would we send out patient information elsewhere? We want to do it, we
think it’s necessary for better care of the patient, but we’ll lose money by doing it.”" (2) governance because "Federal, state, and private
representatives were fairly unanimous in their opinions that the functioning of RHIOs was not a technical issue" and the necessity of trust;
(3) mismatch of geographical model with reality of large integrated multistate delivery systems; (C) alternatives include Direct (lightweight,
treatment focused, lower organizational overhead; enterprise RHIOs, e.g.. "he hospital systems, they are the RHIO and they don’t want to
play with anybody else because they basically have quasi monopolies and cartels.” and don't need outside connection or support; Vendor
models likely but suboptimal; any of these not c/w state intent; (D) Sustainability quixotic, aims are contrary to market, contradiction of "
tension between providing a public good with little market incentives and operating like a private business"; alternatives: grow exchange
effort, specify a focus, evolve as an organization
F-215
Vest, et al., Case control Display and analyze the pattern Western New Nonprofit RHIO Log files, RHIO information about The log file was
102 of radiology report requests York State working with job title, job type, and location, limited to patients 18
2013
among organizations participating Hospital systems, and claims data. years and older and
in an HIE, and identify the patient reference reflected patient
and provider factors associated laboratories, encounters from
with use of a HIE system to radiology groups, January 2009-March
access radiology report insurance 2011
providers, and
county offices
Vest, et al., Retrospective Examines the hypothesis that Rochester, New HEAL NY Log file 2009-2010
56 cohort usage of an HIE system reduces York legislation,
2014 Claims files from 2 health plans
the odds that a patient in the ED statewide HIE that insure more than 60% of the
will be hospitalized. initiatives area population, log files of
usage, RHIO roster of users
F-216
Date HIE
Vest, et al., Rochester RHIO Commercial query-based web portal product, which includes NR Patients in health system in
102 patients’ discharge summaries, prior diagnoses, radiology western New York
2013
reports, medication history, and payer information. Both
radiology reports and images are accessible within the HIE
system and are typically available in near-real time after signoff.
Imaging studies are accessible only if the user first views the
radiology report. Our analysis is limited to the viewing of reports
only.
Vest, et al., Rochester RHIO >70 organizations in 13 county regions of western New York. Fully operational in 1,318 users accessed patient
56 Web-based portal that includes discharge summaries, March 2009 records in 156 different
2014
diagnoses, radiology reports and images, medication history, outpatient, emergency,
and payer information inpatient, long-term care, and
specialty care settings via a
web portal. 7 EDs were
included; 800,000 patients
(>70% of the area's adult
population)
F-217
Vest, et al., 29,528 radiology documents originating at 17 different Claims data only covers 60% of <18 years, not in health NA
102 source organizations, including hospitals and radiology population, included consenting system (included 60% of
2013
practices. A total of 126 different practice locations patients with ≥1 encounter in 6 pop, not the other 40%),
viewed these documents. months after consent had claims (64%, not the
other 36%)
Vest, et al., 1,5645 Claims files for 65% of patients None reported HIE access vs. no HIE access
2014
56 ≥18 years with valid consent (from log files)
dates (n=198,067) who had ≥1
encounter with a provider
registered to use the HIE system
in the 6 months following their
consent date.
F-218
Vest, et al., Radiology report access -Demographics NR Quantitative multivariate
102 -Encounter history
2013 Using network/graph analysis
-User characteristics assessed the difference between
-Insurance type the average number of
-AHRQ CCS ICD-9 codes connections among sources vs.
user practice locations, as well as
-Use of services in 30 days prior to
the average number of radiology
access
documents exchanged by data
-Claims for imaging procedures
sources vs. data users. Then (2)
-Health professional encounters
mixed effects logistic regression
on 134,127 sessions, 64% linked
to claims files, with some
accounting for clustering by
patient, user, workplace - report
results without control for
confounders, multiple
comparisons problem
Vest, et al., Hospital admission via the ED HIE system use at the time of the ED -Gender Quantitative
56 Economic visit, measured in a yes/no fashion -Age
2014 Logistic regression models. The
-Payer full model adjusts for all
-Disease severity in the 12-month independent variables with
period patient age, the count of major
-Any primary care, specialty care, aggregated diagnostic groups,
or ED visits in the 30 days after the and the number of prior
index hospitalization (or up until hospitalizations treated as
the date of readmission) continuous variables, 4 sensitivity
analyses to explore the
robustness including physician
effects and patient subgroup
(sickest) effects
F-219
Risk of
Vest, et al., Network: each source organization sent on average 971 (range: 6-8,002) documents to 49 (3-106) other organizations. User Low
102 organizations accessed on average 49 (1-8,444) documents from 6 (1-17) source organizations. Algorithm suggests 11/17 source
2013
organizations represent a core set of data providers, including 8 hospitals and 3 stand-alone radiology sites. Thus the overall number of
radiology reports retrieved in the outpatient setting was 16.9 times greater than the number of reports retrieved in the ED and inpatient
settings combined (23,201 outpatients vs. 1,333 ED and 313 inpatients).
Factors: 86,152 user sessions with associated claims files represented the activity of 1,119 different users representing 145 different
workplace locations. 86.4% were staff; physicians represented only about 4% of all sessions; overall 11.2% of sessions included access
of radiology reports.
Vest, et al., -ED visit within 6 months of consent: 15,645 Low

56 -Of ED visits, HIE accessed: 2.4% (n=374)
2014
-16/229 MDs used system
-OR of admission for Medicare: 2.02; Medicaid: 0.61; male: 1.47
-Adjusted OR of HIE access: 0.7; HIE access on same day as ED visit: 0.83 (95% CI, 0.55 to 1.25)
-Odds of an admission were 30% lower when the system was accessed after controlling for confounding (OR 0.70; 95% CI, 0.52 to 0.95)
-Annual savings in the sample was $357,000
F-220
Vest and Issel, Cross-Sectional To examine factors related to United States State and local Surveys 2007-2008
157 public health organizations data health
2014
exchange capabilities departments
Vest, et al., Retrospective To determine the association Rochester, New HEAL NY Log file 2009-2010
2014
55 cohort between usage of an HIE system York legislation, Claims files from 2 health plans
post- discharge and 30-day same- statewide HIE that insure more than 60% of the
cause hospital readmissions. initiatives.
area population, log files of
Outpatient
usage, RHIO roster of users
Willis, et al., RCT To evaluate 2 decision support North Carolina Outpatient Database -December 7, 2009-
67 interventions: patient adherence December 6, 2010
2013 EHR and claims as well as logs
reports to providers and reports of contacts and cost/revenue was intervention
to providers and emails to care data period
managers by comparing to usual -Followup for
care. outcomes ended
August 30, 2011
F-221
Date HIE
Vest and Issel, Varies, any system that would allow Varies Varies U.S. states
157 data sharing
2014
Vest, et al., Rochester RHIO Web based portal that includes discharge summaries, Fully operational in 800 000 patients (>70% of the
55 diagnoses, radiology reports and images, medication history, March 2009 area's adult population)
2014
and payer information, 38 healthcare organizations in 11
counties
Willis, et al., Northern Piedmont Community -Included 9 clinics and 5 hospitals NR Network Medicaid
67 Care Network. Set up a system -Data collected by the system include: 1) administrative data beneficiaries
2013
called COACH (Community- 2) care management data; 3) claims/billing data ; 4)
Oriented Approach to Coordinated scheduling data; 5) clinical data; 6) data on communications
Healthcare)
F-222
Vest and Issel, 44 states with representatives who responded to both Executive officer of local health States missing data on Public health organizations that
157 surveys department and state health either survey don't have the capacity to
2014
officials exchange data
Vest, et al., 196,314 patients, 11 hospitals (2/3 of sample) ≥18 years, consented during <30 observations in the HIE access vs. no HIE access
55 2009-2010, continuously dataset (n=11) (from log files)
2014
enrolled in health plan, ≥1
encounter in 6 months following
consent, (196,314 patients met
these requirements). Only the
patient's first hospital admission
within the first 5 months after
consent. Each patient appears in
the dataset only once and each
discharge could be followed for
≥30 days.
Willis, et al., N=2219 Patients with ≥1 of 6 targeted Not continuously enrolled Provider report vs. provider
67 739 to usual care IOM priority conditions during the intervention report and case manager event
2013
744 clinic reports period vs. usual care in which neither
735 clinic reports and care manager notices type of alert was delivered
F-223
Vest and Issel, Bidirectional data sharing for childhood Organizational characteristics including None reported Quantitative Multivariate Analysis
157 immunizations, vital records and reportable size, structure, processes and IT
2014
conditions readiness
Vest, et al., Readmission within 30 days of discharge HIE system usage -Gender Quantitative
55 for the same cause as the index -Age
2014 Random effects logistic
hospitalization -Payer regression models, a series of
-Disease severity in the 12-month models adjusting for patient
period any primary care, specialty characteristics, then adding post-
care, or ED visits in the 30 days discharge utilization measures,
after the index hospitalization (or and lastly including hospital-level
up until the date of readmission) characteristics. Controlled for
-Described the index potential hospital-level clustering
hospitalization site: hospital bed using the index admission
size, teaching status, affiliation with hospital as a random intercept.
a multi-hospital healthcare system, Then 2 sensitivity analyses.
and critical access hospital
classification, case mix index
derived from the relative values of
diagnosis-related groups seen at
the hospital
Willis, et al., -Clinical outcomes including: medical Group assignment None reported Quantitative
67 adherence, outpatient, ED visits, and
2013 Generalized estimating equation
hospitalizations models that accounted for
-Care coordination clustering by family
costs/revenues
-Clinician satisfaction
F-224
Risk of
Vest and Issel, Data sharing capacity varied by activity. 66% had capacity for Immunizations Moderate
157 30.2% for vital records and
2014
18.9% for reportable conditions
Vest, et al., -Readmitted within 30 days: 9.8% (668/6,807); 29.6% at a different facility; 394 had HIE access within 30 days after discharge, 20 (5.8%) Low
55 readmitted; p=0.00113
2014
-ED visits within 30 days post discharge: NS
-HIE access associated with lower readmissions: OR 0.43 (95% CI, 0.27 to 0.70)
-Primary care or specialty care associated with lower readmissions rates: ORs 0.48 and 0.67 in final model
-ED visits associated with higher rates: OR 9.3 in final model
-Accessing patient information in the HIE in the 30 days after discharge associated with a 57% lower adjusted odds of readmission (OR
0.43; 95% CI 0.27 to 0.70). Estimated annual savings in the sample from averted readmissions associated with HIE usage was $605,000.
Willis, et al., Control vs. reports vs. reports and email Moderate
67
2013 % medication adherence: 41.3% vs. 41.2% vs. 42.9%, p=NS; no differences between groups at 6 months
Encounter rates of outpatient: 46.0 vs. 46.6 vs. 44.5, p=NS
Encounter rates of ED: 0.87 vs. 0.84 vs. 0.89, p=NS
Encounter rates of hospitalizations: 0.19 vs. 0.21 vs. 0.21, p=NS
-15% to 50% of reports were not available to providers at time of patient encounter
-Even when they had reports, clinicians did not always discussion medication adherence with patients
F-225
Winden, et al., Case series To determine value of Epic Care Minneapolis, ED Observations January-November,
71 Everywhere in an ED Minnesota 2012
2014 Chart review, focus groups,
survey
Yeager, et al., Cross-sectional To examine the barriers and Louisiana NR in this paper Interview March to April 2013
137 facilitators affecting the decision
2014
to participate in an HIE and,
separately, which factors are
affecting the use of HIE.
F-226
Date HIE
Winden, et al., Allina Health and local Directed transfer of Epic records to Allina ED August, 2010 All patients for whom CE used;
71 organizations using Epic focus groups of clinician users
2014
Yeager, et al., Louisiana HIE (LaHIE), statewide. Louisiana HIE (LaHIE). LaHIE functions as a hybrid centralized NR Patients in Louisiana
2014
137 Number of centers/settings not and federated model, web-based platform for providers to share
presented in this paper. patient care continuity documents (commonly referred to as
CCDs), laboratory results, and electrocardiogram results.
F-227
Winden, et al., Focus groups: 49 clinicians in 4 hospitals; Survey: 118 Focus groups: clinicians; Survey: Notes: CE not used Focus group and survey: value
71 of 408 ED staff; review of 1,488 notes where CE used ancillary staff; Notes: use of CE for care; Chart review: tests
2014
avoided
Yeager, et al., 16 Healthcare representatives from organizations NR NR NA

137 interested in joining LaHIE but not yet enrolled (n=4),
2014
not interested in joining (n=4), or already enrolled (n=8)
F-228
Winden, et al., Focus groups: provided value for patient Focus groups and survey: value for None Quantitative
71 care, especially for avoiding duplicate patient care; Chart review: procedures
2014 Survey, chart review
testing and detecting drug-seeking avoided
behavior; Survey: provided value in patient
care; Chart review: procedures avoided
Yeager, et al., Barriers to implementation of LaHIE as NA NA Qualitative, content analysis

137 identified by interviews with health care
2014
representatives
F-229
Risk of
Winden, et al., Focus groups: provided value for patient care, especially for avoiding duplicate testing and detecting drug-seeking behavior; Survey: 74% Moderate
71 agreed provided value in patient care; Chart review: 560 procedures avoided in 237 notes out of 1,488 assessed
2014
Yeager, et al., "Findings suggest that Meaningful Use requirements are a critical factor influencing the decision to participate in the HIE, specifically the Moderate
137 mandate that hospitals be able to electronically transfer summary of care documents. Creating buy-in within a few large hospital networks
2014
legitimized the HIE and hastened interest in those markets. Fees charged by electronic health record (EHR) vendors to develop HIE
interfaces have been prohibitive. Funding from the federal incentive program is intended to offset the costs associated with EHR
implementation and increase the likelihood that HIEs can provide value to the population; however, costs and time delays of EHR interface
development may be key barriers to fully integrated HIEs. State HIEs may benefit from targeted involvement of state health care leaders
who can champion the potential value of the HIE"
F-230
eHealth Initiative Cross-sectional To assess the status of data Nationwide Any Survey responses 2013; comparison to
73 exchange in the U.S. 2011
2013 Report
F-231
Date HIE
eHealth Initiative Various 199 of 315 completed the survey; these were a mix of Varies 315 data exchange initiatives
73 community data exchanges, statewide efforts, & healthcare were identified
2013 Report
delivery organizations.
F-232
eHealth Initiative -199 of 315 completed the survey; these were a mix of NR NR NA
73 community data exchanges, statewide efforts, &
2013 Report
healthcare (HC) delivery organizations.
-90 organizations self-identified as community-based
HIEs; 45 as state; 50 as health care delivery
organizations.
-There is no single dominant model for HIE; 125
organizations used a query model, 124 used secure
electronic messaging; 111 used end-to-end integration;
84 used a combination of models.
-'Direct' is a standards-based protocol for securely
exchanging data; 90 organizations use M117'Direct',
mostly in transitions of care.
-Patient consent for data exchange generally remains
an 'all-or-nothing' proposition, with 'opt-out' the most
common consent model.
F-233
eHealth Initiative -Number of initiatives reaching 'advanced NR NR NR
73 stages of operation, sustainability or
2013 Report
innovation (as defined by eHI's
developmental framework)
-Number of years to become operational
-Trends in use since 2011
-Number of organizations self-identifying as
community, state-, or HC delivery system
-Types of professionals most commonly
providing and using data
-Types of data most commonly
provided/viewed
-Number having hired personnel from
ONC's Workforce Development Program
(WDP)
-Protocol used for securely exchanging
information
-Key Findings
-Issues for the future
F-234
Risk of
eHealth Initiative 84 organizations had reached an 'advanced' stage of operation, sustainability, or innovation. NA
73 Most took 2 years to become operational.
2013 Report
Among organizations responding in 2011 and 2013, 27 more had reached stages 5, 6, or 7 in 2013.
90 organizations self-identified as community-based HIEs; 45 as state-; 50 as HC delivery organizations.
Hospitals and Am Care providers are stakeholders most commonly providing/viewing data. Labs also commonly provide data. Community
public health clinics commonly view data.
24 reported they had hired staff from the ONC's WDP, compared to only 3 in 2011.
'Direct' is a standards-based protocol for securely exchanging data; 90 organizations use 'Direct', mostly in transitions of care.
There is no single dominant model for HIE; 125 organizations used a query model, 124 used secure electronic messaging; 111 used end-
to-end integration; 84 used a combination of models.
Key Findings:
1) Achieving interoperability with disparate information systems is a major concern; 68 initiatives have had to connect with more than 10
different systems;
2) To overcome interoperability challenges, exchanges would like to see standardized pricing and integration solutions from vendors;
3) Many exchanges are not sharing data with competing organizations;
4) Exchanges are focusing on functionalities to support health reform and advance analytics;
5) Patient engagement remains low amongst organizations exchanging data;
6) Patient consent for data exchange generally remains an 'all-or nothing' proposition, with 'opt-out' the most common consent model;
7) Since 2011, more initiatives have become more financially viable. However, hospitals and payers are still expected to fund most
exchange activity; of the 51 that were NOT sustainable, 31 (of 51) receive more than 50% of their funding from the federal government
and 22 report they are a state-HIE.
Overall, in 2011, 16 reported they were sustainable; in 2013, 35 reported they were sustainable. Organizations realize the precariousness
of government funding and are trying to offer valuable services for a fee.
Issues for the future:
1) Interoperability concerns need to be addressed;
2) Health reform provides exchanges an opportunity to show value;
3) Patient engagement remains poor.
F-235
eHealth Initiative Cross-sectional To assess the status of data Nationwide Any Survey responses 2013; comparison to
74 exchange in the US. 2011
2014 Report
F-236
Date HIE
eHealth Initiative Various 199 of 315 completed the survey; these were a mix of Varies 315 data exchange initiatives
74 community data exchanges, statewide efforts, & healthcare were identified
2014 Report
delivery organizations.
F-237
eHealth Initiative -199 of 315 completed the survey; these were a mix of NR NR NA
74 community data exchanges, statewide efforts, &
2014 Report
healthcare (HC) delivery organizations.
-90 organizations self-identified as community-based
HIEs; 45 as state; 50 as health care delivery
organizations.
-There is no single dominant model for HIE; 125
organizations used a query model, 124 used secure
electronic messaging; 111 used end-to-end integration;
84 used a combination of models.
-'Direct' is a standards-based protocol for securely
exchanging data; 90 organizations use M117'Direct',
mostly in transitions of care.
-Patient consent for data exchange generally remains
an 'all-or-nothing' proposition, with 'opt-out' the most
common consent model.
F-238
eHealth Initiative -Number of initiatives reaching 'advanced NR NR NR
74 stages of operation, sustainability or
2014 Report
innovation (as defined by eHI's
developmental framework)
-Number of years to become operational
-Trends in use since 2011
-Number of organizations self-identifying as
community, state-, or HC delivery system
-Types of professionals most commonly
providing and using data
-Types of data most commonly
provided/viewed
-Number having hired personnel from
ONC's Workforce Development Program
(WDP)
-Protocol used for securely exchanging
information
-Key Findings
-Issues for the future
F-239
Risk of
eHealth Initiative Who provides data: 112 hospitals, 100 Am Care providers, 56 labs, 52 community/public health clinics. NA
74 Who accesses data: 111 Am Care providers, 104 hospitals, 75 community/public health clinics, 65 behavioral or mental health providers.
2014 Report
Key Barriers: 1) Cost and technical challenges are key barriers to interoperability; 2) Regulatory policies appear to have prompted
increased use of core HIE services such as 'Direct', care summary exchange, and transitions of care; 3) Advanced initiatives are
supporting new payment and advanced delivery models; 4) Sustainable organizations have replaced federal funding with revenue from
fees and membership dues.
Key finding 1: Interoperability Challenges include costs of building interfaces, getting consistent and timely response from EMR vendors
and interface developers, and technical difficulty of building interfaces. 112 organizations have had to construct multiple interfaces and 18
have had to construct more than 25 interfaces.
Suggestions for overcoming interoperability challenges include: 1) standardized pricing and integration solutions from vendors; 2) 'plug
and play' platform; 3) federally mandated standards; 4) cultural changes in willingness to share data; 5) greater use among providers of
consensus-based standards.
Key finding 2: Regulatory Policies prompt use of core HIE Services:
101 incorporate secure messaging into their models; 78 offer a 'Direct' address directory; more respondents are using 'Direct' for all given
use cases (when compared to last year). 74 have met at least one Stage 2 Meaningful Use criteria. 7 stages of Development are
delineated (see slide in report for detail);
Key finding 3: Advanced initiatives are supporting new payment & delivery models: 106 reported they have reached stage 6 (operating) or
higher on the eHI's HIE maturity scale (an increase of 11% over 2013).
64 support an ACO; 52 support a PCMH; 21 support a State Innovation Model; 12 support a bundled payment initiative.
Key finding 4: Sustainable groups replace fed funding with fees and membership dues: 45 use fees to completely cover operational
expenses; 38 use fees but need additional funding. 41 report that dues or fees are greatest revenue source; 89 believe dues or fees will
eventually be their primary revenue stream.
Looking to the future:
1) Data exchange is reaching a point of stability and acceptance.
2) Organizations are settling on a set of core service offerings and a standard approach to sustainability (sub-bullet: despite expiration of
large funding sources, radical changes in overall landscape are not evident);
3) As organizations mature, they will offer new and innovative services (public health has already leverages HIE; alert notification services
may help ACOs to track patients);
4) Organizations are encouraged to work collaboratively to overcome remaining challenges (especially work with regional/community
partners to avoid creating 'pockets' of exchange).
F-240
* this is from billing data, not EHR
†one site dropped that didn't have comparable qualitative data.
A1c= glycated hemoglobin; AHA= American Hospital Association; AHRQ= Agency for Healthcare Research and Quality; aka= also
known as; AMIE= Arizona medical information exchange; ANOVA= analysis of variance; BHIX= Brooklyn Health Information
Exchange; CCD= continuity of care document; CCHIT= Certification Commission for Healthcare Information Technology; CCR=
community care record; CCS= clinical classification software; CD4= HIV helper cell count; CDA = clinical document architecture;
CDC= Centers for Disease Control and Prevention; CE= Care Everywhere; CEN= clinical event notification; CHIC RHIO= Carolina
HIV information cooperative regional health information organization; CI= confidence interval; CIO= chief information officer;
COACH= Community Oriented Approach to Coordinated Healthcare; CPT4= Current procedure Terminology; CT= computed
axial tomography scan; DOD= Department of Defense; e= electronic; e.g.= for example; ebSML RIM= electronic business using
extensible markup language registry information model; ebXML RS= electronic business using extensible markup language; ECCI=
Electronic Clinical Communication Implementation Program; ED= emergency department; EDI= electronic data interchange;
EDIFACT= electronic data interchange for administration, commerce and transport; eHIE= electronic health information exchange;
EHR= electronic health records; EKG= electrocardiogram; ELRs = enhanced laboratory reports; EMR= electronic medical records;
EMS= emergency medical services; e-OP= electronic outpatient appointment booking; EPIC= electronic privacy information center;
et al.= and others; etc.= etcetera; EPR= electronic patient records; e-RR= electronic results reporting; EU27= 27 nations in the
European Union; FITT= fit between individuals tasks and technologies; FUHN= Federally Qualified Health Center Urban Health
network; FQHCs= federally qualified health centers; GDP= gross domestic product; GP= general practitioner; HC= Health Care;
HCAHPS= Hospital Consumer Assessment of Healthcare Providers and Systems; HEAL = Health Care Efficiency and Affordability
Law; HEAL NY= Health Care Efficiency and Affordability Law for New York; HEDIS= health care effectiveness data and
information set; HIE= health information exchange; HIMSS= healthcare information and management systems society; HIMSS-
AD= healthcare information and management systems society analytical database; HIO= Health Insuring Organization; HIPAA=
Health Insurance Portability and Accountability Act; HITECH= Health Information Technology for Economic & Clinical Health
Act; HL-7= Health Level 7; HL7; HMO= health maintenance organization; HRR= unadjusted hazard ratio; HRSA= `Health
Resources and Services Administration; Id = Identifier; i.e.= that is; ICC= integrated care collaboration; ICD-9= Ninth Revision of
the International Classification of Diseases; ICD-9-CM= International Classifications of Diseases, Clinical Modification; ICU=
intensive care unit; IDS= integrated delivery system; I-EMS= Indianapolis Emergency Medical Services; IHIE= Indiana Health
Information Exchange; IM & T=information management & technology; INPC= Indiana Network fro Patient Care; IOM= Institute
of Medicine's; IQR= interquartile range; IS = information system; IT= information technology; KP= Kaiser Permanente?;
F-241
LaHIE=Louisiana HIE; LaPHIE= Louisiana Public Health Information Exchange; LBNH= Long Beach Network for Health;
LOINC= Logical Observation Identifiers Names and Codes; MAeHC= Massachusetts eHealth Collaborative; MANOVA=
multivariate analysis of variance; MD= Doctor of Medicine; MEGAHIT= Medical Evidence Gathering Through Health IT;
MHDC= Massachusetts Health Data Consortium; mL= milliliter; mm= millimeter; MN= Minnesota; MPI= master patient index;
MRI= magnetic resonance imaging; MRSA= Methicillin Resistant Staphylococcus Aureus; MSeHA= MidSouth e-Health Alliance;
N= sample size; NA= not applicable; NAMCS= National Ambulatory Medical Care Survey; NDC= National Drug Code; NE=
northeast; NHIN= Nationwide Health Information Network; NLM= National Library of Medicine; NR= not relevant; NS= not
significant; NY= New York; NYCLIX= New York Clinical Information Exchange; OLS= ordinary least squares; ONC= Office of
the National Coordinator for Health Information Technology; OR= odds ratio; PBMs= pharmacy benefit managers; PC= primary
care; PCP = primary care provider; PDF= portable document format; PHI= personal health information; PHR= personal health
record; PPO= preferred provider organization; QUIS= Questionnaire for User Interaction Satisfaction; RCT= randomized,
controlled trial; RHIE = regional health information exchange; RHIO= regional health information organization; RLS= record
locator service; RNA= ribonucleic acid; RR= relative risk; SCR= summary care record; SD= standard deviation; S.E.= southeast;
SF-12= Short Form-12 item survey; SHIN-NY= Statewide Health Information Network for New York; SMRTnet= Secure Medical
Records Transfer Network; SNOCAP-USA= State Networks of Colorado Ambulatory Practices & Partners United States of
America; SNOMED= Systemized Nomenclature of Medicine; SSA= Social Security Administration; SUNBH = Seoul National
University Bundang Hospital; TILAK= Tiroler Landeskrankenanstaleten ; TOE= technological, organizational and environmental;
TUMA= Trans-mural exchange of medication data in Almere; U.K.= United Kingdom; U.S.= United States; URL= uniform
resource locator; USB= universal serial bus; VA= U.S. Department of Veterans Affairs; VL= viral load; VLER= Veterans Lifetime
Electronic Record; VRE= Vancomycin resistant enterococci; vs.= versus; WHIE= Wisconsin Health Information Exchange; XML=
extensible markup language.
F-242
Appendix G. Risk of Bias Assessment Criteria
Our assessment of risk of bias was based on the recommendations in the Agency for
Healthcare Research and Quality Methods Guide for Effectiveness and Comparative
Effectiveness Reviews.1 Included studies were classified according to type of design (see
Appendix E) as part of the data abstraction phase, and each major type of study was assessed for
bias according to relevant criteria. This criteria included questions that assessed selection bias,
performance bias, detection bias, attrition bias, and reporting bias (i.e., those about adequacy
of randomization, similarity of groups at baseline, appropriateness of the comparators,
consideration of concurrent interventions or unintended exposures, quantity of missing data,
methods of handling missing data, identification and assessment of important confounding
variables, use of intention-to-treat analysis, reliability and validity of outcome measures, and
reporting of pre specified outcomes).
Criteria for Randomized Controlled Trials

Selection bias
• Was randomization adequate?
• Was allocation concealment adequate?
• Were groups similar at baseline?
• Did the study maintain comparable groups throughout the study?
• Was the eligibility criteria specified?
Detection bias
• Was the study adequately blinded (outcome assessor, care provider, and patient)?
Attrition bias
• Was the loss to followup not differential or high?
Reporting bias
• Did the study report attrition, crossovers, adherence, and contamination?
• Was an intention-to-treat analysis used?
• Were outcomes prespecied?
Criteria for Cohort, Case-Control, and Other Observational

Studies
Selection bias
• Are the comparison groups or time periods appropriate?
• Were the inclusion and exclusion criteria specified and applied equally to each group?
• Did the design and analyses account for important potential confounding and modifying
variables appropriately?
• Were valid and reliable measures used (inclusion/exclusion, confounding, outcomes)?
Detection bias
• Were non-biased and valid ascertainment methods used (inclusion/exclusion,
confounding, outcomes)?
G-1
• Was the timing and/or time period for the measurement of the intervention and outcomes
appropriate?
Attrition bias
• Was there NO missing data? If missing data, was it handled appropriately?
Reporting bias
• Were outcomes prespecified and were prespecified outcomes reported?
Definition of ratings based on above criteria:

Low risk of bias:
Studies rated “low risk of bias” were considered to have the least risk of bias, and their
results are considered valid. Low risk of bias studies include clear descriptions of the
population, setting, interventions, and comparison groups clear reporting of missing data;
appropriate means for preventing bias; and appropriate measurement of outcomes.
Moderate risk of bias:
Studies rated “moderate risk of bias” were susceptible to some bias, though not enough to
necessarily invalidate the results. These studies may not meet all the criteria for a rating
of low risk of bias, but do not have flaws likely to cause major bias. The study may be
missing information, making it difficult to assess limitations and potential problems. The
moderate risk of bias category is broad, and studies with this rating will vary in their
strengths and weaknesses. The results of some moderate risk of bias studies are likely to
be valid, while others may be only possibly valid.
High risk of bias:
Studies rated “high risk of bias” have significant flaws that imply biases of various types
that may invalidate the results. They will have a serious or “fatal” flaw in design,
analysis, or reporting; large amounts of missing information; or discrepancies in
reporting. The results of these studies will be least as likely to reflect flaws in the study
design as the true difference between the compared interventions. We did not exclude
studies rated as being high risk of bias a priori, but high risk of bias studies were
considered to be less reliable than lower risk of bias studies when synthesizing the
evidence, particularly if discrepancies between studies were present.
Criteria for Surveys, Focus Groups, and Interview Studies

Selection bias
1. Is the sampling strategy or selection criteria reported and appropriate?
2. Are the response or participation rates reported and are they acceptable given the type of
study?
3. Are characteristics (e.g., demographics) of respondents/participants reported?
Detection bias
4. Is how the questions were developed/selected reported and is it appropriate?
5. Were confounders considered (could be in analysis or presentation, such as stratifying
results)?
Other
6. Is analysis appropriate (given the type of data)?
G-2
Reference for Appendix G
1. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ
Publication Number 10(14)-EHC062-EF. Rockville, MD: Agency for Healthcare
Research and Quality. January 2014. Available at: www.effectivehealthcare.ahrq.gov.
Accessed April 18, 2014. PMID: 21433403.
G-3
Appendix H. Strength of Evidence Criteria1
The set of five required domains comprises the main constructs that Evidence-based Practice
Centers (EPCs) should use for all major outcomes and comparisons of interest. As briefly
defined below in Table H1, these domains represent related but separate concepts and each is
scored independently. The concepts are explained in more detail below.
Table H1. Required domains and their definitions

Domain Definition and Elements Score and Application
Study Study limitations is the degree to which the included Score as one of three levels, separately
Limitations studies for a given outcome have a high likelihood of by type of study design:
adequate protection against bias (i.e., good internal • Low level of study limitations
validity), assessed through two main elements: • Medium level of study limitations
• Study design: Whether RCTs or other designs such as • High level of study limitations
nonexperimental or observational studies.
• Study conduct. Aggregation of ratings of risk of bias of
the individual studies under consideration.
Directness Directness relates to (a) whether evidence links Score as one of two levels:
interventions directly to a health outcome of specific • Direct
importance for the review, and (b) for comparative • Indirect
studies, whether the comparisons are based on head-
to-head studies. The EPC should specify the If the domain score is indirect, EPCs
comparison and outcome for which the SOE grade should specify what type of indirectness
applies. accounts for the rating.
Evidence may be indirect in several situations such as:
• The outcome being graded is considered intermediate
(such as laboratory tests) in a review that is focused on
clinical health outcomes (such as morbidity, mortality).
• Data do not come from head-to-head comparisons but
rather from two or more bodies of evidence to compare
interventions A and B—e.g., studies of A vs. placebo
and B vs. placebo, or studies of A vs. C and B vs. C but
not direct comparisons of A vs. B.
• Data are available only for proxy respondents (e.g.,
obtained from family members or nurses) instead of
directly from patients for situations in which patients are
capable of self-reporting and self-report is more reliable.
Indirectness always implies that more than one body of

evidence is required to link interventions to the most
important health outcome.
Consistency Consistency is the degree to which included studies find Score as one of three levels:
either the same direction or similar magnitude of effect. • Consistent
EPCs can assess this through two main elements: • Inconsistent
• Direction of effect: Effect sizes have the same sign • Unknown (e.g., single study)
(that is, are on the same side of no effect or a MID)
• Magnitude of effect: The range of effect sizes is Single-study evidence bases (including
similar. EPCs may consider the overlap of CIs when mega-trials) cannot be judged with
making this evaluation. respect to consistency. In that instance,
use “Consistency unknown (single
The importance of direction vs. magnitude of effect will study).”
depend on the key question and EPC judgments.
H-1
Domain Definition and Elements Score and Application
Precision Precision is the degree of certainty surrounding an Score as one of two levels:
effect estimate with respect to a given outcome, based • Precise
on the sufficiency of sample size and number of events. • Imprecise
• A body of evidence will generally be imprecise if the
OIS is not met. OIS refers to the minimum number of A precise estimate is one that would
patients (and events when assessing dichotomous allow users to reach a clinically useful
outcomes) needed for an evidence base to be conclusion (e.g., treatment A is more
considered adequately powered. effective than treatment B).
• If EPCs performed a meta-analysis, then EPCs may
also consider whether the CI crossed a threshold for an
MID.
• If a meta-analysis is infeasible or inappropriate, EPCs
may consider the narrowness of the range of CIs or the
significance level of p-values in the individual studies in
the evidence base.
Reporting Bias Reporting bias results from selectively publishing or Score as one of two levels:
reporting research findings based on the favorability of • Suspected
direction or magnitude of effect. It includes: • Undetected
• Study publication bias, i.e., nonreporting of the full
study. Reporting bias is suspected when:
• Selective outcome reporting bias, i.e., nonreporting (or • Testing for funnel plot asymmetry
incomplete reporting) of planned outcomes or reporting demonstrates a substantial likelihood of
of unplanned outcomes. bias,
• Selective analysis reporting bias, i.e., reporting of one
or more favorable analyses for a given outcome while And/or
not reporting other, less favorable analyses. • A qualitative assessment suggests the
likelihood of missing studies, analyses,
Assessment of reporting bias for individual studies or outcomes data that may alter the
depends on many factors–e.g. availability of study conclusions from the reported evidence.
protocols, unpublished study documents, and patient-
level data. Detecting such bias is likely with access to Undetected reporting bias includes all
all relevant documentation and data pertaining to a alternative scenarios.
journal publication, but such access is rarely available.
Because methods to detect reporting bias in
observational studies are less certain, this guidance
does not require EPCs to assess it for such studies.
CI = confidence internal; EPC = Evidence-based Practice Center; MID = minimally important difference; OIS = optimal
information size; RCT = randomized controlled trial[ SOE = strength of evidence
Study Limitations Domain Definition

Scoring the study limitations domain is the essential starting place for grading strength of the
body of evidence. It refers to the judgment that the findings from included studies of a treatment
(or treatment comparison) for a given outcome are adequately protected against bias (i.e., have
good internal validity), based on the design and conduct of those studies. That is, EPCs assess
the ability of the evidence to yield an accurate estimate of the true effect without bias
(nonrandom error).
Directness Domain Definition

Directness of evidence expresses how closely available evidence measures an outcome of
interest. Assessing directness has two parts: directness of outcomes and directness of
comparisons. Applicability of evidence (external validity) is considered explicitly but separately
from strength of evidence.
H-2
Consistency Domain Definition
Consistency refers to the degree of similarity in the direction of effects or the degree of
similarity in the effect sizes (magnitudes of effect) across individual studies within an evidence
base. EPCs may choose which of these two notions of consistency (direction or magnitude) they
are scoring; they should be explicit about this choice.
Precision Domain Definition

Precision is the degree of certainty surrounding an estimate of effect with respect to an
outcome. It is based on the potential for random error evaluated through the sufficiency of
sample size and, in the case of dichotomous outcomes, the number of events. A precise body of
evidence should enable decisionmakers to draw conclusions about whether one treatment is
inferior, equivalent, or superior to another.
Reporting Bias Definition

Reporting bias occurs when authors, journals, or both decide to publish or report research
findings based on their direction or magnitude of effect. Table 2 defines the three main types of
reporting bias that either authors or journals can introduce: publication bias and outcome and
analysis reporting bias.
Four Strength of Evidence Levels

The four levels of grades are intended to communicate to decisionmakers EPCs’ confidence
in a body of evidence for a single outcome of a single treatment comparison. Although assigning
a grade requires judgment, having a common understanding of the interpretation will be useful
for helping EPCs as they conduct their own global assessment and for improving consistency
across reviewers and EPCs.
Table H2 summarizes the four levels of grades that EPCs use for the overall assessment of
the body of evidence. Grades are denoted high, moderate, low, and insufficient. They are not
designated by Roman numerals or other symbols. EPCs should apply discrete grades and should
not use designations such as “low to moderate” strength of evidence.
Table H2. Strength of evidence grades and definitions

Grade Definition
High We are very confident that the estimate of effect lies close to the true effect for this
outcome. The body of evidence has few or no deficiencies. We believe that the findings are
stable, i.e., another study would not change the conclusions.
Moderate We are moderately confident that the estimate of effect lies close to the true effect for
this outcome. The body of evidence has some deficiencies. We believe that the findings are
likely to be stable, but some doubt remains.
Low We have limited confidence that the estimate of effect lies close to the true effect for this
outcome. The body of evidence has major or numerous deficiencies (or both). We believe that
additional evidence is needed before concluding either that the findings are stable or that the
estimate of effect is close to the true effect.
Insufficient We have no evidence, we are unable to estimate an effect, or we have no confidence in
the estimate of effect for this outcome. No evidence is available or the body of evidence has
unacceptable deficiencies, precluding reaching a conclusion.
H-3
Each level has two components. The first, principal definition concerns the level of
confidence that EPCs place in the estimate of effect (direction or magnitude of effect) for the
benefit or harm; this equates to their judgment as to how much the evidence reflects a true effect.
The second, subsidiary definition involves an assessment of the level of deficiencies in the body
of evidence and belief in the stability of the findings, based on domain scores and a more
holistic, summary appreciation of the possibly complex interaction among the individual
domains.
Assigning a grade of high, moderate, or low implies that an evidence base is available from
which to estimate an effect for either the benefit or the harm. The designations of high, moderate,
and low should convey how confident EPCs would be about decisions based on evidence of
differing grades, which can be based on either quantitative or qualitative assessment.
For comparative effectiveness questions, the comparison is typically a choice of either
direction (A>B, A=B, A<B) or magnitude (difference between A and B). In some instances
assigning different grades regarding the direction and the magnitude of an effect may be
appropriate. An example of this situation is when studies consistently find that an intervention
improves an outcome (e.g., apnea-hypopnea index is reduced by a statistically significant amount
or beyond a minimally important difference), but the degree of heterogeneity about the estimate
is high (e.g., range -10 to -46 events/minute; I2 = 86%).
The importance of the distinctions among high, moderate, and low levels (and the distinction
with insufficient strength of evidence) can vary by the type of outcome, comparison, and
decisionmaker. EPCs understand that some stakeholders may want to take action only when
evidence is of high or moderate strength, whereas others may want to understand clearly the
implications of low versus insufficient evidence. Even when strength of evidence is low or
insufficient, consumers, clinicians, and policymakers may find themselves in the position of
having to make choices and decisions, and they may consider factors other than the evidence
from a specific systematic review, such as patient values and preferences, costs, or resources.
Reference for Appendix H

1. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ
Publication No. 10(13)-EHC063-EF. Rockville (MD) :Agency for Healthcare Research
and Quality. January 2014. Availible at: www.effectivehealthcare.ahrq.gov.
H-4
Appendix I. Quality Assessment Tables
Table I-1. Quality assessments of randomized controlled trials
Allocation Outcome
Maintain Eligibility Care provider
Author, Year Randomization concealment Groups similar at assessors Patient masked?
Comparable criteria masked?
adequate? adequate? baseline? masked?
Groups? specified?
Afilalo, et al., Yes Yes Yes Yes Yes Yes No No
66
2007
Lang, et al.,
65
2006
Willis, et al., Yes Not Reported Unclear Unclear Yes Unclear No No
67
2013
I-1
Reporting of No Loss to Intention-to-treat
attrition, crossovers, followup: analysis No Post- Outcomes
Author, Year Funding source Risk of bias
adherence, and differential/ randomization Prespecified
contamination high exclusions
Afilalo, et al., Unclear No No No Yes Yes Moderate
66
2007
Lang, et al.,
65
2006
Willis, et al., Yes Yes Yes Yes Yes Agency for Moderate
67 Healthcare
2013
Research and
Quality
I-2
Table I-2. Quality assessments of cohort, case-control, and other observational studies
Did the design and Was the timing
analyses account Were non-biased and and/or time period
Are the for important Were valid and reliable valid ascertainment for the
comparison Were the inclusion and potential measures used? methods used? measurement of the
groups or time exclusion criteria confounding and (inclusion/exclusion, (inclusion/exclusion, intervention and
periods specified and applied modifying variables confounding, confounding, outcomes
Author, Year appropriate? equally to each group? appropriately? outcomes) outcomes) appropriate?
40 Yes Yes Yes Yes NR Yes
Bailey, et al., 2013
39 Yes Yes Yes Yes NR Yes

Ben-Assuli, Shabtai, Yes NR Yes Yes NR Yes

41
and Leshno, 2013
Ben-Assuli, Shabtai, Yes Yes Yes Yes NR Yes

72
and Leshno, 2015
70
Carr, et al., 2014 No No Unclear NR Yes Yes
Dixon, McGowan, Yes NA NA Yes Yes NA
42
and Grannis, 2011
Feldman and Horan Yes No No Yes Yes Yes

43
2011
44
Frisse, et al., 2012 Yes Yes Yes Yes NR Yes
Jones, Friedberg Yes Yes Yes Yes Yes Yes

and Schneider,
68
2011
45
Kern et al., 2012 Yes Yes Yes Yes Yes Yes
88
Kho et al., 2013 NA Yes No Yes Yes Yes
Lammers, Adler- Yes Yes Yes Yes Yes Yes
Milstein, and
69
Kocher, 2014
Lobach, et al., NA NA NA Yes NA Yes
100
2007
Magnus, et al., Yes Yes Yes Yes Yes Yes
47
2012
I-3
Was there no Were outcomes
missing data? If prespecified and
missing data, was it were prespecified
handled outcomes
Author, Year appropriately? reported? Risk of bias
40 Yes Yes Low
39 NR Yes Low
Ben-Assuli, Shabtai, Yes Yes Low

41
and Leshno, 2013
Ben-Assuli, Shabtai, Yes Yes Low

72
and Leshno, 2015
70
Carr, et al., 2014 Yes Yes Moderate
Dixon, McGowan, NA Yes Low
42
and Grannis, 2011
Feldman and Horan No Yes Moderate

43
2011
44
Frisse, et al., 2012 Yes Yes Moderate
Jones, Friedberg Yes Yes Low

and Schneider,
68
2011
45
Kern et al., 2012 No Yes Low
88
Kho et al., 2013 Yes NA Low
Lammers, Adler- Yes Yes Low
Milstein, and
69
Kocher, 2014
Lobach, et al., Unclear NA Low
100
2007
Magnus, et al., NR Yes Low
47
2012
I-4
for important valid ascertainment for the
Are the Were valid and reliable
potential methods used? measurement of the
comparison Were the inclusion and measures used?
confounding and (inclusion/exclusion, intervention and
groups or time exclusion criteria (inclusion/exclusion,
modifying variables confounding, outcomes
periods specified and applied confounding,
Author, Year appropriately? outcomes) appropriate?
appropriate? equally to each group? outcomes)
Mäenpää, et al., Yes Yes Yes Yes Yes Yes
115
2011
McCarthy, et al., Unclear Yes No No No Yes

161
2014
McGowan, et al., No No No No No No
148
2007
Miller and Tucker, Yes Yes No Yes No Yes

149
2014
Moore, et al., No comparison Yes No Yes No Yes

106 group
2012
Nagykaldi, et al., Yes Unclear Unclear Unclear Unclear Yes

48
2014
Onyile, et al., Yes Yes NA Yes Yes Yes
125
2013
Overhage, Grannis, Yes Yes Yes Yes Yes Yes
and McDonald,
49
2008
50
Ross, et al., 2013 Yes Yes Yes Yes Yes Yes
154 NA NA No Uncertain Unclear Yes

Saff, et al., 2010
Shapiro, et al., Yes Yes Yes Yes Yes Yes

51
2013
Silvester and Carr, Yes Yes No Yes Yes Yes
114
2009
I-5
handled outcomes
Author, Year Risk of bias
appropriately? reported?
Mäenpää, et al., Yes Yes Low
115
2011
McCarthy, et al., NA No Moderate

161
2014
McGowan, et al., Unclear No High

148
2007
Miller and Tucker, Potentially missing Yes Moderate

149 data handled to best
2014
of their ability
Moore, et al., Yes Yes Moderate

106
2012
Nagykaldi, et al., NR Yes Moderate

48
2014
Onyile, et al., Not clear NA Low
125
2013
Overhage, Grannis, Yes Yes Low
and McDonald,
49
2008
50
Ross, et al., 2013 Unclear Yes Low
154 Unclear No High

Saff, et al., 2010
Shapiro, et al., Yes Yes Moderate

51
2013
Silvester and Carr, Yes Yes High
114
2009
I-6
for important valid ascertainment for the
Are the Were valid and reliable
potential methods used? measurement of the
comparison Were the inclusion and measures used?
confounding and (inclusion/exclusion, intervention and
groups or time exclusion criteria (inclusion/exclusion,
modifying variables confounding, outcomes
periods specified and applied confounding,
Author, Year appropriately? outcomes) appropriate?
appropriate? equally to each group? outcomes)
Tzeel, Lawnicki, and Yes Yes Yes Yes Unclear Yes
53
Pemble, 2012
Tzeel, Lawnicki, and Yes Yes Yes Yes Unclear Yes

52
Pemble, 2011
54
Vest, 2009 Yes Yes Yes Yes Yes Yes
155
Vest, 2010 Yes Yes Yes Yes Yes Yes
104 Yes Yes Yes Yes Yes Yes
Vest, et al., 2011

Vest, et al., 2011
Vest and Miller, Yes Yes Yes Yes (no information on Yes (Data are from Yes
64 survey reporting) multiple surveys)
2011
Vest, et al., 2012
Vest and Jasperson, No comparison Yes Yes Yes Yes Yes

103 group; time period
2012
appropriate
102 Unclear Yes No Yes No No
Vest, et al., 2013
55
Vest, et al., 2014 Yes Yes Yes Yes Yes Yes
56
Vest, et al., 2014 Yes Yes No Yes Yes Yes
71
Winden, 2014 No Yes No Unclear Unclear Yes
I-7
handled outcomes
appropriately? reported?
Tzeel, Lawnicki, and Yes Yes Low
53
Pemble, 2012
Tzeel, Lawnicki, and Yes Yes Low

52
Pemble, 2011
54
Vest, 2009 Yes Yes Low
155
Vest, 2010 Yes Yes Low
104 Not clear Yes Low
Vest, et al., 2011

Vest, et al., 2011
Vest and Miller, Unclear Yes Low

64
2011
101 Unclear Yes Low
Vest, et al., 2012
Vest and Jasperson, Yes Yes Low

103
2012

Vest, et al., 2013
55
Vest, et al., 2014 Not clear Yes Low
56
Vest, et al., 2014 Not clear Yes Low
71
Winden, 2014 No Yes Moderate
NA= not applicable; NR = not relevant.
I-8
Table I-3. Quality assessment of surveys, focus groups, and interview studies
1. Is the sampling
strategy or 3. Are characteristics (e.g.,
selection criteria 2. Are the response or participation demographics) of 4. Is how the questions were
reported and rates reported and are they respondents/participants developed/selected reported and is it
Author, Year appropriate? acceptable given the type of study? reported? appropriate?
Abramson, et al., Yes Yes, 72% Yes, hospitals in New York Yes
76 State
2012
Abramson, et al., Yes 59.3% (375/632) response rate Yes, nursing homes in New Yes
77 York State
2014
Abramson, et al., Yes Yes Yes Yes
96
2014
Adler-Milstein, et al., Yes Yes, 60% Yes
81
2008
Adler-Milstein, Bates, Yes Yes, 78% Yes, operational RHIOs Yes, pilot testing
78
and Jha, 2009
Adler-Milstein, Yes Yes, 83% Yes, operational RHIOs Yes

Landefeld, and Jha,
80
2010
Adler-Milstein, Bates, Yes Yes, 84% Yes, operational RHIOs Yes
79
and Jha, 2011
Adler-Milstein, Yes Yes - 69% Yes Yes
DesRoches, and Jha,
107
2011
Adler-Milstein, Bates, Yes Yes, 78% Yes, operational RHIOs Yes, pilot testing
25
and Jha, 2013
Adler-Milstein and Jha, Yes Yes Yes Yes
108
2014
Altman, et al., Unclear; Yes, 70% (14/20) Yes Yes
57 convenience sample
2012
Audet, Squires, and Yes Yes, 35% Yes Yes
109
Doty, 2014
Caffrey and Park- Lee Yes Yes Yes Yes
93
2013
I-9
5. Were confounders
considered? (could be in 6. Is analysis appropriate?
analysis or presentation, (given the type of data)
such as stratifying
results)
Abramson, et al., Unclear Yes Low
76
2012
Abramson, et al., Unclear Yes Low
77
2014
Abramson, et al., Yes Yes Moderate
96
2014
Adler-Milstein, et al., Unclear Yes Low
81
2008
Adler-Milstein, Bates, and Unclear Yes Low
78
Jha, 2009
Adler-Milstein, Landefeld, Unclear Yes Low
80
and Jha, 2010

79
Jha, 2011
Adler-Milstein, DesRoches, Yes Yes Low
107
and Jha, 2011

25
Jha, 2013
Adler-Milstein and Jha, Unclear Yes Low

108
2014
Altman, et al., NA, descriptive Mostly descriptive results Moderate

57 interviews presented
2012
Audet, Squires, and Doty, Unclear Yes Low

109
2014
Caffrey and Park- Lee Yes Yes Low

93
2013
I-10
1. Is the sampling 2. Are the response or 3. Are characteristics
strategy or selection participation rates reported (e.g., demographics) of 4. Is how the questions were
criteria reported and and are they acceptable respondents/participan developed/selected reported and is it
Author, Year appropriate? given the type of study? ts reported? appropriate?
Campion, et al., Yes Yes (19%) Yes Yes
58
2012
Codagnone, Yes Yes Yes Yes
Lupiañez-
94
Villanueva 2013
Chang, et al., No No, 9 primary care Yes yes
59 physicians selected for
2010
Dixon, Miller, and Yes Yes Yes Yes
141
Overhage, 2013
Dixon, Jones, and Yes Yes, 69% (44/63) "Infection preventionists" Yes, pilot administration with
83 modification of survey
Grannis, 2013
Fairbrother, et al., Yes NR but these were interviews Yes NR

143
2014
Finnell and Yes Yes, 32% response rate Yes Unclear. Survey not well described.
Overhage,
131
2010
84 Unclear-basically Yes No NR - survey URL broken
Foldy, 2007
asked experts whom to
ask
Fontaine, et al., Yes NR NR Yes
85
2010
Furukawa, Yes Yes Yes Unclear
110
2 014
Furukawa, Yes Yes No Yes
111
2013
Gadd, et al., Yes Yes, email survey responses Yes Yes
86 from with 70% response rate
2011
from health care
professionals (165/237).
Genes, et al., Yes Yes, 18/22 participated in Yes Yes
2011
145 interviews
Goldwater, et al., Yes Yes for interviews. 20% No NR
146 response to emailed survey.
2014
I-11
5. Were confounders
considered? (could 6. Is analysis
be in analysis or appropriate? (given
presentation, such as the type of data)
stratifying results)
Campion, et al., Yes Yes Moderate
58
2012
Codagnone, Yes Yes Low
Lupiañez- Villanueva
94
2013
Chang, et al., No, descriptive only Yes Moderate
59
2010
Dixon, Miller, and No Yes Moderate
141
Overhage, 2013
Dixon, Jones, and Unclear Yes Moderate

83
Grannis, 2013
Fairbrother, et al., NA NR High

143
2014
Finnell and NA Yes, descriptive only Moderate
131
Overhage, 2010
84 No Yes Moderate
Foldy, 2007
Fontaine, et al., No Yes Moderate
85
2010
110 Yes Yes Low
Furukawa, 2014
111 Yes Yes Low

Furukawa, 2013
86
Gadd, et al., 2011 Yes Yes Low
145
Genes, et al., 2011 NA Yes Low
146
Goldwater, et al., 2014 NA Yes, descriptive only Moderate
I-12
1. Is the sampling 2. Are the response or 3. Are characteristics (e.g.,
strategy or selection participation rates reported and demographics) of 4. Is how the questions were
criteria reported and are they acceptable given the type respondents/participants developed/selected reported and is it
Author, Year
appropriate? of study? reported? appropriate?
Greenhalgh, et Yes No No Yes
121
al., 2010
Hamann and Yes Yes NA Yes
Bezboruah,
113
2013
Hessler, et al., Yes No Yes Yes
87
2009
Hincapie, et al., Yes Yes No, no table of participants. Yes
132 Types of providers were
2011
mentioned with qualitative
themes.
Hyppönen, et al., Yes Yes Yes Yes
133
2014
Jha, et al., Yes No No No
117
2008
Kaushal, et al., No Yes No Yes
60
2010
Kern, et al., Yes Yes Yes Yes
173
2009
Kern, et al., No Yes No Yes
171
2011
Kierkegaard, Yes NA Yes, characteristics of sites NR
Kaushal, and reported and types of HIE
127 users are described but not
Vest, 2014
quantified.
Lee, et al., Unclear (post given to No (rate given but low; only collected Yes Yes
89 all, for pre this is for 2 weeks)
2012
unclear)
Machan, Yes, questionnaire Yes, 43% (104/242) practitioners Yes, physician users of HIE. Yes, development process for
Ammenwerth, sent to all practitioners responded. interviews guide and questionnaire
and registered in HIE described thoroughly. No
Schabetsberger, project. psychometrics presented.
62
2006
I-13
5. Were confounders
considered? (could be 6. Is analysis
in analysis or appropriate? (given the
Author, Year presentation, such as type of data) Risk of bias
Greenhalgh, et Yes Yes Low
121
al., 2010
Hamann and Yes Yes Low
Bezboruah,
113
2013
Hessler, et al., No No High
87
2009
Hincapie, et al., NA Yes Moderate
132
2011
Hyppönen, et al., Yes Yes Low

133
2014
Jha, et al., No Unclear High
117
2008
Kaushal, et al., No Yes High
60
2010
Kern, et al., No Yes Moderate
173
2009
Kern, et al., No No High
171
2011
Kierkegaard, NA Yes, coded interviews Moderate
Kaushal, and with Nvivo
127
Vest, 2014
Lee, et al., Yes No High

89
2012
Machan, No, only descriptive Yes, descriptive analysis Low

Ammenwerth, analysis only.
and
Schabetsberger,
62
2006
I-14
Author, Year
Massy- Yes, convenience Reported as 42% (55/80) but this No No
Westropp, et al., sample of 82 users of doesn’t account for 50 controls so
134 HIE and then the response rate is 24% (55/132).
2005
additional sample of
50 providers not in HIE
program as controls.
Maass, et al., Yes, only 1 person NR NR NR

61 interviewed
2008
McCullough, et Yes, used purposive Yes, reported recruitment rate of Yes Yes
135 sample strategy practices.
al., 2014
Merrill, et al., Yes Yes Yes Yes

174
2013
Messer, et al., Yes, interviews and NR, it is not clear how many surveys No Yes
138 assessment with were sent out to compute a response
2012
39 stakeholders; pre rate.
and post survey of 29
providers' satisfaction
with HIE, relationships
with other providers,
barriers.
162 Yes NR, but these were interviews NR Yes, questions developed jointly by the
Miller, 2012
University of California, San Francisco,
and Consumers Union
Myers, et al., Yes, used purposive Yes, 62/102 emailed invitations to Yes for key respondents. No Yes, developed after literature review.
128 sample strategy survey for survey. Reported Chronbach alphas of .57-.97
2012
for scaled items.
I-15
5. Were confounders
presentation, such as type of data)
Massy- No NA High
Westropp, et al.,
134
2005
Maass, et al., No Yes High

61
2008
McCullough, et NA Yes Low
135
al., 2014
Merrill, et al., Yes Yes Low

174
2013
Messer, et al., NA Yes, for qualitative and Moderate
138 quantitative.
2012
162 NA Yes Moderate

Miller, 2012
Myers, et al., Stratified by role Yes Low

128
2012
I-16
1. Is the sampling 2. Are the response or participation 3. Are characteristics (e.g.,
strategy or selection rates reported and are they demographics) of 4. Is how the questions were
criteria reported and acceptable given the type of study? respondents/participants developed/selected reported
Author, Year
appropriate? reported? and is it appropriate?
139
Nøhr, et al., 2001 Yes Yes Yes NR/Yes
Nykänen and Karimaa, Yes Yes No Yes

150
2006
Ozkaynak and Brennan, Yes NA Yes NR

129
2013
Pagliari, Gilmour, and Yes Yes No No

122
Sullivan, 2004
63
Patel, et al., 2013 Yes Yes Yes Yes
63
Park, et al., 2013
Yes Yes Yes Yes
Phillips, et al., Yes NA No Yes
164
2014
Pirnejad, Bal, and Yes for RN surveys; No Yes Yes Yes for surveys - published
152 for interviews surveys used to identify
Berg, 2008
questions; No for interviews
Ross, et al., Yes Yes No Yes
167
2010
Rudin, et al., Yes Not reported No Yes
153
2009
Rudin, et al., Yes NR Yes Yes/Yes
136
2011
Schoen, et al., Yes Yes No No

95
2012
168 Yes Yes NR Yes
Sicotte and Paré, 2010
Steward, et al., Yes NR but these were interviews NR Partnered with UCSF
169 qualitative experts to
2012
I-17
5. Were confounders
considered? (could be in 6. Is analysis
analysis or presentation, appropriate? (given
such as stratifying the type of data)
results)
139
Nøhr, et al., 2001 NA Yes Moderate
Nykänen and No Yes Moderate

150
Karimaa, 2006
Ozkaynak and NA Yes Moderate

129
Brennan, 2013
Pagliari, Gilmour, and No Yes Moderate

122
Sullivan, 2004
63
Patel, et al., 2013 Yes Yes Low
63
Park, et al., 2013
No Yes Low
Phillips, et al., Yes Yes Low
164
2014
Pirnejad, Bal, and No Yes Moderate
152
Berg, 2008
Ross, et al., No Yes Moderate

167
2010
Rudin, et al., No Yes Moderate
153
2009
Rudin, et al., NA Yes Low
136
2011
95
Schoen, et al., 2012 No Yes High
Sicotte and Paré, Yes Yes Low

168
2010
Steward, et al., NA Yes Moderate
169
2012
I-18
Author, Year
Soderberg and Yes Yes No Unclear
Laventure,
90
2013
Thorn, Carter, Yes, used purposive Yes, mentioned all physicians agreed Yes Types of questions mentioned but no
and Bailey, sample strategy to participate and no one dropped mention of interview guide.
130 out.
2014
Unertl, et al., Yes Yes NR Yes
170
2013
Unertl, Johnson, Yes NA Yes. Characteristics of sites NR in main text but mentioned online
and Lorenzi, and interviewees described. appendix but no link to access it.
119
2012
Yeager, et al., Yes Yes Yes No/Yes. Types of questions mentioned

137 but no mention of interview guide.
2014
I-19
5. Were confounders
presentation, such as type of data)
Soderberg and Yes Yes Moderate
Laventure,
90
2013
Thorn, Carter, NA Yes Low
and Bailey,
130
2014
Unertl, et al., Yes Yes Low
170
2013
Unertl, Johnson, NA Yes. Coded interviews Moderate

and Lorenzi, with Nvivo
119
2012
Yeager, et al., NA Yes Moderate

137
2014
HIE= health information exchange; NA= not applicable; NR= not relevant; RHIO= regional health information organization;
RN= registered nurse; UCSF= University of California, San Francisco; URL= uniform resource locator.
I-20

Ahrq 15 Hie PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ahrq 15 Hie PDF

Uploaded by

Copyright:

Available Formats

Evidence Report/Technology Assessment

Evidence-Based Health Information

Health Information Exchange

Contract No. 290-2012-00014-I

AHRQ Publication No. 15(16)-E002-EF

Richard G. Kronick, Ph.D. Arlene S. Bierman, M.D., M.S.

Stephanie Chang, M.D., M.P.H. Edwin Lomotan, M.D.

Julia Adler-Milstein, Ph.D. Deven McGraw, J.D., M.P.H.

Gilad Kuperman, M.D., Ph.D.

Technical Expert Panel

Julia Adler-Milstein, Ph.D. Melinda Buntin, Ph.D.

*This TEP member also provided review of the draft report.

Jessica S. Ancker, Ph.D., M.P.H. Christopher Chute, M.D., D.P.H.

Ann McKibbon, B.Sc., M.L.S., Ph.D.

Results. We included 136 studies overall, with 34 on effectiveness, 26 of which reported

Table A. Summary of evidence

Factors related to sustainability

Use of Health Information Exchange

Usability and Other Barriers and Facilitators to Use

Implementation and Sustainability

Scope of Review and Key Questions

Key Question 6. What facilitators and barriers impact use of HIE?

Key Question 7. What facilitators and barriers impact implementation of HIE?

Key Question 8. What factors influence sustainability of HIE?

Topic Development and Refinement

Literature Search Strategy

Process for Study Selection

Intervention and Comparators

Outcomes by Key Question

Data Abstraction and Data Management

Assessment of Methodological Risk of Bias of Individual

Data Synthesis and Organization of Report

Grading the Body of Evidence for Each Key Question

Peer Review and Public Commentary

Figure 2. Literature flow diagram

HIE = health information exchange; KQ= Key Question

Description of Included Studies

Key Question 1. Is HIE effective in improving clinical, economic, and

Improving Resource Use

Referrals and Consultations

Public Heath Reporting

Other Aspects of HIE

Table 3. Factors that may affect outcomes

Level of Use and Primary Uses: Type of HIE

Participation in HIE, Types of Data Exchanged, Characteristics of

Key findings: 106 had reached stage 6 (sustaining)

Most commonly exchanged data were radiology

-Summary of patient interviews found general

-Challenges: concerns about data ownership

Frequency of laboratory test and imaging increased.

The higher the numbers of emergency visits and

Clinicians most common users of reporting/ referrals;

In the U.S. capacity for electronic exchange of

Key Question 6. What facilitators and barriers impact use of HIE?

Usability of HIE by Type

Facilitators and Barriers Impacting HIE Use

Inefficient Workflow Byrne, et al., Thoughtful implementation Byrne, et al., 2014116

HIE = Health information exchange; vs. = versus

Addressing Lack of Critical Mass

Addressing Inefficient Workflow in Electronically Exchanging Data

Addressing Poorly Designed Interface and Update Features

Key Question 7. What facilitators and barriers impact implementation of

General Structural Characteristics