Enabling Health and Healthcare Through Ict

ENABLING HEALTH AND HEALTHCARE
THROUGH ICT
Studies in Health Technology and
Informatics
This book series was started in 1990 to promote research conducted under the auspices of the EC
programmes’ Advanced Informatics in Medicine (AIM) and Biomedical and Health Research
(BHR) bioengineering branch. A driving aspect of international health informatics is that
telecommunication technology, rehabilitative technology, intelligent home technology and many
other components are moving together and form one integrated world of information and
communication media. The series has been accepted by MEDLINE/PubMed, SciVerse Scopus,
EMCare, Book Citation Index – Science and Thomson Reuters’ Conference Proceedings Citation
Index.
Series Editors:
Dr. O. Bodenreider, Dr. J.P. Christensen, Prof. G. de Moor, Prof. A. Famili, Dr. U. Fors,
Prof. A. Hasman, Prof. E.J.S. Hovenga, Prof. L. Hunter, Dr. I. Iakovidis, Dr. Z. Kolitsi,
Mr. O. Le Dour, Dr. A. Lymberis, Prof. J. Mantas, Prof. M.A. Musen, Prof. P.F. Niederer,
Prof. A. Pedotti, Prof. O. Rienhoff, Prof. F.H. Roger France, Dr. N. Rossing,
Prof. N. Saranummi, Dr. E.R. Siegel, Prof. T. Solomonides and Dr. P. Wilson
Volume 183
Recently published in this series
Vol. 182. A.C. Smith, N.R. Armfield and R.H. Eikelboom (Eds.), Global Telehealth 2012 –
Delivering Quality Healthcare Anywhere Through Telehealth – Selected Papers from
Global Telehealth 2012 (GT2012)
Vol. 181. B.K. Wiederhold and G. Riva (Eds.), Annual Review of Cybertherapy and
Telemedicine 2012 – Advanced Technologies in the Behavioral, Social and
Neurosciences
Vol. 180. J. Mantas, S.K. Andersen, M.C. Mazzoleni, B. Blobel, S. Quaglini and A. Moen
(Eds.), Quality of Life through Quality of Information – Proceedings of MIE2012
Vol. 179. M. García-Rojo, B. Blobel and A. Laurinavicius (Eds.), Perspectives on Digital
Pathology – Results of the COST Action IC0604 EURO-TELEPATH
Vol. 178. A.J. Maeder and F.J. Martin-Sanchez (Eds.), Health Informatics: Building a
Healthcare Future Through Trusted Information – Selected Papers from the 20th
Australian National Health Informatics Conference (HIC 2012)
Vol. 177. B. Blobel, P. Pharow and F. Sousa (Eds.), pHealth 2012 – Proceedings of the 9th
International Conference on Wearable Micro and Nano Technologies for Personalized
Health
Vol. 176. T. Kotwicki and T.B. Grivas (Eds.), Research into Spinal Deformities 8
Vol. 175. S. Gesing, T. Glatard, J. Krüger, S.D. Olabarriaga, T. Solomonides, J.C. Silverstein,
J. Montagnat, A. Gaignard and D. Krefting (Eds.), HealthGrid Applications and
Technologies Meet Science Gateways for Life Sciences
ISSN 0926-9630 (print)

ISSN 1879-8365 (online)
Enablin
ng Heaalth an
nd Heaalthcarre
thro
ough ICT
I
Available, Tailored and
a Closerr
Edited by
y
Karen
n L. Cou
urtney
Omiid Shabeestari
and
A Kuo
Alex o
S
School of Heealth Informa ation Sciencee
Univversity of Vicctoria
Victoria, Brritish Columbia, Canada a
Amstterdam • Berrlin • Tokyo • Washington, DC

© 2013 ITCH 2013 Steering Committee and IOS Press.
All rights reserved. No part of this book may be reproduced, stored in a retrieval system,
or transmitted, in any form or by any means, without prior written permission from the publisher.
ISBN 978-1-61499-202-8 (print)

ISBN 978-1-61499-203-5 (online)
Library of Congress Control Number: 2013930665
Publisher
IOS Press BV
Nieuwe Hemweg 6B
1013 BG Amsterdam
Netherlands
fax: +31 20 687 0019
e-mail: [email protected]
Distributor in the USA and Canada

IOS Press, Inc.
4502 Rachael Manor Drive
Fairfax, VA 22032
USA
fax: +1 703 323 3668
e-mail: [email protected]
LEGAL NOTICE
The publisher is not responsible for the use which might be made of the following information.
PRINTED IN THE NETHERLANDS

Enabling Health and Healthcare through ICT v
K.L. Courtney et al. (Eds.)
IOS Press, 2013
© 2013 ITCH 2013 Steering Committee and IOS Press. All rights reserved.
Preface
Information and communication technologies (ICT) can enable the delivery of
healthcare in ways unimaginable in a paper-bound world. Internationally, the use of
ICT in healthcare is growing rapidly as the need to provide efficient and effective care
becomes more urgent to patients, providers and payors alike. Research in new models
of healthcare delivery is starting to examine how ICT can increase the effectiveness
and efficiency of healthcare delivery.
Universal health care is still not the norm for many people in developing and de-
veloped countries alike. Socio-economic status and geography have been substantial
barriers to accessible care. Leveraging the diversity and ubiquity of available ICT may
be one of the keys in enabling accessible healthcare across geographic and socio-
economic divides.
Improvement in the health of nations begins with the individual. Recent develop-
ments in genomics and mobile networked information technologies have re-generated
interest in individualizing health care to leverage current knowledge for the greatest
individual benefit. In the not-so-distant past, it seemed unachievable to collect, store,
analyze and share information and advice tailored to a specific individual. At best, in
the past, clinical decision support systems made recommendations to clinicians about
how to apply population health probabilities based on only a few individual character-
istics; usually age and gender. Now researchers are testing personal decision support
systems, which provide recommendations directly to the individual based on a myriad
of characteristics, symptoms, signs and personal preferences. The inclusion of personal
genomic information as part of this cluster of characteristics is in the near future.
Governments, health systems and other payors are beginning to recognize not only
the subjective value of individualized care provided at a location of the patient’s choice,
but also the potential economic benefits as well. Globally, evidence is being gathered
that supports health care delivery models that embrace co-management of health be-
haviors between health care providers and patients and encourage “localized” health
care delivery. In this context, local care might take place in a community clinic, in the
home or even in the pocket, as telehealth takes healthcare mobile. The term, “house
calls”, may once again become familiar in 21st century healthcare, albeit with a differ-
ent implementation.
We hope that the knowledge shared between ITCH 2013 participants will generate
further discussions and collaborations and lead to breakthroughs in delivering effective
and efficient healthcare worldwide.
Karen Courtney
School of Health Information Science
University of Victoria
Victoria, British Columbia
Canada
This page intentionally left blank
vii
ITCH 2013 Steering Committee

Conference Chair
Abdul Roudsari
Program Co-Chairs
Elizabeth Borycki, Andre Kushniruk, Abdul Roudsari and Scott Macdonald
Editorial Committee
Karen Courtney, Alex Kuo and Omid Shabestari
Workshop Coordinator
Omid Shabestari
Multi Media Production

John Bartle-Clar and James McDaniel
Local Organizing Committee

John Bartle-Clar, Shawna McNabb, Omid Shabestari and Leslie Wood
ix
ITCH 2013 Scientific Review Committee

Raza Abidi
Dalhousie University
Halifax, NS Canada
James Anderson
Purdue University
West Lafayette, IN USA
Riccardo Bellazzi
University of Pavia
Pavia, Italy
Joseph Cafazzo
University Health Network
Toronto, ON Canada
Dominic Covvey
National Institutes of Health Informatics
Waterloo, ON Canada
Eric Eisenstein
Duke Clinical Research Institute
Durham, NC USA
John Fox
Oxford University
Oxford, UK
Marilynne Hebert
University of Calgary
Calgary, AB Canada
Don Juzwishin
Alberta Health Services
Edmonton, AB Canada
Jonathan Kay
Oxford John Radcliffe Hospital
Oxford, UK
Craig Kuziemsky
University of Ottawa
Ottawa, ON Canada
x
James McDaniel
Victoria, BC Canada
Christian Nohr
Aalborg University
Aalborg, Denmark
Grace Paterson
Dalhousie University
Halifax, NS Canada
Stephen Rees
Aalborg University
Aalborg, Denmark
Paul Taylor
UCL Centre of Health Informatics
London, UK
Maggie Wang
University of Hong Kong
Hong Kong
Jens Weber
Victoria, BC Canada
Jennifer Zelmer
Canada Health Infoway
Toronto, ON Canada
xi
Contents
Preface v
Karen Courtney
ITCH 2013 Steering Committee vii
ITCH 2013 Scientific Review Committee ix
Electronic Health Records
Clinician-Led Development of Electronic Health Records Systems 3

John Chelsom, Raju Ahluwalia and Naveed Dogar
Implications of the Mature Personal Health Record for the Empowered
Consumer 9
Wesley Rohrer and Mowafa S. Househ
Applying the Clinical Adoption Framework to Evaluate the Impact of an
Ambulatory Electronic Medical Record 15
Francis Lau, Colin Partridge, Gurprit Randhawa and Mike Bowen
Role of Healthcare Information Technology in Handoffs 21
Ross Koppel and Joel Leon Telles
Computerisation in General Practice: Lessons for Canada from the UK and
Australia 28
Nicola T. Shaw and Mike Bainbridge
Challenges in Data Quality Assurance for Electronic Health Records 37
Omid Shabestari and Abdul Roudsari
A Systemic Approach to E.H.R. Implementation (Abstract) 42
Bogdan Motoc
A Scoping Review on Health Records for Child-in-Care 43
Cori Thompson and Francis Y. Lau
Application of the Technological Pedagogical Content Knowledge Framework
in Integrating an Educational EMR into Health Informatics Education 49
Jesdeep Bassi, Andre W. Kushniruk and Elizabeth M. Borycki
Telecare, Telemedicine and Telelaboratory
TeMaD System: Telecare for Managing Diabetes in Saudi Arabia 57

Khulud Alkadi and Abdul Roudsari
Telecardiology on Vancouver Island: Imagination to Implementation 63
Trent Horwood, Mya Aylott, Margarita Loyola, Jill Henderson,
Gary Frank and Mary Lyn Fyfe
xii
Developing a Strategy for Studying Critical Thinking in a Nurse Telehealth

Setting: A Participatory Approach 68
Danica Tuden, Hanne Gidora, Peter D. Quick, Nikki Ebdon, Karen Glover,
Sherrill Harmer, Wendy Miller, Mary Taylor and Elizabeth M. Borycki
A “Realist Review” Approach to e-Health: The Case of Type 2 Diabetes in
Youth 74
Marilynne A. Hebert
Designing the Community Multi-User Health Kiosk 79
Julie M. McMillan, Karen L. Courtney, Judith T. Matthews, Asim Smailagic
and Daniel Siewiorek
Public Health Informatics
Public Reporting of Hospital Infection Rates: Ranking the States on Credibility

and User Friendliness 87
Ava Amini, David W. Birnbaum, Bernard Black and David A. Hyman
Methodological Approaches to Comparing Information About Bicycle Accidents
Internationally: A Case Study Involving Canada and Germany 93
Christian Juhra, Britta Wieskötter, Paule Bellwood, Ariane von Below,
Murray Fyfe, Sonia Salkeld, Elizabeth Borycki and Andre Kushniruk
The Development of a Standardized Software Platform to Support Provincial
Population-Based Cancer Outcomes Units for Multiple Tumour Sites: OaSIS –
Outcomes and Surveillance Integration System 98
Jonn Wu, Cheryl Ho, Janessa Laskin, David Gavin, Paul Mak,
Keith Duncan, John French, Colleen McGahan, Sherry Reid,
Stephen Chia and Heidi Cheung
Clinical Decision Support Systems
Neuroanatomical Basis for Recognition Primed Decision Making 107

Darren Hudson
Use of Knowledge Discovery Techniques to Understand Nurse Practitioner
Practice Patterns and Their Integration into a Healthcare System 111
Esther Sangster-Gormley, Mu-Hsing Kuo, Elizabeth M. Borycki and
Rita Schreiber
Decision Support for Evidence-Based Pharmacotherapy Detects Adherence
Problems but Does Not Impact Medication Use 116
Janese M. Willis, Rex Edwards, Kevin J. Anstrom, Fred S. Johnson,
Guilherme Del Fiol, Kensaku Kawamoto, Nancy M. Allen Lapointe,
Eric L. Eisenstein and David F. Lobach
Analysis of Continuous Oxygen Saturation Data for Accurate Representation of
Retinal Exposure to Oxygen in the Preterm Infant 126
Josie Cirelli, Carolyn McGregor, Brenda Graydon and Andrew James
xiii
Tailoring Decision Support to Suit User Needs: A Diagnostic Imaging Example 132
Janessa Griffith, Elizabeth M. Borycki and Andre W. Kushniruk
Supporting Cystic Fibrosis with ICT 137
Erin Roehrer, Elizabeth Cummings, Paul Turner, Jenny Hauser,
Helen Cameron-Tucker, Sean A. Beggs, Nicole A. Micallef,
Claire Wainwright, Joyce Cheney, Mel Jessup, Heather Saddington,
Leonie Ellis, Haydn Walters and David W. Reid
Human Computer Interaction
Information Accountability and Usability: Are There Any Connections? 145

Tony Sahama, Andre Kushniruk and Shigeki Kuwata
A Review of Healthcare Information System Usability & Safety 151
Simon Minshall
Commercial Versus In-Situ Usability Testing of Healthcare Information Systems:
Towards “Public” Usability Testing in Healthcare Organizations 157
Andre W. Kushniruk, Elizabeth M. Borycki and Joseph Kannry
Integrating Human Factors in an International Research Project: Lessons
Learned from the PSIP Project 162
Romaric Marcilly, Marie-Catherine Beuscart-Zephir and Régis Beuscart
Usability Inspection to Improve an Electronic Provincial Medication Repository 168
Nicole A. Kitson, Morgan Price, Michael Bowen and Francis Lau
The Long and Twisting Path: An Efficiency Evaluation of an Electronic
Whiteboard System 174
Rasmus Rasmussen and Andre Kushniruk
Applying Usability Methods to Identify Health Literacy Issues: An Example
Using a Personal Health Record 179
Helen Monkman and Andre Kushniruk
Patient Safety, Medical Errors & Quality Management
Clinician Variations in Data Trust and Use 189

Karen L. Courtney, Yun Jiang, Scott R. Beach, Judith T. Matthews
and Richard Schulz
Design and Implementation of Synoptic Operative Report Template Using
Interoperable Standards 195
Wilfred Bonney, Sean Christie, Grace Paterson, Ginette Thibault-Halman
and Areej Alsulaiman
Processing Medical Reports to Automatically Populate Ontologies 201
Luís Borrego and Paulo Quaresma
xiv
Health Modeling
A Complex Adaptive Systems Perspective of Health Information Technology

Implementation 209
Karim Keshavjee, Craig Kuziemsky, Karim Vassanji and Ahmad Ghany
A Systems Theory Classification of EMR Hazards: Preliminary Results 214
Fieran Mason-Blakley and Jens Weber
Benefits of a Clinical Planning and Coordination Module: A Simulation Study 220
Sanne Jensen, Søren Vingtoft and Christian Nøhr
Mobile Technologies and Telehealth
Issues and Considerations for Healthcare Consumers Using Mobile Applications 227
Elizabeth Cummings, Elizabeth Marie Borycki and Erin Roehrer
The Introduction and Evaluation of Mobile Devices to Improve Access to
Patient Records: A Catalyst for Innovation and Collaboration at BCCA 232
Jonn Wu, John Waldron, Shaina Hood, Adam Kahnamelli, Mohamed Khan,
Jeff Barnett, John French, Stacey Slager, Shadi Melhem and Omid Shabestari
Moving Mobile: Using an Open-Sourced Framework to Enable a Web-Based
Health Application on Touch Devices 238
Joseph Lindsay, J. Allen McLean, Amrita Bains, Tom Ying and M.H. Kuo
The Use of Social Media in Healthcare: Organizational, Clinical, and Patient
Perspectives 244
Mowafa Househ
Evaluation
POND4Kids: A Global Web-Based Database for Pediatric Hematology and

Oncology Outcome Evaluation and Collaboration 251
Yuri Quintana, Aman N. Patel, Magada Arreola, Federico G. Antillon,
Raul C. Ribeiro and Scott C. Howard
Knowledge Translation in eHealth: Building a Virtual Community 257
Jesdeep Bassi, Francis Lau, Simon Hagens, Chad Leaver and Morgan Price
Complex Interventions in Healthcare and Health Informatics: A Scoping Review 263
Julie Kim
Consumer Informatics
Forumclínic: The Shaping of Virtual Communities to Assist Patients with

Chronic Diseases 271
Immaculada Grau, Francisco J. Grajales III, Joan Gene-Badia,
Antoni Siso and Marc de Semir
The PLU Problem: Are We Designing Personal eHealth for People Like Us? 276
Chris Showell and Paul Turner
xv
Islamic E-Health: Definitions, Applications, and Challenges 281

Mowafa Househ
A User-Centred Methodology for Designing an Online Social Network to
Motivate Health Behaviour Change 286
Noreen Kamal and Sidney Fels
Social Media and Patient Self-Management: Not All Sites Are Created Equal 291
Leonie Ellis, Chris Showell and Paul Turner
Empowering Saudi Patients: How Do Saudi Health Websites Compare to
International Health Websites? 296
Mowafa Househ, Abdulrahman Alsughayar and Maha Al-Mutairi
Taming Mental–Health-Focused Popular Literature: A Crazy Idea? 302
Robert T. Zozus Jr., Audrey Bricker, Rob Lunblad, Dennis Elias and
Meredith Nahm
Considerations for Personal Health Record Procurement 308
Helen Monkman and Andre Kushniruk
An Initial, Qualitative Investigation of Patient-Centered Education in Dentistry 314
Marc Clayton, Chuck Borromeo, Sherry Hess, Harry Hochheiser and
Titus Schleyer
Tailored Care Management with Patient-Centered Web-Based Portal in Primary
Health Care: Sustaining a Relational Context 319
James P. Ronan
Ontologies, Trust and Standards
The Social Act of Electronic Medication Prescribing 327

Jos Aarts
Improving Decision Quality in Healthcare with an Error Prevention Model
(EPM) 332
David T.S. Chio, Andre Kushniruk and Elizabeth Borycki
Software as a Medical Device: Regulatory Critical Issues 337
Sylvia Pelayo, Sabrina Bras Da Costa, Nicolas Leroy, Séverine Loiseau
and Marie-Catherine Beuscart-Zephir
Nursing Informatics
Developing National Level Informatics Competencies for Undergraduate

Nurses: Methodological Approaches from Australia and Canada 345
Elizabeth M. Borycki, Joanne Foster, Tony Sahama, Noreen Frisch and
Andre W. Kushniruk
What Nurses Are Talking About: Content and Community Within a Nursing
Online Forum 350
Kathleen Abrahamson, Rebekah Fox and James G. Anderson
xvi
A Framework for Leveling Informatics Content Across Four Years of a

Bachelor of Science in Nursing (BSN) Curriculum 356
Noreen Frisch and Elizabeth Borycki
Patient Safety Perspectives: The Impact of CPOE on Nursing Workflow 367
Mowafa Househ, Anwar Ahmad, Anwar Alshaikh and Fatimah Alsuweed
Healthcare Modeling and Simulation
Developing a Multivariate Electronic Medical Record Integration Model for

Primary Health Care 375
Francis Lau, Morgan Price and Mary Lesperance
Towards Assessing the Socio-Economic Impact of VPH Models (Abstract) 382
Rainer Thiel, Karl A. Stroetmann and Marco Viceconti
The Use of Discrete-Event Simulation Modeling to Compare Handwritten
and Electronic Prescribing Systems 383
Ahmad Ghany, Karim Vassanji, Craig Kuziemsky and Karim Keshavjee
A Model of Collaborative Agency and Common Ground 388
Craig E. Kuziemsky and Janet Alexandra Cornett
Subject Index 393

Author Index 397
Electronic Health Records
Enabling Health and Healthcare through ICT 3
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-3
Clinician-Led Development of Electronic

Health Records Systems
John CHELSOM a1, Raju AHLUWALIA b, and Naveed DOGAR a
a
Centre for Health Informatics, City University, London
b
NHS, N W Thames Orthopaedic Rotation
Abstract. The open source cityEHR system was evaluated as a toolkit for
clinician-led development of an Electronic Health Record for management of
patients in the Ponseti clinic of a major London hospital. As a toolkit, it was found
that the ontology-driven approach of cityEHR was too complex for clinicians to
use. The toolkit was refined to use more familiar spreadsheet s to represent the
ontology and was then used successfully to create an effective clinical system,
generated automatically from the information model.
Keywords. Electronic Health Records, Open Standards, Open Source, Clinician-

led Development
Introduction
The cityEHR electronic health records system has been developed as an integration of
'best of breed' open source software components, configured to support open standards
for clinical records such as ISO-13606 [1] and HL7 CDA [2]. It provides clinical users
with a toolkit that enables them to develop their own Electronic Health Records (EHR)
systems and then to deploy them on an enterprise-scalable platform. The system was
originally conceived to address the fundamental issues that caused the failure of the
EHR systems implemented as part of the National Programme for IT in England from
2004 to 2010 [3].
The platform for the deployed EHR is shown in Figure 1. It combines major open
source software components that run on an Enterprise Java platform and that are freely
available on the web: Apache Tomcat (tomcat.apache.org), eXist (exist-db.org), Mirth
Connect (www.mirthcorp.com), Orbeon (www.orbeon.com) and FusionCharts
(www.fusioncharts.com). These components are integrated and configured using the
XML-based languages XForms, XSLT and XQuery; there is no compiled code in the
cityEHR. The EHR deployed on this platform is configured entirely from an
information model represented as an ontology, in the Web Ontology Language (OWL)
[4]. This ontology uses a core architectural model (represented in OWL/XML syntax as
classes, object properties and data properties) to supply the building blocks for the
information model that contains specific individuals and property assertions.
1
Corresponding Author
4 J. Chelsom et al. / Clinician-Led Development of Electronic Health Records Systems
Figure 1. cityEHR Reference Platform of Open Source Software Components

This OWL information model is then transformed using XSLT into a s et of XML
configuration files that define how the (automatically) generated cityEHR system
functions for clinical users . The main XML representation in this configuration is the
XForms language for specification of clinical data entry forms, summary views and
letter templates. Behind the XML configuration of the user interface lies the model of
the patient data which uses the XML-based Clinical Document Architecture (CDA)
from HL7.
Thus there are four stages to the generation of patient data in the functional
cityEHR system: architecture, clinical information model, runtime configuration and
finally the stored data in the clinical system [5].
The purpose of this study was to evaluate the cityEHR toolkit by developing a
system for recording patient encounters at the Ponseti out-patients clinic at a major
hospital in London, UK. The main research question to be answered was : 'Can
clinicians develop their own information model that can be used to automatically
generate a functional EHR system?'
1. Requirements
The requirements for the Ponseti system were gathered without reference to the
capabilities of the cityEHR system or its modeling toolkit. A clinical analyst worked
with clinical staff (an orthopaedic surgeon and specialist clinic nurses ) to gather
requirements using structured interviews and analysis of existing paper-based data
records . The products of this analysis were a set of Use Cases, a System Requirements
Specification, a Data Dictionary and a set of User Interface Wireframes. The base
functional requirements (Table 1) were relatively simple and in our experience are
typical of the requirements for systems used in out-patient clinics and research studies .
J. Chelsom et al. / Clinician-Led Development of Electronic Health Records Systems 5
Table 1. Main Requirement s for the Electronic Health Records System in a Ponseti Clinic
Clinical Data Collection Other Requirements
Edit patient data systematically and store in a Secure authentication method to verify users
database
Clean, functional, form based user interface Basic role-based access control
Pre-defined datasets in a drop-down menu
Forms clearly sectioned into given headings Complex queries on the patient database
Attach/upload multimedia t o t he patient record Output result ant queries in a report
Conditional inclusion of sect ions/entries on forms Graphical or/and tabular format for reports
Enforce mandatory fields for data entry Display summary view(s) for existing records
Hints - short help assistance with data entry Generate standard letters (e.g. in MS word)
Hyperlinks to external web resources Query to assemble cohort s for research studies
Calculations and scoring on assessment forms Automatically calculate standard data (e.g. patient
age)
2. Implementation Using cityEHR
The basic functional requirements and user interface wireframes were used as the
starting point for configuration of the cityEHR. The data model elicited from clinical
users was coded as an ontology by the clinical analyst, using the open source
Protégé tooling from Stanford University [6]. Figure 2 shows the user interface of
Protégé with the basic architecture of the cityEHR represented as classes and the
specific in formation model for the Ponseti clinic system represented as individuals,
object property assertions and data property assertions.
The resulting OWL model was then loaded to the city EHR system and generated a
working EHR prototype which was compared with the original wireframes and
functional specification. It was found that the user functionality and general layout of
data entry forms could be created to the general satisfaction of users , but that a
number of specific functional requirements were not met by the automatically
generated system, specifically:
• complex calculations and s coring on assessment forms
• calculation and representation of patient age in days , weeks , months or years
depending on the age of the patient
• attach/upload multimedia to the patient record
These features were then added to the cityEHR system in a two stage process:
firstly adding the semantics to the cityEHR architectural ontology so that the features
could be specified in the information model and secondly adding functional support for
the features when the EHR system was generated from the model. Having added these
features to cityEHR it was then possible to generate a clinical EHR system that was
acceptable for piloting in the Ponseti clinic, as shown in Figure 3.
3. Spreadsheets for clinical users
Although it was possible for the clinical analyst to create an information model using
Protégé and use that model to generate a clinic system in the cityEHR that met the
functional requirements, there were some major issues with the usability of the toolkit.
Figure 2. Information Model as an Ontology, Edited Using Protégé

As the information model increased in size the clinical analyst struggled to create a
consistent model in a timely fashion; it proved impossible for clinical staff to use the
Protégé tooling themselves. The three main issues encountered were:
• The atomic assertions of the OWL language mean that even simple constructs
(e.g. the title of a section in a form) require two or three levels of abstraction
in the model; as the model grows , it becomes very difficult for a user to work
with this level of abs traction, even using a graphical tool such as Protégé
• Although reasoners can be run on the OWL model to check its consistency,
the language (and the Protégé tooling) does not provide easy mechanisms for
constraining user input so that cons is tent models are made in the first
instance.
• For clinical users, unfamiliar with the concepts of ontology and with the
Protégé tooling, the learning curve proved too steep to allow them to work
effectively within an already busy clinical schedule.
Figure 3. Electronic Health Records System Generated from Information Model

J. Chelsom et al. / Clinician-Led Development of Electronic Health Records Systems 7
Figure 4. OpenOffice Calc Spreadsheet Mirrors the OWL/XML cityEHR Architecture

To address these three issues, the toolkit was 'refaced' to use a standard spreadsheet
to specify the model. The spreadsheet follows the structure of ISO-13606 and mirrors
the ontology model but allows constraints to be placed on the data entered into various
fields. The spreadsheet can then be saved as XML and transformed using XSLT into
the OWL representation of the model. Figure 4 shows a sample of the spreadsheet
tooling that was developed using OpenOffice Calc (the open source equivalent of
Microsoft Excel).
4. Conclusions
Our study in the Ponseti clinic proved that it is possible to generate a usable EHR
system directly from an information model specified by clinical users. However, it also
showed that the complexity of the ontology-based approach (both conceptually and in
the available open source tooling) made it impossible for clinicians to develop that
model themselves directly using the cityEHR tools. Once a simpler spreadsheet-based
interface was introduced to specify the information model then it d id become possible
for clinical users to create and modify their information models directly.
In subsequent studies we have found that a variety of tooling can be used to create
the models (not just spreadsheets). This more general approach is shown in Figure 5.
The openness of the OWL/XML language, the widespread support of XML across
modeling tools and the capabilities of XSLT for trans formation combine to provide a
flexible approach to information modeling that can be used by clinical users to
automatically generate functional EHR systems using open source platforms such as
cityEHR.
Figure 5. Information Model in OWL/XML Transformed to any XML for Editing
References
[1] ISO 13606-1:2008 Health informatics - Electronic health record communication - Part 1: Reference
model. Available at:
http://www.iso.org/iso/iso_catalogue/catalogue_t c/cat alogue_det ail.ht m?csnumber=40784
[2] Dolin RH, Alschuler L, Boyer S et al. HL7 Clinical Document Architecture, Release 2. J Am Med
Inform Assoc. 2006;13(1):30-9.
[3] Chelsom, J. Open Health Informatics–A Fresh Approach for NHS IT. City University Challenge Paper
(2011). Available at: http://www.openhealthinformatics.org
[4] World Wide Web Consortium. OWL Web Ontology Language Overview. W3C Recommendation 10
February 2004. Available at: http://www.w3.org/T R/owl-features/
[5] J. J. Chelsom, I. Pande, R. Summers, I. Gaywood. Ontology-driven development of a clinical research
information system. 25th IEEE International Symposium on Computer-Based Medical Systems
(CBMS), pp. 1-4, 2011 24t h International Symposium on Computer-Based Medical Systems, 2011
[6] John H. Gennari , Mark A. Musen , Ray W. Fergerson , William E. Grosso , Monica Crubzy , Henrik
Eriksson , Natalya F. Noy , Samson W. Tu. The Evolution of Protégé: An Environment for Knowledge-
Based Systems Development (2002) International Journal of Human-Computer Studies.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-9
Implications of the Mature Personal Health

Record for the Empowered Consumer
Wesley ROHRERa,1, Mowafa S. HOUSEHb
a,b
College of Public Health and Health Informatics,
King Saud Bin Abdulaziz University for Health Sciences (KSAU-HS), National Guard
Health Affairs (NGHA), Riyadh, Saudi Arabia
Abstract. The purpose of this paper is to present a future scenario projecting a

fully developed, i.e., mature, personal health record (MPHR) that is instrumental
in linking health care services delivery with preventive and wellness activities to
provide informed and responsive consumer-driven health care. The legal-
regulatory, institutional management, socio-cultural and technological challenges
to implementation are discussed to guide future policy development in this area.
We conclude that the potential pay-offs for enhancing the individual’s health and
well-being and protecting and sustaining the public’s health overall is worth a
considerable investment in policy analysis, technological R&D, partnership
building and the expenditure of political capital by influential advocates and
decision makers.
Keywords. Personal health record, mature personal health record, health

information, challenges.
Introduction
The Personal Health Record (PHR) has become a point of focus for a number of the
trends characterizing medical informatics, health information technology and their
intersection with public (community) health in the early decades of the 21st Century.
“Meaningful use” of the EHR stipulations required by the American Recovery and
Reinvestment Act for reimbursement for providers under US Medicare and Medicaid
programs, patient-centered care, patient-provider partnerships, accountable care
organizations (ACOs) all can be seen as converging on the ultimate goals of the
empowered consumer of health care and preventive services. Parallel to these factors is
the priority given to translational research in medicine and public health to inform and
justify evidence-based clinical practice to insure patient safety and cost-effective
delivery of health care and community health interventions to eliminate health
disparities and sustain the public’s health and wellness at the local, regional and global
levels.
The purpose of this conceptual paper is to provide a future vision of a “mature”
PHR (referred to subsequently as the MPHR), identify the potential benefits of and
1
Corresponding Author: Dr. Wesley Rohrer, Associate Dean, King Saud Bin Abdul Aziz University for
Health Sciences, College of Public Health and Health Informatics, Riyadh, Kingdom of Saudi Arabia; Email:
[email protected]
10 W. Rohrer and M.S. Househ / Implications of the Mature PHR for the Empowered Consumer
challenges to MPHR implementation, and present an agenda for health policy research
relevant to MPHR development. By doing so, we wish to focus attention on the MPHR
as a useful focus for health system reform from the perspective of the empowered
consumer.
1. Defining the Personal Health Record
Although various but similar definitions of the Personal Health Record (PHR) have
been proposed, the following language is succinct and consistent with the HIPPA
regulatory logic. The US Office of Civil Rights defines the PHR as “an electronic
record of an individual’s health information by which the individual controls access to
the information and may have the ability to manage, track, and participate in his or her
own health care.”[1] The PHR as defined and discussed above is viewed not only as a
repository for Personal Health Information but more importantly as a channel and
mechanism for enhanced communication between the individual consumer of health
care and related services and the provider system.
2. Future Scenario for the MPHR
To suggest the potential capacities and benefits of the “mature” PHR (MPHR), we
provide the following brief scenario of the empowered health consumer using her
MPHR in 2020:
After checking her Outlook calendar in preparation for the work week ahead,
Maria realizes that she has a potential conflict with a mandatory staff meeting on
Monday at 11:00 and a previously scheduled routine follow-up visit with her PCP
three months after knee surgery. She accesses her PHR though her Apple
MyPhone/MyLife Pad. She places an instant message to the office of her PCP,
requesting a cancellation and new appointment on one of the available dates and
times highlighted in her on-line calendar. Within 10 minutes the confirmation of
the new appointment the following week is received along with a copy of an
inquiry to her physical therapist whether the range of motion data for Maria’s
knee is current. If not, Maria is asked to schedule a PT appointment prior to the
PCP visit. Also she is reminded to update her fitness and dietary plan with the
current week’s entries and to complete the six-month personal wellness survey.
Once she does so she will receive a table of her responses during the past three
years of key variables, e.g., weight, BM, HCL levels, blood pressure, pulse/OX
reading, mood (depression) scale, as compared to statistics for the health plan of
which to which she belongs and national benchmarks in the aggregate and by age,
gender and Regional cohort. If she had been interested Maria might have accessed
her cumulative use and cost of services history again with bench make
comparisons. Having requested a PT appointment later in the week, the PT office
Maria provides an on-line Pain Chart for her to complete on the touch screen, 24
hours prior to the confirmed appointment.
After her meeting Maria finished her workout at her employer’s fitness club.
During lunch she enters her weight training and aerobic exercise data and her
calorie count for the lunch she is enjoying, intending to trade off excess calories
W. Rohrer and M.S. Househ / Implications of the Mature PHR for the Empowered Consumer 11
for the slice of fruited cheesecake with a light dinner of fish and rice. While
completing her records, she receives an alert from the County Health Department
about a food-borne illness associated with a salad bar at a chain of family
restaurants in the Region with a request that anyone who dined at any of the
locations identified within the past 48 hours should complete an on-line survey
attached and if experiencing any severe gastric symptoms, contact the PCP or
receive expedited treatment at the closest ER. Finally a broadcast message in her
Healthy Community folder alerted subscribers of a current need for her blood type
at the Blood Bank and that first time donors would receive bonus points toward a
discount of their health plan premium payment .Next Maria logged into the MPHR
of her aging mother who had recently been diagnosed with early onset Alzheimer’s.
Having obtained power of attorney for all her mother’ affairs, Maria had complete
access to her PHR and the tethered EHR. She duly noted the alert that her mother
had missed her medication last evening, an issue that Maria intended to address
with the home health aide directly. Realizing that the time was drawing near when
her mother would require institutional long term care, Maria used the messaging
utility linked to her MPHR to contact the local Area on Aging to access their
quality ratings and prices for long term care facilities with dementia care unit.
2.1.Challenges to the vision of the Mature PHR
The scenario described presents a PHR fully embedded in the consumer’s preferred
communication channels and serving as the nexus for linking medical care, personal
health behaviors and public health services and information potentially relevant to the
consumer and family members. The supporting technology is already in place and
almost certainly will become more powerful and available at lower cost within this
decade. So the technological infrastructure for the applications identified is neither
remote nor radical. The challenges presented by this scenario from a public health
policy perspective are related to use, acceptance and satisfaction with the MPHR on the
consumer side and the will and capacity for data system integration, provider system
commitment and inter-institutional bridge building on the provider side.
2. 2. Legal and Regulatory Challenges
One obstacle to consumer acceptance may be heightened concern about maintaining

the security and privacy of the consumer’s PHI from inappropriate access and use. The
PHI embedded in the PHRs that are tethered to the EHRs under the control of
institutional health care providers in the US are covered by the protections of HIPAA
legislation. However, PHI contained in proprietary PHRs available commercially is not
covered under HIPAA. The research evidence related to patient perceptions about the
security and privacy of their PHI presents a mixed message about the level of consumer
concern. While one national survey found that two-thirds of all respondents expressed
concern about the privacy of their PHI, the majority of those who identified as PHR
users did not report concerns [2]. Another study of the use of Web-based access to the
patient’s electronic health record showed that patients who were frequent users
reported less concern than providers about privacy and security [3]. One could argue
that barring any statutory changes that loosen restrictions on PHI, the US consumer is
adequately assured that the privacy of her/his PHI is well-protected. Nonetheless, the
pervasive societal concern about potential assaults on privacy of information and

confidentiality, identity theft and other threats to personal security –given the ease of
on-line communication and economic transactions may increasingly be perceived as a
threat to privacy during the decade ahead..
2.3. Institutional and Management Challenges
A mature and fully functional PHR entails an integrated health care system in which
medical services, in-hospital, ambulatory, rehabilitation and home health services are
coordinated with and mutually reinforce the disease prevention, environmental
protection and health promotion population-based services associated with public
health practice. Such system integration is inherently complex even if the political will,
appropriate incentives and regulatory mechanisms are in place to support it. The
daunting and ongoing challenges faced in implementing the Patient Protection and
Affordable Care Act (2010) in the US, at best a partial health care system reform, are
instructive and not encouraging for comprehensive health system integration and
reform, at least for systems that are non-nationalized hybrids. The existence of multiple
payers and conflicting reimbursement logics, e.g., fee-for-service, bundled, capitation
premiums, pay for performance and other managed care incentive plans, makes total
integration even more complex. The alignment of medical care provider practices and
incentives with community and population-based program designs and intervention
strategies while not inherently incompatible present the challenge of coordinating and
accommodating two different organizational cultures and institutional frameworks.
2.4. Societal and Cultural Factors
Perhaps an even greater concern from the public health perspective is that the
accelerating proliferation of information available electronically provides cover for
intentional generation of misinformation whether motivated by narrowly political,
financial or doctrinal purposes. In the health arena such misinformation masking as
scientific evidence and/or supported by (for example) celebrity endorsements can lead
to adverse health consequences for the individual or population targeted. The false
attribution of a link between immunization and autism, even in spite of the repudiation
of any causal link by medical researchers, can result in gullible, uniformed parents
refusing vaccinations for their children resulting in increased risks to the community’s
health by reduction in herd immunity.
The sophisticated health consumer may be expected to weigh the evidence and
reject implausible and unfounded claims and advice from unreliable sources. However,
the less educated and more vulnerable consumer will likely be at greater risk of basing
health behavior on bad information and fraudulent claims. Consequently, the ocean of
health related information potentially available to the consumer can pose risks to the
integrity of the MPHR and/or the consumer’s effective use of it in sustaining her/his
health and well-being. It should be noted that this situation is complicated by the rapid
release and distribution of translational research which may offer contradictory or
ambiguous findings which can baffle even the more well-informed consumer.
W. Rohrer and M.S. Househ / Implications of the Mature PHR for the Empowered Consumer 13
2.5. Technological Challenges

For the MPHR to be successful, various technological challenges must be
addressed. First, the technological infrastructure must ensure that health-related
information is being shared across organizations according to nationally recognized
standards [4]. Without an adequate infrastructure, it will be difficult to develop a
MPHR that combines various data sources from the multiple interactions the patient
will have within the healthcare system, across venues of care. The MPHR requires a
functional health information exchange to ensure that patient interactions are recorded
so as to comply with nationally recognized standards to allow EHR systems to share
patient information that is timely, accurate, relevant and secure.
Furthermore, with the rise of social networking, the MPHR should include social
networking elements to help patients connect to various support groups. Given the
widespread use of the internet and its relatively inexpensive bandwidth, social
networking media are increasingly used by healthcare professionals and patients. In a
commentary in the Journal of the American Medical Association, Shachak and Jadad
argue that the use of social networks will eventually lead to a more people-centered
healthcare system that will improve communication and information flow between
patients, providers, and administrators [5]. Other studies have shown that users of
Facebook currently seek health information online [6]. How to incorporate social
networking utilities within the MPHR while maintaining the privacy and security of
patient records will be a significant challenge.
3. Conclusion
In summary, the evolution of the PHR beyond its current state of adoption, use and
functionality into a more mature and widely used tool for the empowered health
consumer is an exciting prospect but one that presents an array of challenges, both
regulatory, managerial, socio-cultural and technological. We conclude that the
potential pay-offs for enhancing the individual’s health and well-being and protecting
and sustaining the public’s health overall is worth a considerable investment in policy
analysis, technological R&D, partnership building and the expenditure of political
capital by influential advocates and decision makers.
Acknowledgements
We would like to thank the King Abdullah Institute for Medical Research for their help
in editing this document.
References
[1] US Department of Health & Human Services, Office for Civil Rights. Personal health records and the
HIPAA privacy rule, p. 1, undated)
[2] Curran WJ, Steams B, Kaplan H. Privacy, confidentiality and other legal considerations in the
establishment of a centralized healthdata system. N Engl J Med. 1968;281:241-8.
[3] Thompson LA, Black E, Duff WP, Paradise Black N, Saliba H, Dawson K. Protected Health
Information on Social Networking Sites: Ethical and Legal Considerations. J Med Internet Res
2011;13(1):e8. URL: http://www.jmir.org/2011/1/e8/ doi: 10.2196/jmir.1590 PMID: 21247862
[4] NAHIT. Report to the office of the national coordinator for health information technology. Defining
key health information technology terms. April 08, 2008.
[5] Shachak, A. and Jadad, A.R. (2010). Electronic health records in the age of social networking and
global telecommunications. JAMA. 303(5): 452-453.
[6] Househ M. Sharing sensitive personal health information through Facebook: the unintended
consequences. Stud Health Technol Inform. 2011;169:616-20.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-15
Applying the Clinical Adoption Framework

to Evaluate the Impact of an Ambulatory
Electronic Medical Record
Francis LAUa,1, Colin PARTRIDGEb, Gurprit RANDHAWAa, and
Mike BOWENa
a
School of Health Information Science, University of Victoria, Canada,
b
Hope Admirals Medical Clinic, Victoria, Canada
Abstract. This paper describes the application of the Clinical Adoption (CA)
Framework to evaluate the impact of a recently deployed electronic medical record
(EMR) in a Canadian healthcare organization. The CA Framework dimensions
evaluated were EMR quality, use and net benefits at the micro level; and people,
organization and implementation at the meso level. The study involved clinical and
support staff from two ambulatory care clinics, and managers and technical staff
from the organization. A number of issues were identified at both levels of the CA
Framework that had affected EMR adoption in the two clinics. Some perceived
benefits in care coordination and efficiency were reported despite challenges that
arose from early deployment decisions. There were five lessons that could be
applied to other ambulatory care settings. The CA Framework has proved useful in
making sense of ways that EMR can add value to the organization.
Keywords. Electronic medical record, clinical adoption, benefits evaluation
Introduction
Despite increased adoption of the electronic medical record (EMR) in primary health
care reported in recent years, its impact on provider performance and patient outcomes
has been mixed [1,2]. For instance, a systematic review by Lau et al. [3] of 43 EMR
evaluation studies has shown only 51.2% had positive impact, while 30.2% had no
effect. Given the recent attention to transform primary health care in Canada and the
great expectations on EMR as the enabler [4], the lack of positive evidence to date is
cause for concern. That said there are successful primary health care practices that have
shown improved care thru innovative use of their EMR. Examples are the use of
individualized decision support and reminder tools by Holbrook et al [5] to improve
diabetes care in a Canadian community, and the EMR supported quality improvement
project by Nemeth et al [6] in 99 primary care practices in the United States. In this
paper we describe the formative evaluation of a recently deployed EMR in two
ambulatory clinics of a regional healthcare organization in Canada. We applied the
Clinical Adoption (CA) Framework [7] to make sense of the impact of EMR adoption
in both clinics and the lessons learned.
1
Corresponding Author: Francis Lau, School of Health Information Science, University of Victoria,
P.O. Box 3050 STN CSC, Victoria BC, Canada V8W3P5
16 F. Lau et al. / Applying the CA Framework to Evaluate the Impact of an Ambulatory EMR
1. Objective
This formative evaluation study [8] applied the Clinical Adoption Framework (CA) to
examine the impact of EMR adoption in two ambulatory care clinics in a regional
healthcare organization, and the lessons learned to guide future implementation effort.
2. Methods
2.1. Evaluation Design
A team of four university-based researchers conducted the evaluation over a 6-week

period. The conceptual model used to make sense of the impact was based on the
Clinical Adoption Framework by Lau et al [7]. This Framework provides an organizing
scheme by which one can examine the quality, use and net benefits of a health
information system (HIS) in specific contexts. For this evaluation the relevant micro
level metrics were EMR system, information and service quality, usage and
satisfaction, and care quality in terms of coordination and efficiency. For the meso level
people, organization and implementation issues were examined. People included roles,
expectations and experiences. Organization included infrastructure, strategy and
process. Implementation included deployment process and EMR-practice fit.
2.2. Data Collection and Analysis
The research team employed a set of rapid evaluation methods [9] that consisted of
EMR adoption survey [10], user assessment, usability/workflow analysis, document
review, data quality review, project risk assessment, and group practice reflection. The
data collection took place over four weeks including review of project documents and
EMR data which took place concurrently. The EMR support staff organized sessions,
assembled relevant documents, and extracted EMR data for the research team. During
the study, the research team worked closely with EMR support staff to verify the
information collected to ensure its accuracy. All notes taken during interview,
usability/workflow, and focus group sessions were transcribed and analyzed for
common themes. The evaluation report was finalized in the last two weeks.
3. Evaluation Findings
Forty-three participants were recruited in the study that included providers and support
staff from both clinics, EMR support staff, and managers involved in the project. Over
4 weeks the team visited the clinics 3 times and completed 13 adoption surveys, 14
usability/workflow sessions, 13 user assessment interviews, 11 project risk assessment
interviews, 3 focus group sessions, and reviewed 65 project documents and 3 months of
EMR data. The findings were summarized according to the micro and meso dimensions
of the CA Framework. Due to space limitation only selected findings are included.
F. Lau et al. / Applying the CA Framework to Evaluate the Impact of an Ambulatory EMR 17
3.1. EMR Quality

For system quality, six EMR functional areas and four types of usability issues were
examined through two patient scenarios. The functional areas covered general system
flow, medications, appointments, referrals, lab tests and radiology. The EMR was
found to be a sophisticated product with many configurable features. The usability
issues related to workflow, user interface design, system feature/performance, and user
knowledge. In total 36 usability issues were identified. Most issues were in general
system flows and referrals in terms of unexpected system behaviors, sub-optimal
performance and inadequate default settings.
For information quality, the EMR successfully captured the patient information
need to deliver care. In general the recording of encoded problem lists and
medications were satisfactory. Free-text was often used especially for recording
allergies where ~50% of the entries were uncoded. Some medications such as HIV
drugs and frequencies were in free-text. The use of free-text impaired the ability to
perform drug-allergy/interaction checks. Since implementation the EMR had
accumulated some inconsistent user login configuration records and unmatched
clinical documents. The master code lists for diagnosis, medications and allergies were
incomplete or had incorrect entries. Several patient records had mismatched diagnosis
codes and descriptions, or missing information such as being treated for diabetes but
without the diagnosis present.
For service quality, while EMR support staff was highly responsive this resource
was limited. On a feature by feature basis, the EMR was as sophisticated as the
inpatient HIS with such core clinical functions as patient scheduling, team-based care
planning, order processing, clinical documentation, specialist referral and decision
support. At the user level, EMR support was expected for all aspects of patient care in
the clinics. Such expectations were beyond those in the inpatient setting, where
providers were not yet into full electronic charting in their hospital information system.
3.2. EMR Usage and Satisfaction
For usage and satisfaction, the EMR adoption level was 2.1/5 for Clinic A and 1.4/5 for
Clinic B across 10 functional areas based on the EMR adoption surveys (0 as paper and
5 as advanced). The best sub-scores in Clinic A were for referrals at 3.4/5 and
medications at 3.1/5, whereas in Clinic B they were for health information at 2.9/5,
medications at 2.5/5 and lab at 2.5/5. The low scores in both clinics were in decision
support and electronic communication and patient support. Clinic users ranked their
overall satisfaction with the EMR at 2.4/5 with 5 being highly positive.
Participants revealed that Clinic A was a high volume short visit environment
typical of primary care. Providers worked full-time and needed information to be
immediately at hand and in one place. This environment provided strong incentive for
providers and support staff to become proficient in using the EMR. Yet the time
pressures could also drive users to find workarounds for tasks that interfere with
workflow. Many providers from Clinic B had their own practices elsewhere and were
only present for 1-2 days a week at the clinic. It was often easier for them to use
paper than to invest time into learning the EMR. The survey revealed decision
support for prescribing such as drug interactions had been disabled.
Despite frustrations, clinic users were hopeful and expectant that there would be
improvement over time. For EMR functions that were already in place, there was a
need to revise existing user and system configurations to improve their intended use.
These included medication prescribing, lab/imaging investigations, referrals and patient
charting. For EMR functions in limited use, revisions and testing were needed to ensure
they work as intended. These included care plans and chronic disease management
forms. The EMR functions not yet in use that should be considered were flowsheets,
database queries and clinical decision support for drug interactions and allergy checks.
3.3. Net Benefits in Coordination and Efficiency
For care coordination, clinic staff saw value in capturing encounter notes, medications,
lab work, referrals and billing electronically in the EMR. The technical interface
between the HIS and EMR had created a potential patient safety issue. The HIS was the
source of truth for patient demographics, scheduling and allergy information. Though
the EMR had the ability to schedule appointments and record allergies, clinic staff were
not able to do so. For allergies, providers had to relay the information to the clerks who
then had to enter it into the HIS. This information would then be downloaded to the
EMR later. Clinic appointments had to be booked through the HIS, which did not allow
detailed reason for visit to be recorded or timely updating of patients who showed up
late at the clinic. The billing interface for physician services and labs required legal
patient names which sometimes were different from what was in the EMR. As a result,
clerks had to change the names for billing then matched the lab results.
For efficiency, clinic staff appreciated the EMR allowed them to find the chart and
the information needed at the time of patient visit. Yet providers faced challenges
working with fragmented charts. Clinical documents such as diagnostic imaging,
discharge summary and consultation reports were not stored in the EMR as they
were out of scope for the initial implementation. This had impeded workflow and
incented staff to find workarounds. Some consultation reports were supposed to be
stored in the HIS but were entered in the EMR since that was where the providers
needed the information. If a provider needed an imaging report and it was not found in
the EMR, they had to login to the HIS to look for it then returned to the EMR. Often
they would add a free-text comment in the patient’s medical history to indicate an
imaging result was in the HIS.
3.4. People, Organization and Implementation Issues
For people issues, clinic staff described having the opportunity for initial training but
with little follow-up afterward. These included existing users who wished to improve
their proficiency with additional EMR functionality and training for new staff that
joined the clinic after the system went live. Clinic staff recognized the EMR could do
much more for them if they only had time and resources dedicated to training them and
helping them move to the next level. For instance, the EMR had the ability to generate
automatic reminders to recall patients for follow-up lab work and investigations.
However this function was unavailable early on thus staff created workaround routines
which then had to be undone when it became available. At a more fundamental level,
some clinic staff indicated a lack of basic information management knowledge such as
ICD-9 coding that could be of value for EMR users.
For organizational issues, the two clinics had unique challenges in that the EMR
was deployed at the same time when the ambulatory care centre was being opened. As
such, there were lack of clearly defined staff roles on such tasks as scanning
documents, no clear clinical workflows on handling of scheduling, referral and billing,
and no detailed planning of the business requirements of how an EMR supported clinic
F. Lau et al. / Applying the CA Framework to Evaluate the Impact of an Ambulatory EMR 19
would function. These had led to different expectations of what the EMR should do and
how the system should be configured. For instance, the EMR had care plans,
custom templates and chronic disease management flowsheets that were not exploited
due to lack of defined requirements on where/how to use them.
For implementation issues, managers and EMR support staff involved with the
initial implementation rated their experience at 2.4/5 on a Likert scale of 1 to 5 with 5
being strongly positive. Most felt that there were insufficient resources available
and the EMR was not aligned with the clinic workflows. Interviews with staff
involved with the initial EMR implementation revealed their frustration that it was
viewed by others in the organization as a simple project. There was also a perception
that the EMR was a shrink-wrap product ready for plug-and-play with little training
required once installed. This view was reinforced by the project approach used to
implement the EMR, where an aggressive timeline was established to deploy the EMR
and the measure of success was that it got implemented and went live at the clinics.
The review of project documents confirmed the interview findings where similar issues
were reported.
4. Lessons Learned
The EMR adoption level achieved at the two ambulatory care clinics was comparable
to similar clinics that we had studied in the past two years that were within 1-year after
their EMR was implemented. Our experience suggested EMR users often had variable
experiences with the system depending on their expectations and prior IT experiences.
We should point out this evaluation represented only a point in time shortly after the
EMR was implemented at the two clinics. Despite the challenges reported, we found
clinic staff was resilient believing that the EMR was better than what they had before,
and that the system would improve over time and add further value to their work.
The lessons from this study that could guide future EMR implementation efforts are
to: (1) recognize EMRs as sophisticated clinical applications that require resource,
effort and commitment over time to adopt successfully; (2) have clearly defined staff
roles, clinical workflows and business requirements for the clinics involved to
maximize EMR-practice fit; (3) pay early attention to EMR data quality to minimize
subsequent database cleanup and realize benefits sooner; (4) provide incremental user
training over time to maximize EMR value based on a comprehensive training strategy
with different modalities; and (5) Redefine successful EMR adoption as being able
to demonstrate tangible clinical value beyond system going live. We believe the CA
Framework provided a useful organizing scheme for us to make sense of the EMR
impact from the dimensions of EMR quality, use and benefits, and contextual factors
involved.
References
[1]. C. Schoen, R. Osborn, M.M. Doty, D. Squires, J. Peugh, S. Applebaum. A survey of primary care
physicians in eleven countries, 2009: perspectives on care, costs and experiences. Health Affairs 28
(2009), w1171-w1183.
[2]. CMA. National Physician Survey 2010. http://www.nationalphysiciansurvey.ca. March 25, 2012.
[3]. F. Lau, M. Price, J. Boyd, C. Partridge, H. Bell, R. Raworth. Impact of electronic medical record on
physician practice in office settings – systematic review. BMC Med Inform Decision Making 12
(2012):1-10.
[4]. B. Hutchison, J. Levesque, E. Strumpf , N. Coyle. Primary health care in Canada: Systems in
motion.Milbank Quarterly 89 (2011), 256-88.
[5]. A. Holbrook, L. Thabane, K. Keshavjee, L. Dolovich, B. Bernstein, D. Chan, S. Troyan, G. Foster,
H.Gerstein, for the COMPETE II Investigators. Individualized electronic decision support and
reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ 181
(2009), 37-44.
[6]. L.S. Nemeth, A.M. Wessell, R.G. Jenkins, P.J. Nietert, H.A. Liszka, S.M. Ornstein. Strategies to
accelerate translation of research into primary care within practices using electronic medical records.
Journal of Nurs Care Qual 22 (2008), 343-9.
[7]. F. Lau, M. Price, K. Keshavjee. From benefits evaluation to clinical adoption: making sense of health
information system success in Canada. Healthc Q 14 (2011), 39-45.
[8]. J.J. McGowan, C.M. Cusack, E.G. Poon. Formative evaluation: A critical component in EHR
implementation. JAMIA 15 (2008), 297-301.
[9]. UVic eHealth Obsevatory. EMR evaluation toolkit v3.0. University of Victoria.
[10]. M. Price, F. Lau, J. Lai. Measuring EMR adoption: A framework and case study.
ElectronicHealthcare 10 (2011), e25-e30.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-21
Role of Healthcare Information

Technology in Handoffs
Ross KOPPEL a,1 and Joel Leon TELLES b
a
School of Medicine & Sociology Dept, Univ. of Pennsylvania; & Harvard Medical
School
b
Main Line Health System, Bryn Mawr, PA
Abstract. Handoffs—transfer of patient care from one clinician or service to

another—are well known patient safety dangers. Healthcare Information
Technology (HIT) as an intervening and powerful force in handoffs has received
comparatively little attention. The role of HIT in concert with paper
documentation has received even less attention. We analyze handoffs in relation to
electronic records and hybrid systems (both paper and HIT) to identify sources of
error and miscommunication. We propose a typology of handoffs and illustrate
several of them..
Keywords: Handoffs, Handovers, Care Coordination, Healthcare IT
Introduction
In the inpatient setting, handoffs occur several times a day for each patient. Handoff
communication failures are a known and frequent source of errors.[1-5] These errors
occur in almost all medical settings, e.g., emergency departments, among units in a
hospital, between hospitals and nursing homes. Yet, the role of healthcare information
technology (HIT) in handoffs has received little attention, especially in light of HIT’s
growing role in all medical settings and its potential to enhance handoff safety. Equally
important, the potential usefulness of HIT in handoffs when combining paper and HIT
has received even less attention—even though several HIT programs print out paper
handoff forms from programs within the EHRs themselves.[4]
In our observations, EHRs and other digital media are inconsistently incorporated
in handoffs even when the hospital is fully wired. Moreover, clinicians’ use of HIT
during handoffs is frequently an afterthought or a supplement that can distract as well
as illuminate—a reality affecting both departing and incoming clinicians. Oral
communication and paper remain the media of choice, and even paper is not
systematically employed. Integration of paper and HIT is less common and is even less
well thought out when it does occur. Hospitals and other medical settings lack a
coherent strategy for handoff processes that use HIT and/or HIT with paper documents.
In this paper, we examine handoffs that often incorporate HIT (usually EHRs)–
or—more commonly, handoffs that fail to incorporate HIT or incorporate HIT in
haphazard fashion. Until recently, HIT/EHRs were seldom a part of the handoff process.
But with the growing availability of HITs, ignoring its information in the handoff is a
1
Corresponding Author: email: [email protected].
22 R. Koppel and J.L. Telles / Role of Healthcare Information Technology in Handoffs
potentially significant loss of patient safety and clinical efficacy. Equally disconcerting
is allowing the HIT to structure the handoff process haphazardly – perhaps missing
needed emphases while mechanically repeating obvious or insignificant information.
Most often, handoffs involve pieces of paper that are subsequently discarded. Even
though the EHR is presumably the repository of the official and most accurate
information, it is often ignored or relegated to a marginal role. This may be because of
its widespread use is comparatively recent; because it is not part of the routine training
process; because it is seldom designed for handoffs; or because the use of HIT has
heretofore required proximity to a computer screen—a situation that has been altered
with computer tablets.
Information Source and Limitation
The presentation is based on several years of observations of handoffs in many

hospitals. We examine handoffs that are primarily face-to-face between two staff
members. Thus, we exclude hospital discharges, transfers from ambulatory care, or
transfers between hospitals and long-term care facilities unless there was an additional
meeting of clinicians that we could observe. We exclude phone conversations since we
could only overhear one side of the interaction.
1. A Typology of Handoff Elements
To create a context for analyzing handoffs, we explore some of the parameters within
which they occur, e.g., specific locations; whether they occur with or without paper,
structured formats, access to computers, participation of other clinicians, presence of
senior clinicians, and so on. The table (Table 1) presents a typology of factors that
influence handoffs’ clarity and accuracy. We refer to clinicians leaving the patient, shift,
or hospital as “departing” clinicians and to clinicians accepting care of the patient,
starting the shift, or managing the floor as “incoming” clinicians.
While the table reflects months of observations and interviews over several years,
we offer no systematic analyses of the frequency of each category or of the correlations
among the categories. Some relationships are obvious: Patient involvement is more
likely if handoffs are in or near patients’ rooms; proximity to computers is needed for
their incorporation in handoffs. Other physicians are more likely to be nearby if the
handoff is conducted in a group setting or at physicians’ workstations. Moreover, the
parameters provide a basis for examining additional issues such as: errors if the
documented information is wrong or outdated; errors of memory; staff sanitizing
phrasing due to concern about a patient overhearing prognoses; time required to create
a structured report versus information loss or misunderstandings with unstructured or
poorly documented reports; corrections provided by third parties and by those from
other disciplines; inefficiencies of ‘translating’ information to another discipline’s argot.
In addition, other categories could be included, e.g., fragmentation of the
information by other activities, role of clinician experience (head nurse, interns, senior
physicians), and delays caused by efforts to collect needed additional information. This
typology is elaborated in the following discussion, in which we use paintings and
etchings as metaphors to illustrate vulnerabilities involving handoff communications.
R. Koppel and J.L. Telles / Role of Healthcare Information Technology in Handoffs 23
TABLE 1. Sixteen Parameters of Handoff Interactions: Roles of Media (HIT, Paper), Memory, Type & Use
of Structured Formats, Location, Patient Presence, Other Artifacts, Dyadic or Group Setting, Intra- or Inter-
disciplinary, Face-to-Face or Mediated, & Involvement of Other Clinicians
1. Media Generally includes Oral. If also other media, they could be: Paper only; Paper
combined with HIT; HIT only. Different media might be used by incoming and
departing staff, e.g., HIT information from departing recorded on paper by
incoming; paper information entered in HIT and also on paper, etc. There might
even be a note in lieu of face-to-face communications.
2. Documentation Oral handoff with no recording of information in any medium. Paper, and even
more, HIT provide documentation accessed by more people. Many handoffs were
observed without reference to paper or HIT documentation.
3 Reliance on Extent to which information is provided based on what the departing clinician
Memory knows or remembers rather than/in addition to what is documented.
4. Incoming clinician Recipient’s questions and recipient looking up information. Reflects incoming
seeks information clinician’s active involvement in researching information and asking questions.
5. Degree and Type of Categories by problem, medication, organ system or temporal proximity, etc –
Structure in adhered to by departing clinicians. Use of structured discussion (such as SBAR)
Handoffs. may be informally recorded.
6. Use of Available Structured handoff reflected in templates or forms: HIT, paper or a combination.
Structured Forms Templates/forms may be used, partially used or ignored.
7. Accessibility & For paper: Small print summary sheets with little room for additions and notes.
Readability For computer: large-enough screen, viewable/findable by relevant parties
8. Handoff Location Patient Room; Nurse Station; Physician work area; Hallway; Other
9. Proximity of Role of patient’s presence in the handoff process. Role of other artifacts (e.g.,
patient & artifacts medications, IV bags)
10. Dyadic or Group Two clinicians alone or as part of team, rounds or other group settings
11. Intra- v. inter- MD to MD; RN to RN versus interdisciplinary, e.g., MD to RN or multi-
disciplinary disciplinary rounds.
12. Hierarchy Peers (resident to resident); or different levels (nurse manager tostaff nurse)
13. Face-to-Face or Traditional face-to-face, or involving the use of tape or voice mail.
Mediated
14. Consultations Others contacted or not; participation by others by chance or actively sought.
15. Speed Time allotted or available for handoffs.
16. Fragmentation Degree to which staff are interrupted and face other distractions to the handoff.
2. HANDOFFS AND HIT
Hierarchy and status are always elements of handoffs (Figure 1), and their roles in
patient safety are critical—whether between peers (e.g., nurse to nurse; resident to
resident) or between those of different positions (e.g., resident to attending).
Figure 1 Figure 2
2.1. Structure and Meaning
The Kandinsky painting (Figure 2) depicts the often amorphous structure and freeform
presentation in handoffs--allowing clinicians much latitude in evaluating essential vs.
less important information as long as they use the two forms well and the incoming
clinician sees which is which. Many of the handoffs we observe combine areas of clear
and understandable information with opaque, and sometimes information that is missed
entirely or assumed away. These handoffs are largely unstructured either by the
departing clinician or the incoming clinician. Notes are sporadic and not organized in
any clear way. For example, we often observe clinicians referring to areas of a page or
screen that encompass medications, diagnoses, and laboratory reports. Some of the
information is specific (e.g., laboratory values) but other parts are suggestions for new
observations or tests. Of course, assumptions about the other clinician’s focus or
understanding may not be shared.
As suggested by the next Kandinsky (Figure 3), even when there is clear structure
in the handoff process, clinicians may differ about what data are most essential and
what should be covered. That is, one or both of the clinicians has a physical or mental
check list or set of items that is a backdrop to the interaction. Only a few of the items,
however, are discussed in the handoffs and no structured handoff record is maintained.
The rest of the checklist is taken for granted as known. This is not necessarily
dysfunctional: in the hands of experienced clinicians both of whom know what items
are essential and what are sufficiently routine or well known, much information can be
skipped.[6,7] (Some information might be intentionally excluded [e.g., prior drug use]
to protect patients by only allowing sanitized medical records.) Moreover, there is an
infinite amount of information that can be conveyed about the patient; only certain
items are news. It is impossible, however, to be sure both sets of clinicians are fully
aware of all patient issues, and such assumptions are risky. Even attentive and
experienced people might miss or forget an essential item, or they might assume a level
of understanding and familiarity that the other clinician does not possess. Patient status
and lab results change, sometimes unexpectedly—and clinicians who have been
treating a patient for a week or more may be believe there is little new information in
the chart. In addition, because not all relevant information is always systematically
recorded, some information might not be available for later review. In turn, systematic
documentation, while it can be reviewed and interrogated, is subject to the same
selectivity and need for assumptions about each participant.
Figure 3 Figure 4
2.2. Level of involvement
The disaffected woman in the Modigliani (Figure 4) reminds us that clinicians, like all
humans, vary in their level of attention to and involvement in the handoff process. Is
the departing clinician/team exhausted or eager to leave? Is the incoming clinician/team
a passive recipient due to overconfidence, unrealistic beliefs about the patient’s
condition, disciplinary presumptiveness or risky eagerness to start care? On the other
hand, perhaps the incoming clinician already knows a lot about the patient, so only new
information is needed. Adding more information would be wasteful or at least one
presumes so. In these cases, the failure to incorporate structured handoff forms, EHRs,
and/or paper records could produce unwanted consequences. Relying on memory,
when the initial transfer of information was poorly attended and documented, seems
unwise.
The image of two angels with Christ (Figure 5) poses the dilemma when an
authority figure (e.g., attending physician) offers insights and great learning, but may
also inhibit free (and needed) communication. Some handoffs are mediated by a
respected authority, such as a senior physician or a nurse manager. The respected
authority may help clarify ambiguous information, demand additional information,
raise new issues to increase understanding and improve treatment plans. As noted
above, however, the authority figure may prevent questioning and additional inquiry.
2.3. HIT Handoffs and Hierarchy
We often observed younger clinicians in the presence of older and respected authorities
use available HIT as a neutral means to broaden the discussion by referring to patient
information, clinical guidelines, or even recent articles on patients’ conditions. Here the
authority’s power is moderated by the HIT in respectful but helpful ways. We also
repeatedly observed settings where senior clinicians, while acknowledging the HIT-
provided information, offered additional insights to impart lessons on the complexity of
care and the need to balance factors missing in the standard protocols.
Figure 5 Figure 6
2.4. Templates and Structured Handoff Forms
Ellsworth Kelly’s painting (Figure 6) highlights the role of structure. Highly structured
handoff forms & processes increase the probability that all items will be included. But
critical issues and consequential nuance may be lost. Highly structured forms or
templates can simplify and organize handoff information—making information entry
and retrieval faster and more secure than other methods. The downside of such forms is
that they often encourage a disgorging of routine and redundant data, in loss of
emphasis and critical nuance. These advantages and disadvantages hold true for both
HIT-based templates and paper forms. One possible advantage of paper is that the
affordances of the medium allows easier annotation and comment than does the digital
form. Also, independent of the medium, each of these forms and templates are met with
varying levels of compliance.
2.5. Paper and HIT Integration
We insert a computer screen into Durer’s Adam and Eve (Figure 7) to ask whether HIT
helps to convey needed information or is a distraction. EHRs, lab reports, I-O charts,
progress notes (in paper or HIT or both) are used in some handoffs. How can these
media be integrated to help focus clinicians on essential elements and present their
missing critical information? What are the risks these media will disperse focus and
lead clinicians to incorporate irrelevant or obvious information? Do they generate more
“busy work” that detracts from the clinicians’ thought-flow, workflow and time for
direct care? Are they used consistently or erratically? Does their use vary by unit,
individual, or supervisor? Most important: how are the paper and HIT (if both)
integrated? If they are hybrid systems – which most appear to be – do they maximize
the advantages of each medium? Are they redundant, complementary, reinforcing, or
distracting? While these are not new questions for any handoff analysis, the framework
suggested in Table 1 may help structure questions and a research agenda for further and
systematic investigation.
The meaning of Rembrandt’s Abraham and Isaac (Figure 8) depends on knowing
the biblical story. In addition to conveying information (in whatever form), a successful
handoff must reflect not only the recipient’s level of understanding--both of medical
knowledge and the patient’s condition--but also accommodate the participants’
interactions and relationships. There is always a context, expertise of each participant,
and a history of previous interactions in addition to the information exchange.
Efficient and effective handoffs reflect both needed information and awareness of the
participants’ context, backgrounds, and expectations.
Figure 7 Figure 8
3. Summary
Handoffs are consequential for patient care, clinician learning, patient safety, clinical
efficiency and liability. Improving our understanding of handoffs--especially how to
integrate HIT into handoffs--can improve outcomes. We suspect the use of HIT in
handoffs will improve care and efficiency. To date, HIT use in handoffs appears
inconsistent, is frequently supplemental, and may distract as well as illuminate. Paper
remains the medium of choice and integration of oral, paper and HIT is not well
thought out. To address this essential patient safety concern, we need a coherent
strategy for handoff processes that use of HIT and/or HIT with paper documents. We
hope this typology and the illustrated discussion spur further examination of HIT in
handoffs, leading ultimately to efficacious use of HIT and other media to improve the
quality and efficiency of patient care.
References
[1] J. Johnson and V. Arora, Improving clinical handovers: creating local solutions for a global problem
Qual Saf Health Care 18:4 (2009), 244-245.
[2] I. Philbert, Use of strategies from high-reliability organisations to the patient hand-off by resident
physicians: practical implications Qual Saf Health Care 18 (2009), 261-266 .
[3] V. Arora V, J. Johnson, D. Lovinger, H. Humphrey, D. Meltzer, Communication failures in patient
sign-out and suggestions for improvement: a critical incident analysis Qual Saf Health Care 14 (2005)
401-407.
[4] M. Flanagan, E. Patterson, R. Frankel, B. Doebbeling Evaluation of a Physician Informatics Tool to
Improve Patient Handoffs, JAMIA 16:4, (2009), 509–515.
[5] L. Riesenberg, J. Leisch, J. Cunningham, Nursing Handoffs: A Systematic Review of the Literature ,
AJN, 110:4 (2010 ), 24-34.
[6] H. Garfinkel, ’Good’ organizational reasons for ‘bad’ clinic records in H. Garfinkel, Studies in
Ethomethodology, Prentice-Hall, Inc., Englewood Cliffs, 1967, 186-207.
[7] C. Heath and P. Luff, Documents and Professional Practice: 'bad' organisational reasons for 'good'
clinical records in Proceedings of the Conference on Computer Supported Cooperative Work, ACM
Press, Boston, 1996, 354-363.
28 Enabling Health and Healthcare through ICT
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-28
Computerisation in General Practice:

Lessons for Canada from the UK and
Australia
Nicola T. Shawa,1 and Mike Bainbridge b
a
Health Informatics Institute & Algoma University, Sault Ste. Marie, Ontario
b
University of Victoria, Victoria, British Columbia
Abstract. In 2000 Shaw and Kidd published an article on the lessons that could be
learned from the UK in general practice computerization. Over a decade later
many of these lessons remain yet to be learned. Hence Shaw & Bainbridge felt that
it was time to revisit these issues and review progress made against each in both
the UK and Australia in an effort to help Canada learn the lessons as it follows
behind both countries. Nine lessons are identified, reviewed and discussed in the
hope that Canada will choose to take note and leapfrog these jurisdictions by
learning from history, rather than being doomed to repeat it.
Keywords. General practice computerization; EHR; EMR; education, training &

support; common data model; coding; information exchange; business case; data
validation & verification; system support; security & confidentiality; research
Introduction
“Those that fail to learn from history, are doomed to repeat it.” ~ Winston Churchill
At the turn of the century Shaw (formerly Ellis) and Kidd published an article on what
lessons Australia could learn from the United Kingdom (UK) in the area of general
practice computersisation [1]. Just over a decade later and it seems that many of these
lessons remain outstanding. Further, that other developed countries with comparable
medical systems, such as Canada, seem to be ignoring what has been learned through,
recent global experience of success as well as failure. We therefore thought that it was
an appropriate time to revisit the original article and draw attention to the lessons that
need to be learned, and acted upon, if we don’t wish to simply repeat the failures that
have gone before us.
Given also that the Canadian government has committed to primary care reform
and to a primary care led health care system and that Canada Health Infoway has
finally been allocated funding to support the adoption and implementation of Electronic
Records (ERs) in general practice it seemed especially appropriate to re-target these
lessons.
1
N.T. Shaw and M. Bainbridge / Computerisation in General Practice 29
1. The Good, the Bad and the Ugly
1.1. Education, Training & Support

The fundamental need for appropriate education, training and support is a lesson that
Australia learned well from the UK. By establishing the General Practice Computing
Group (GPCG), from 1998-2001, with a specific mandate to provide localised
education, training and support independent of which vendor system was being used,
Australia moved from less than 10% of general practitioners using computers to over
86% in just 3 years[2]. In the UK only, there have also been notable initiatives around
‘informatics professionalism’3 as well as ensuring informatics is ‘embedded’ into the
clinical curriculum[4]. These national-level initiatives do not currently exist in either
Australia or Canada.
Except in the UK it is still rare that clinical systems are used to document the
entire clinical record in a structured and coded fashion. Many are still only used as
billing tools with little use for management or audit. What is also missing is a culture of
information proficiency both within the General Practices and the local agencies
charged with the delivery of healthcare in an area.
Lesson for Canada – Successful implementation and sustained use of clinical
information systems requires a bedrock of informatics support as well as an acceptance
that informatics is a core clinical skill for the 21st century. Appropriate resource and
effort needs to be urgently mobilized to achieve this wrapped around supportive policy.
A piecemeal approach will be ineffective for reasons we shall discuss in this document.
1.2. Common Data Model
Being able to transfer data between different vendors clinical systems whilst
maintaining semantic integrity has long been desired. Were this to be realized, multiple
benefits would accrue from avoiding duplicate data entry to the removal of barriers to
better research as well as the more obvious ones of safer and more coherent patient care.
Over the years a number of different models have been tried to achieve data
interoperability without scalable success.
The English National Program for Information Technology (NPfIT) has achieved a
world first where over 57% of all GP practices are now transferring the clinical record
with semantic integrity when a patient moves[5]. This work was achieved by NHS
clinicians and informaticians working together taking forward the culmination of the
work achieved in the European EN 13606 standard [6], the OpenEHR and GEHR
projects [7] linked with HL7 [8] and SNOMED-CT [9]. The GP extraction project [10]
is also set to derive a large research database based on this same ‘logical data model’.
Over the last decade there has been a slight reduction in the number of clinical system
vendors (There are now 7 actively marketing with over 95% of the market represented
by four suppliers.). A contraction in the marketplace occurred in the decade preceding
the National Programme for reasons largely to do with commercial pressures and an
increasingly robust mandatory accreditation process linked to the 2003 ‘Quality-based’
GP contract [11].
Australia is in a similar position with only 6 vendors attaining membership of the
‘GP vendor panel’ in 2010 [12] .The position with a common data model is, however,
largely unchanged. All systems use proprietary coding systems and structures and there
is no transfer of data between systems apart from conversions when a practice changes
their vendor [13]. Multiple agencies (both Government and Commercial) are
30 N.T. Shaw and M. Bainbridge / Computerisation in General Practice
developing data extraction and aggregation programmes but one must question both the
clinical and research value of these data without standardisation of definitions, coding,
structures or semantics. The Personally Controlled Electronic Health Record [14]
(PCEHR) for Australia went live in July 2012. It is based on a core ‘document’ set
indexed by a Unique National Individual Health Identifier (IHI) (shared health
summary, event summary, discharge summary, Medicare data, & consumer entered
data with pathology, specialist letters, prescriptions and dispensing notices to follow).
At present, the conformance criteria for some documents are low and permit free text
and ‘pdf’ content. However, the system is built to allow structured and coded data [15]
and will, over time, catalyze change and be delivered through increasing the level of
conformance criteria.
Lesson for Canada -The implementation and use of the PCEHR is a program that
Canada should follow closely. Minimal initial change from both vendors and clinicians
is needed at first but this approach also sets a path towards strategic goals which is
flexible.
1.3. Clinical Coding
Read Codes for diagnosis and symptoms were accepted as the standard for English
general practice computerisation in 1988 [17]. Their initial use wasn’t standardised
with slightly different versions being by different vendors. The editorial process was
seen as unresponsive and some vendors also authored and released their own additional
proprietary codes. This practice has now died out as the twin forces of interoperability
and a quality-based GP contract have emerged. As electronic records expanded beyond
general practice, the need for a more comprehensive terminology was recognised. This
led eventually to the international development of the SNOMED Clinical Terms
(SNOMED-CT) and its current instantiation as a ‘not for profit’ collaborative
organization between 19 Countries [18]. It has become a ‘fundamental’ standard for
clinical coding in England [19] with other countries set to follow this lead. Since the
advent of the 2003 GP contract the use of the computer at the point of care is almost
ubiquitous. There is also much more delegation and sharing of information between
Health Care Providers (beyond the scope of this paper). Both Canada and Australia are
participating members of the International Health Terminology Standards Development
Organisation (IHTSDO) and have named SNOMED-CT as their ‘preferred’
nomenclature nationally. However, in both countries, the proliferation of other coding
schemes and the confusion between nomenclature and classification has been allowed
to continue. Classifications such as ICD-10 and ICPC2 remain in use at the point of
care rather than being derived from a nomenclature such as SNOMED-CT. Many
vendor systems still don’t encourage or support clinical coding and rely heavily on
financial coding undertaken for billing purposes to derive clinical utility, audit and
reporting data.
Lesson for Canada – It is vital to set a course toward a mandated SNOMED and
Canadian National drug extensions as soon as possible. This effort has demonstrably
moved England forward. Australia has been less demonstrative but has set the same
direction of travel through conformance with CDA architecture [20]. Consideration
must be given for effective Vendor and Clinical engagement as the usability of systems
incorporating SNOMED will determine acceptability.
1.4. Information Exchange/Interoperability
Despite almost complete general practice computerisation since the mid 1980’s it was
only in 2007 that records were able to be transferred between practices using different
UK vendor systems. Five years after initial success nearly all clinical systems have
been deployed containing (or about to contain) accredited software [21]. To date
2,137,301 (27th June 2012) patient records have been transferred. There has been great
investment of time, effort and money in establishing data transfer standards, building
and then deploying a central ‘spine’ through which data would be transferred.
Awareness and acceptance of the need for record transfer amongst the medical
professional community who traditionally felt a great deal of ownership towards “their”
records has also had to be achieved on top of safe and professionally acceptable and
semantically correct transfer standards.
Despite the domination of the market by one single vendor there is still a complete
lack of interoperability in Australian general practice although the intention is that the
PCEHR will catalyse resolution in this area. The PCEHR is a document based
implantation of Clinical Document Architecture [15,16-20] with an ultimate goal of
fully structured and SNOMED coded content. Increasing levels of compliance will be
driven through the Standards-Australia process. There is considerable overlap with this
section and section 1.1 the common data model. The two are synergistic and
inseparable.
Lesson for Canada – Interoperability is the best catalyst for change as it forces
the re-examination and re-configuration in multiple areas from definitions to
infrastructure. There is an opportunity to shape the hard-won successes in this area for
local use in Canada. This, however, will only be achieved through a coordinated
national approach.
1.5. Business Case
Developing a business case for electronic records has become the subject of many case
studies and classroom debates. Whilst more readily accepted by larger groups of
practitioners and those concerned with population level health, as opposed to individual
general practitioners, a true business case for adoption and implementation of
Electronic Health Records (EHR) has not yet been mad [22].
Despite this there is an increasing consumer demand for service, excellence and
availability that electronic record supporters purport that such systems can support and
achieve if only they were implemented. At the same time consumers are increasingly
sharing their own personal information electronically, through social media. Other
service industries are increasingly reliant on online consumer participation for much of
their service provision.
Likewise, there is a growing emphasis on clinical safety as a driver for
computerization. Whilst it makes inherent sense that computerisation could improve
safety, by such measures as forcing compliance with standardization and ensuring
legibility, there is little, to no, evidence that electronic records do actually improve
clinical safety beyond the area of prescribing [23,24]. Some of the reasons behind this
lack of evidence lie in the proprietary nature of the systems and the continued
difficulties in deriving interoperable and truly comparable data at a scale to prove the
hypothesis.
Lesson for Canada – Consumer demand for excellence and safety will rapidly
mount if the current situation continues – It is also important to keep in mind that
ubiquity of implementation of the current systems will at best mechanize current
practice and this is insufficient to meet upcoming demands.
1.6. Data Validation & Verification Procedures
The quality of data within an organization has been a recognized problem for decades
but clinics rarely run simple data quality checks of their own accord. In the UK the
government attempted to address this issue by establishing the PRIMIS programme
[25] to provide localized training and data analysis. One of the primary components of
this program is to help general practitioners ensure that they have good data quality.
Australia has recently published training materials for Clinicians to use in support of
the PCEHR[26] and has recently funded the Royal Australian College of GPs
(RACGP) to intervene on the broader e-health agenda [28] .There is, as yet, however,
no national ‘bottom up’ support for e-health and information proficiency although the
eCollaboratives [27] set up to innovate in Chronic disease care and self-management
may grow to fulfill this role.
Lesson for Canada - Installing the computer into the practice with appropriate
infrastructure is the easy bit. What is less easy and traditionally less well done is to help
the practices to become more able with electronic clinical data. All the benefits derive
from high quality data. To achieve this requires a complex and personalised mix of
usability, infrastructure and culture.
1.7. System Support
Traditionally, most primary care clinical systems, their hardware, software and a
maintenance contract are sourced from a single clinical system vendor. In all three
countries in the past two decades this has slowly changed so that buyers can now buy
these items independently. The cost, for this freedom, however, is that clinics
embracing technology have had to become incredibly technically literate or have had to
hire specific IT staff. For larger clinics this may be a reasonable administrative cost;
not so much for those that are smaller. Often the response to this issue is for
practitioners to work in groups. However, this discriminates against those practitioners
choosing to work in the more isolated, rural regions where having a practitioner is a
luxury and a group of them merely a pipe-dream.
In the UK, hosted systems are becoming the norm. Clinical System, data,
infrastructure and maintenance are managed centrally. To date, this has not been
possible to replicate in Australia due to the high capacity internet requirements that
result. However, the Australian National Broadband Network [29] which will be
installed over the next 2-3 years may make this a more feasible option reducing the cost
of ownership, mean time between failures (MTBF) and also removing the need for
specialized knowledge at the clinic. Since the implementation of the 2003 Contract UK
GPs do not pay for their clinical systems and choose them from a ‘guaranteed’ choice
of conformant vendors. The Primary care trust / commissioning group are centrally
funded to provide these systems. Australian General Practice has not moved towards a
contractual change and still works on a strict ‘fee for service’ model with the GP
Practice responsible for their own IT provision. A national incentive programme ePIP
[30] will come into operation in January 2013 and focuses on laying sound foundations
for the uptake of patient identifiers as well as coding of diagnosis and electronic
prescribing.
Lesson for Canada - Australian infrastructure requirements and limitations

closely resemble those of Canada given the similar geographical conditions and rural,
isolated communities. Successful implementation, however, requires vendor systems to
be based around current state of the art practices and design. The approach, however,
also raises many privacy, security and confidentiality issues discussed below. At a
more fundamental level, financial incentives have a history of accelerating change.
1.8. Security & Confidentiality
Despite the plethora of legislation, good practice guidelines, major international

scandals and professional codes of conduct; security and confidentiality issues remain a
major concern. Medical education facilities include little training in recording the
encounter with a patient in an electronic record let alone providing training in security
and confidentiality requirements. It is a common occurrence for nurses to be logging in
to systems “as the doctor” because the doctor has delegated them with a responsibility
beyond their legal scope of practice (which is enforced by the system parameters).
Rather than developing an understanding that the system is designed to protect them
from inappropriate working practices staff simply find ways round the perceived blocks.
To address this issue and the wider implications of security and privacy, the UK
NHS implemented the national Smartcard [31] which is a physical card that must be
inserted and a matching PIN entered (Just like your bank card) before you can access
services available through the ‘spine’. This makes user authentication much simpler
while increasing security. This ‘role-based’ implementation required a single national
security architecture with a single set of roles, a directory of all NHS personnel and the
setting up of ‘registration authorities’ within each organization in the NHS to manage
card administration, job and role changes etc.
In contrast both Australia and Canada currently rely on username and password.
Australia is in the process of setting up a National Authentication Service for Health
[33] although this has been significantly delayed [34].
Hosted systems allow more actors to work with a record but risk divulging
unauthorized information to unauthorized people without excellence in both privacy
and its enforcement at the point of care. A single unique ID per patient is vital to ensure
data is linked to the appropriate person. Consumer control will be resisted but may be a
way forward
Lesson for Canada – There is considerable maturity of thinking in this area and
care should be taken to learn from both the successes and errors in England and
Australia. The requirements are virtually identical and implementation need not be
difficult if a mature and informed national approach is taken.
1.9. Research
Enormous investments in EHRs have been made over the last forty years in the belief
that they will bring cash releasing financial savings to an over-burdened health care
system, address issues of continuity of care by providing continuity of information, and

improve patient care. To date, remarkably little research has been conducted, or
replicated, on just what is the impact that these systems are having. In the last decade
we have seen an increase in the recognition that such systems may result in unintended
consequences and that these may in fact be negative, but still little rigorous research is
undertaken, funded or published. Work in progress in Australia is starting to addesss
this area [32]
The General Practice Extraction Service [10] and PRIMIS [35] are important
exceptions to this. Both programmes are being undertaken in the UK and are based on
the fundamental tenet that research data should be derived from data collected at the
point of care wherever possible rather than being ‘something extra’. As a result these
programmes are set to achieve success in demonstrating that the appropriate use of
electronic records can improve patient care in terms of adherence to clinical practice
guidelines and protocols.
The basic skills of data and information proficiency are still lacking in many health
care providers portfolios. This is changing slowly as younger generations come into the
workplace for whom the internet and similar technologies have always been a
component of their working practices. Unfortunately, being able to use a technology
doesn’t always mean that they understand it or can self-determine which information
should be shared and under what circumstances. Rather there is an increasing sense in
academic circles that this new generation relies on technology to the exclusion of their
own mind. This has resulted in some medical schools banning the use of hand-held
devices during training, except in select circumstances, as students were becoming too
reliant on a programme designed to support clinical diagnosis and were forgetting to
attend to the actual patient in front of them. This is an interesting position given the
opposing informatics opinion long expressed that “Knowledge should be held in tools
that are kept up to date and used routinely—not in heads, which are expensive to load
and faulty in the retention and processing of knowledge”
2. Conclusions
For over three decades now the potential benefits of computerization in health care
have been touted. Despite this and despite billions of dollars of public funding being
invested, the fruits of these systems have not yet been fully realized. We believe that
until the nine lessons above are truly learned that this will be the status quo for another
quarter century. This, for those of us that work in health informatics and truly believe
in its capabilities when used appropriately is intensely disheartening. We hope that by
laying out these issues and pointing users in the direction of progress in these areas,
where it has been made, that internationally we can pick up the pace and put the
building blocks in place that will mean that Canada will truly be providing the world
class health care that we know its health care workers are striving to provide.
References
[1] Ellis NT, Kidd MR. General practice computerisation: lessons from the United Kingdom. Med J Aust
2000; 172(1):22-24.
[2] Kidd MR. The computerisation of Australian general practice 1998–2001 – what did we get for AU$15
000 000? Informatics in Primary Care 2002; 10:25-29.
[3] UKCHIP The UK Council for Health Informatics Professionals [ 2012 [cited 2012 Oct 31]: Available
from: URL:http://www.ukchip.org
[4] NHS. eICE (Embedding Informatics in Clinical Education [ 2012 [cited 2012 Oct 31]: Available
from: URL:http://learning.nhslocal.nhs.uk/courses/careers/health-informatics/eice-embedding-
informatics-clinical-education
[5] NHS. NHS Connecting for Health. NHS [ 2012 [cited 2012 Sept. 14]; Available from:
URL:http://www.connectingforhealth.nhs.uk/
[6] EN 13606. The EN13606 Association [ 2012 [cited 2012 Oct 31]: Available from:
URL:http://www.en13606.org/
[7] Open EHR GEHR- The good European Health Record[ 2012 [cited 2012 Oct 31]: Available from:
URL: http://www.openehr.org/new-ws/site/resources/gehr
[8] HL7 Health LEvel Seven International [ 2012 [cited 2012 Oct 31]: Available from:
URL:http://www.hl7.org/
[9] IHTSDO International Health Terminology Standards Development Organisation [ 2012 [cited 2012
Oct 31]: Available from: URL:http://www.ihtsdo.org/
[10] NHS. General Practice Extraction Service. NHS [ 2012 Available from:
URL:http://www.ic.nhs.uk/gpes
[11] NHS. GP Systems of Choice (GPSoC). NHS [ 2012 [cited 2012 Sept. 14]; Available from:
URL:http://www.connectingforhealth.nhs.uk/systemsandservices/gpsupport/gpsoc
[12] Healthbase Australia Personally Controlled Electronic Health Record [ 2012 [cited 2012 Oct 31]:
Available from: URL:http://www.healthbase.info/pcehr/page5/page5.html
[14] DOHA About the PCEHR System [ 2012 [cited 2012 Oct 31]: Available from:
URL:http://www.ehealthinfo.gov.au/personally-controlled-electronic-health-records/about-the-pcehr-
system
[15] Dolin R, et al The HL7 Clinical Document Architecture Am Med Inform Assoc. 2006 Jan-Feb; 13(1):
30–39. doi: 10.1197/jamia.M1888
[16] Beeler JW, HL7 version 3--an object-oriented methodology for collaborative standards development.
Int J Med Inform. 1998 Feb;48(1-3):151-61.
[17] Chisholm J. The Read clinical classification. BMJ 1990; 300(6732):1092.
[18] IHTSDO Members of IHTSDO [ 2012 [cited 2012 Oct 31]: Available from: URL:
http://www.ihtsdo.org/members/
[19] DH (England) Common clinical languague for NHS will help improve patient care and safety[ 2012
[cited 2012 Oct 31]: Available from: URL:http://mediacentre.dh.gov.uk/2011/08/16/common-clinical-
language-for-nhs/
[20] HL7. Clinical Document Architecture (CDA). HL7 [ 2012 [cited 2012 Sept. 14]; Available from:
URL:http://wiki.hl7.org/index.php?title=Product_CDA
[21] NHS GP2GP [ 2012 [cited 2012 Oct 31]: Available from: URL:
(http://www.connectingforhealth.nhs.uk/systemsandservices/gpsupport/gp2gp
[22] Song PH, McAlearney AS, Robbins J, McCullough JS. Exploring the Business Case for Ambulatory
Electronic Health Record System Adoption. Journal of Healthcare Management 2011; 56(3):169-180.
[23] Brown M, Shaw NT, Grimm NA, Muttitt S, Gebran J. Electronic Health Records and Patient Safety:
What Lessons Can Canada Learn from the Experience of Others? Healthcare Quarterly 2008;
11(1):112-119.
[24] Brown M, Grimm NA, Shaw NT. The relationship between Electronic Health Records and Patient
Safety: Future Directions for Canada. iCARE, Canada Health Infoway, Canadian Patient Safety
Institute, editors. 2007. Canada Health Infoway.
[25] NHS/ PRIMIS Background to PRIMIS [ 2012 [cited 2012 Oct 31]: Available from:
URL:http://www.primis.nottingham.ac.uk/index.php/about-us/background-information
[26] DOHA eHealth records learning centre [ 2012 [cited 2012 Oct 31]: Available from:
URL:http://publiclearning.ehealth.gov.au/
[27] Improvement foundation eCollaborative-21st Century Patient Care and Self Management [ 2012 [cited
2012 Oct 31]: Available from: URL: http://improve.org.au/our-work/e-collaborative/
[28] RACGP Media Release[ 2012 [cited 2012 Oct 31]: Available from:
URL:http://www.racgp.org.au/yourracgp/news/media-releases/health-minister-announces-$255-
million-funding-to-support-gps-engage-with-the-pcehr/
[29] NBN Case Study [ 2012 [cited 2012 Oct 31]: Available from: URL:http://www.nbn.gov.au/case-
study/dr-stephanie-trust-health/
[30] RACGP Practice Incentive Programme e-health incentive (ePIP) [ 2012 [cited 2012 Oct 31]:
Available from: URL:http://www.racgp.org.au/your-practice/e-health/epip/
[31] Salfordsoftware. NHS Smart Card Authentication. Salfordsoftware [ 2012 [cited 2012 Sept. 14];
Available from: URL:http://www.salfordsoftware.co.uk/identity-management/identity-solutions-for-
the-nhs/nhs-smart-card-authentication
[32] UNSW Use of information and communication technologies to support effective work practice
innovation in the health sector: a multi-site study. [ 2012 [cited 2012 Oct 31]: Available from:
URL:https://research.unsw.edu.au/projects/use-information-and-communication-technologies-support-
effective-work-practice-innovation-h
[33] NeHTA National Authentication Service for Health [ 2012 [cited 2012 Oct 31]: Available from:
URL:http://www.nehta.gov.au/connecting-australia/nash
[34] Delimeter IBM's NASH deal gets terminated [ 2012 [cited 2012 Oct 31]: Available from:
URL:http://delimiter.com.au/2012/10/23/ibms-nash-deal-gets-terminated/
[35] PRIMIS+ Guidelines. 1-114. 2001. Nottingham, UK, PRIMIS+ (Primary Care Information Services).
[36] Weed LL, New connections between medical knowledge and patient care BMJ 1997;315:231


IOS Press, 2013
doi:10.3233/978-1-61499-203-5-37

!"#

#

*

< >

!

Z
\ <^`

*{
*

`

**
|
<

|

} **

|

``

<
**}
|

**
|\

<

<# >
}

}

*
`
` ** <
|
*` | !

`
! \ `! *

** *

}* } `
* < *
*` ` ! *

` `!

<
>

!

!

*
<#

* |

< | *

|> >>
><# >
\

|
>

* *`

*`

<
|

`

<

*!

*`
}

<

**} |
*

!

` < } | | *

*

38 O. Shabestari and A. Roudsari / Challenges in Data Quality Assurance for Electronic Health Records

`

**
> |\

<

}

}
} *
\
***
<
}

`* *
` |\

} | |
*

<

**
!

|

*! *`<

`|}
\* Z

`
*
| | *
}
} }*
!
<
\
!

} |

*
<
\*
` **

*
}
} } *
}
}
! *
}` \
`|<
*

\
*
<

*`

| !
\ | *
**!

<

!

<

**

* | *

<
*
`

`!
`*!

<
}

| *

`
\
|
*

! \
*

| ***

* < *

\! |*
`
`

!

*

<
*
`

}
*

`! |

<

< **

*
**

!
<

!

<

}

*`

*
* <

|
|

|
` *`

*
<

!
**

< |

`
*

}

| \!

< *}

*

!
|
*
< | `

`

<

`!
\|}

*

|
<

` |

`

< #
` |
* !

!

<

<
!
* `

|
|

O. Shabestari and A. Roudsari / Challenges in Data Quality Assurance for Electronic Health Records 39

`!*
<!
**

*
*
<

*

!

} |
! `
<

*

*!

<

<

!
* !

|

| | <

!
<

`
| ** `

{

<

*
! <

*

\| |
!

*

*! ! | * *

`
!
<

* |
| `**

**
!
|

*`

<

<

*
** < `

!

** |
! *

<

* |*}|

* < |* !

**

!
*

`

| *
\
|!\
|*< *!
* |\

`
|
| *

<

|

*
!
`}

Z
<

` *
<

| *

|

*` `
<!`

! |

! *
}

*
<

*
**

*`
!
<
}

|
`! *

<

| !
**
|

*

!

<
^
|

! |
*

<

} | ` ** * !
*

<
* !

` | |

<^`

\

*

*
}
>

`|
|
! | <

40 O. Shabestari and A. Roudsari / Challenges in Data Quality Assurance for Electronic Health Records

}

`
!*

*

>< *
*

! *

*
<
**

*

* !
| |

\ *

<
}
`!\

* * *`

\ *

* ! *

!
|\<
>

*
!
*

*
<
|
!

! ! *

< >

*
! * \|
*
`
\| < !

!

|`

|!
* !

<

* }

|!
* `> *`

!

`
< >
`!

*

< | *

*
`

`<
* !

` | |

O. Shabestari and A. Roudsari / Challenges in Data Quality Assurance for Electronic Health Records 41

<^`

\

>

` |
|
! | <
}

`
! *
*
><

*
*

! *

*
<
**

*

*
!
| |
\
*

<
}
`!\
* * *`

\ *

* ! *

!
|\<
!
>

! !
! * ` |

*` <

`
}
`

*

| ! *

!\

* <
**Z
\*
<

*

*` * <

!

`Z <
"
} *|||<

<
<
! ¡}

<

!
!
"

}
<
¢< } £<< ! }

!

#$
}<
<
}^<#<
}<
}# >

* |\ ¤\
<
%#}<
}'
()*
"+} |} }<

\* ¥¤} }

!

<%-:
-<
^ }¦#} # *
<-: -!}<
\ £}
¦} \ } ` ¤\ } ` £} } ^* `

*
! *
<
$}<
¥ ^}
! ¦#} }

|

`* " ` `
* `
<:
:
$}<
# } #
£} ¤
Z ^} +<

=+<
£}

"}<
¥
} >
!
:<

?

}
"¦}§<
¤} ¨ } ¤! } } ^` !

<:
%}<
} #}

* <
:-%'(}<
¥ #}
¤!
} ¨} | #} #*
}

**
* "¦

` *
* ` `|

<:*'}<

IOS Press, 2013
A Systemic Approach to E.H.R.

Implementation
Bogdan Motoc1
Alberta Health Services
Abstract. E.H.R implementation is a welcome response to the increased

complexity of the health care delivery. An effective implementation relies on
correct systemic understanding of the challenges of diagnostic and treatment as
well as the value of shared information and multi-specialist collaboration. The
paper presents a systemic approach to both health and health care delivery seen
through the perspective of an E.H.R. implementation.
Keywords. Electronic Health Record, Systems thinking, Organization models,

Health Care Economy, specialization, integration, Autopoiesis
The revised paper was not available at the time of publication.
1
Bogdan Motoc, [email protected].
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-43
A Scoping Review on Health Records for

Child-in-Care
Cori THOMPSON1 and Francis Y. LAU
University of Victoria, Victoria, British Columbia, Canada
Abstract. A scoping review was conducted to determine the current state of

knowledge on child-in-care health records in academic literature. Eight studies
describing five such health records were found. Different terms were found
between countries. A key finding from the studies was that research needs to
report on “what worked” to inform policy and practice for positive changes.
Complete, accurate and consistent health records for child-in-care are needed that
can support care and be aggregated to identify policy and practice gaps and
interventions that were effective. Such health records enabled moving from
reactive to proactive care for the child. Social work case data elements found in a
child-in-care health record not included in a child personal health record include:
court dates, dental, abuse, placement, and education. Including these data elements
allows looking at the overall wellbeing and development of the child. With the
exception of two, all studies reported positively on their implementation. Further,
all studies advocated for continued development of a tailored child-in-care health
record. The evidence points toward child-in-care health records as a tool toward
achieving healthy outcomes and policy development.
Keywords. child, in-care, foster, looked after, out of home, health record, review
Introduction
Children-in-care are among the most vulnerable populations and at risk for inadequate
health care compared to other groups of children [1]. When a child is taken into care,
the province or state has a legal obligation to provide reasonable health and dental care
[1-5]. Record keeping is required to provide tailored care to the child-in-care [6], to
procure records for the judiciary system [5,7,8], and for research on interventions for
good health outcomes [9]. The challenge is that “child care [traditionally] lacks an
information-driven culture” [10]. Many health records have been developed for adult
care [11]. Further, intake screening tools [12,13] and systems for child protection
research [9] have been implemented. A health record for a child-in-care needs to
capture additional data to meet statutory care requirements [7,9] and these data are also
needed for current and future health and social care planning [6,7]. Further, in the
United States, maintaining up-to-date foster care health records is legislated [4,7]. To
determine the current state of knowledge on these health records, this review focuses
on published academic studies on paper-based and electronic health records designed
specifically for child-in-care.
1
Corresponding Author - Email: [email protected]
44 C. Thompson and F.Y. Lau / A Scoping Review on Health Records for Child-in-Care
1. Methods
A scoping review was conducted to explore evidence on the current state of the child-
in-care health record in academic journals. Findings would inform whether to proceed
to include grey literature for other systems or to an in-depth systematic review [14].
Two concepts were searched: “child-in-care” and “health record”. The records or
“systems” could be either paper-based or electronic or both. Databases searched were:
Social Services Abstracts, Social Work Abstracts, FAMILY (Informit), Academic
Abstract Complete, and Web of Science (ISI – Web of Knowledge) from 1980 to
August 2012; and Medline (EBSCO) from 2002 to August 2012. Detailed search
protocols are available upon request from the authors.
One reviewer screened the title and abstract of each study for inclusion. Studies
included must 1) detail an electronic or paper-based child-in-care health record used by
providers at point of care, 2) describe a specific information system or record, and 3) be
a primary study. Studies were excluded if they were 1) a commentary or discussion, 2)
a secondary analysis, or 3) a system used for adults. Where it was not clear whether to
include a study, the full article was retrieved and screened. The included studies were
then reviewed.
The studies were reviewed for data elements specific to child-in-care health
records and compared against the data recommended for child personal health record
(PHR) [15]. Child PHR recommendations were chosen as a comparison as PHRs are
longitudinal summaries of key events from all providers, distinct from electronic health
records, and are held by the individual [15]. As a child-in-care moves the record should
move with the child [5,7,19]. Further, close linkages are needed between research,
policy and practice for positive changes [5,16]. Research findings must be presented
within a “what works” framework for research to effectively inform practice and policy
for positive changes [17]. “What works” were defined by the studies and reported as
“results” or “findings” and were thematically aggregated [18].
2. Results
We started with 1264 citations and, after applying the inclusion/exclusion criteria, eight
(8) studies met the criteria and are detailed in Table 1. One study was published in a
medical journal [3], and the other seven studies were published in social work journals.
No studies were published in informatics journals. The studies report on five unique
child-in-care health records. Four records were paper-based; two were paper-based
moving to electronic; one was electronic and generated a paper report; and one was
electronic. Overall, the eight studies reported positive findings on the implementations
and advocated for further research and development. The following sections summarize
the findings in terms of definitions, study descriptions, data elements, and key findings
on what worked for the implementations.
2.1. Definition and Terms
The Child, Family and Community Service Act [2] defines a “child-in-care” as a “child
who is in the custody, care or guardianship of a director or a director of adoption”. In
this context, "child" means “a person under 19 years of age and includes a youth” and
"care” means the “physical care and control of the child”. Different terms were found
C. Thompson and F.Y. Lau / A Scoping Review on Health Records for Child-in-Care 45
to represent the concept “child-in-care” depending on location. “Child-in-care” is used

in Canada [5]. “Looked after” has replaced “in-care” in the United Kingdom
[3,8,10,19]. The United States of America uses “out-of-home care” and “foster care”
[7,20]. Last, Australia uses “in care”, “out-of-home care” and “foster care” [21].
Nuances between terms were not fully stated within studies to allow for comparison.
2.2. Eight child-in-care health record studies.
Table 1 lists the eight studies by first author and publication date (pub date), study aim,
setting and location, participants and records, child-in-care health record (CIC System),
paper or electronic. Adoption levels ranged from not implemented [10] to poor usage
[3] to fully integrated into practice [5,8,19] to able to use record data to generate
reports [7,20,21]. Current levels of adoption are outside the scope of this paper.
Table 1. List of eight studies on child-in-care health records .
First Study Aim Setting; Participants; CIC Record;

Author; Location Records Paper or electronic
pub date
Bundle To identify information gaps Mixed None; Looking After

[3]; 2001 using community child health Residential 36 records for Children;
records against child-in-care Care; 12-16+ age group Paper (inferred)
health record UK
Champion To determine the possibility 32 Community None; Looking After

[21]; 2009 of deriving reliable data from service 614 records for Children Outcomes
the Looking After Children organizations; 0-15+ age group Database;
Assessment and Action Victoria, LAC/A&ARs – paper
Records for outcomes Australia Outcomes database –
monitoring electronic
Hunter To investigate whether a Child care NHS staff, BAAF Health Record
[19]; 2008 specialist nursing service units; children, nursing Booklets
could improve health care for three areas of services; (carer-held health
these children Scotland 162 children in records);
residential care Paper
records
Kerslake Inferred: Evaluation of 12 Local Information Looking After

[10]; 1998 implementation Authorities; Technology, Children;
to resolve difficulties in England, Wales Administrative & Computerizing paper
achieving wider Social Work staff; records
implementation Record count not
stated
Knowles To develop and evaluate use Barnet Interdisciplinary Joint Professional

[8]; 1998 of the Joint Professional Children’s staff ; Record (JPR);
Record Services; 31-39 children of Paper
NHS, UK concern records
First Study Aim Setting; Participants; CIC Record;

Author; Location Records Paper or electronic
pub date
Kufeldt To describe experiences of One location in Social Workers, Looking After

[5]; 2006 one child welfare unit during the 6 Eastern Foster Parents, Children and
Canada’s first National Canadian Youth; Assessment and
Looking After Children and Provinces 37 and 42 AAR Action Records (UK
Assessment and Action records for 10-15+ and Canadian
Records pilot age groups versions);
Paper (inferred)
Lindsay To describe the Care provider Interdisciplinary; Health and Education

[7]; 1993 implementation of the Health consortium; 431 records of Passport for Children
Passport Project; includes a San Diego children in non in out of home care;
content analysis of the health County, CA relative foster care Electronic database,
information stored paper report
Smart To describe CHS’s Department of 2 physician Child Health System

[20]; 1998 development and features, Children and consultants, public (Social Worker system
with a brief overview of the Family health nurses, for health info and
2,688 cases entered during Services; staff; services);
first 7 months Los Angeles 2,688 records of Electronic
County children receiving
protective services
2.3. Child-in-care health record data elements
Child-in-care records contain additional data elements beyond those in a child PHR.
Data elements recommended for a child PHR include: demographic and insurance
information; contact information for family members, other support providers and
health care providers; advanced directives; and clinical information such as: problem
lists, encounters, procedures, chronic conditions, medications, immunizations, and
allergies, vital signs of weight, health, length, body mass index, laboratory results,
family history, birth history, durable medical equipment and supplies [15]. The child-
in-care health record also needs to include: placement information (e.g. at a foster
home), dental, abuse, education and court dates, among other data elements. These data
allow looking at the overall wellbeing and development of the child [21]. For instance,
low education level and high placement changes could be seen as co-existing factor
[21], multiple placements and behaviour issues may be identified concurrently [5],
medical history is known when planning for child adoption [5]. Further, at an aggregate
level, the data can inform policy and practice. For example, a service need for a
therapeutic environment for children who have experienced trauma [21].
3. Discussion
The paucity of studies that met the inclusion criteria suggests that there has been little
development specifically on child-in-care health records; however, studies were found
on other child records: pediatric, school, and personal health records. These records
might contribute to the further development of the child-in-care health record. Of the
eight studies included, three provided few details on the record or implementation
C. Thompson and F.Y. Lau / A Scoping Review on Health Records for Child-in-Care 47
[3,10,19] and two studies did not report positively on the implementation [3,10]. Four
of the eight studies were on the Looking After Children records. All studies anticipated or
reported benefits from using the records. One study reported that using the health
records enabled moving from reactive to proactive care for the child [5]. None of the
studies indicated that the records would no longer be used, except in Smart et al where
the system was decommissioned and replaced by a state-wide system. As several
studies referenced child-in-care health records not found during this review, a fuller
review is needed.
A key concept in the studies was the joining of the social work case management
record and child health status record [7]. Range in integration reached from one study
with six source records [3] to two studies with single source records and ability for
aggregate reporting [7,21] to a real time record with aggregate reporting ability [20].
There is clear direction to link research to policy and practice for evidenced-based
policy and practice changes. For example, identifying factors associated with
developmental progress or deterioration while in care [21]. Interagency cooperation
was found to be integral to developing and effective using the record in all the studies.
Further, to support individual care and research, complete, accurate and consistent
records are needed. Success factors to better recordkeeping included support roles,
audits, education, and results reporting.
4. Study Limitations
Review limitations include a single reviewer did the search, analysis and synthesis so
there could be selection, synthesis and reporting bias. Only studies in English were
included. No formal quality assessment of the eight studies was done. For terms,
through searching, it was found that “child welfare” may reach beyond the connotation
of “a child-in-care” to the broader connotation of overall child health and wellness. In
addition, all the terms relating to “child-in-care” may not have been included in the
searches.
5. Conclusion
There has been some implementation of paper-based and computerized child-in-care

health records. The data elements needed in a child-in-care health record reach beyond
those recommended for a child PHR to include social work case management and
education data [20]. These additional data support holistic care planning for the child.
Several studies highlighted the need for data to inform policy and practice. Further
development of the child-in-care health record needs to integrate research and care
provision data requirements. This step will require an interdisciplinary approach. This
scoping review provides the first step toward determining the current knowledge on the
child-in-care health record. A fuller scoping review is needed to provide more evidence
to develop a tailored tool that supports both surveillance of child-in-care holistic
development and enables research to inform policy and practice changes.
References
[1] Kools S, Kennedy C. Foster child health and development: implications for primary care. Pediatr Nurs.
2003; 29(1): 39-46.
[2] Child, Family and Community Service Act, RSBC 1996 c 46.
[3] Bundle A. Health of teenagers in residential care: comparison of data held by care staff with data in
community child health records. Arch Dis Child. 2001; 84(1): 10-14.
[4] Risley-Curtiss C, Kronenfeld JJ. Health care policies for children in out-of-home care. Child Welfare.
2001; 80(3): 325-350.
[5] Kufeldt K, McGilligan L, Klein R, Rideout S. The Looking After Children assessment process:
promoting resilient children and resilient workers. Families in Society. 2006; 87(4): 565-574.
[6] Kelly S. Obtaining children's health information. Adoption & Fostering. 2007; 31(3): 80-82.
[7] Lindsay S, Chadwick, D, Landsverk J, Pierce E. A computerized health and education passport for
children in out-of-home care: the San Diego model. Child Welfare.1993; 72(6): 581-594.
[8] Knowles R, Blackburn M, Zahir M, Russell M, Carrier A, Nevrkla E. The implementation of a new
parallel child health record. Child Care Health Dev. 1999; 25(4): 253-266.
[9] Sternberg KJ, Knutson JF, Lamb ME, Baradaran LP, Nolan CM, Flanzer S. The child maltreatment log:
a computer-based program for describing research samples. Child Maltreat. 2004; 9(1): 30-48.
[10] Kerslake A. Computerization of the Looking After Children records: issues of implementation.
Children & Society.1998; 12(3): 239-237.
[11] Kim GR. (2011). Snapshots of child health and information technology. In: C. Lehmann, GR Kim, &
KB Johnson (Eds.), Pediatric Informatics: Computer Applications in Child Health. (pp. 3-4). New York,
NY: Springer.
[12] Anderson RL, Lyons JS, Giles DM, Price JA, Estle G. Reliability of the child and adolescent needs and
strengths-mental health (CANS-MH) scale. Journal of Child and Family Studies. 2003; 12(3): 279-289.
[13] Jee SH, Halterman JS, Szilagyi M., Conn AM, Alpert-Gillis L, Szilagyi PG. Use of a brief standardized
screening instrument in a primary care setting to enhance detection of social-emotional problems
among youth in foster care. Acad Pediatr. 2011; 11(5): 409-413.
[14] Rumrill PD, Fitzgerald SM, Merchant WR. Using scoping literature reviews as a means of
understanding and interpreting existing literature. Work. 2010; 35(3): 399-404.
[15] Zuckerman AE, Kim GR. (2011). Personal Health Records. In: C. Lehmann, GR Kim, & KB Johnson
(Eds), Pediatric Informatics: Computer Applications in Child Health. (pp. 293-301). New York, NY:
Springer.
[16] Davies C. (2003). Policy development: making research count. In: K. Kufeldt & B. McKenzie (Eds.),
Child welfare: Connecting research, policy and practice (pp. 377-386). Waterloo, ON: Wilfrid Laurier
Press.
[17] Jones H. (2003). The relationship between research, policy, and practice in delivering an outcome-led
child welfare service. In: K. Kufeldt & B. McKenzie (Eds.), Child welfare: Connecting research,
policy and practice (pp. 367-376). Waterloo, ON: Wilfrid Laurier Press.
[18] Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews.
BMC Med Res Methodol. 2008; 8(1): 45-__.
[19] Hunter D, McCartney G, Fleming S, Guy F. Improving the health of looked after children in Scotland.
Adoption and Fostering. 2008; 32(4): 51-56.
[20] Smart J, Russell J, Custodio C. Developing a computerized health record in a protective services system.
Child Welfare. 1998; 77(3): 347-362.
[21] Champion R, Wise S. Developing outcomes-based data from Looking After Children case records
completed for children in care in Victoria, Australia. Vulnerable Children and Youth Studies. 2009;
4(2): 107-113.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-49
Application of the Technological

Pedagogical Content Knowledge
Framework in Integrating an Educational
EMR into Health Informatics Education
Jesdeep BASSI1, Andre W. KUSHNIRUK, Elizabeth M. BORYCKI
School of Health Information Science, University of Victoria, British Columbia,
Canada
Abstract. The discipline of health informatics is highly immersed in information

technology, specifically health information systems. Students graduating from
Bachelor degree programs in health informatics are expected to be familiar with a
variety of systems upon entering the workforce. The adoption of systems like
electronic medical records is on the rise across Canada, therefore it would be
highly beneficial for students to have exposure to such systems in their coursework.
While some individual instructors have done this to some extent on an ad hoc basis,
formal strategies for EMR integration do not exist. A prominent framework for
technology integration in learning that has been applied in many scientific
disciplines is the Technological Pedagogical Content Knowledge (TPCK)
framework. This paper describes how TPCK was used and applied as the guiding
conceptual framework for exploring the integration of an educational EMR into
undergraduate health informatics education.
Keywords. electronic medical record, electronic health record, health informatics,

education, technological pedagogical content knowledge
Introduction
According to COACH, Canada’s Health Informatics Association, “health informatics

professionals develop and deploy information and systems solutions, drawing on expert
knowledge from fields such as computer science, information management, cognitive
science, communications, epidemiology, management sciences and health sciences” (p.
7)[1]. This statement demonstrates two key needs for students training to become
health informatics professionals: 1) the need to be familiar with information technology
(IT) and 2) the need to be able to apply it in many areas.
Several types of health information systems (HIS) are introduced in undergraduate
health informatics programs. Electronic health record (EHR), computerized patient
record (CPR), and electronic medical record (EMR) are all terms used to describe
systems used to manage patient records. The latter, EMR, is the term used in the
Canadian context to describe systems used in physician offices [2]. Adoption of EMRs
is on the rise in Canada. For example in British Columbia, the Physician Information
1
Jesdeep Bassi, Email: [email protected]
50 J. Bassi et al. / Application of the TPCK Framework in Integrating an Educational EMR
Technology Office (PITO) was established to support the implementation of EMRs in

physician office practices [3]. In preparing health informatics students to enter the
workforce upon graduation, it is important that they obtain competencies in relation to
EMRs and other types of HIS. However, a key challenge is effectively integrating the
technology into the classroom. As Mishra and Koehler explain, “merely introducing
technology to the educational process is not enough”(p. 1018)[4]. The technological
artifact is a tool but it needs to be used correctly in order to produce good outcomes.
This challenge is not specific to the field of health informatics, but it does possess some
unique contextual factors which need to be explored and addressed. Thus far, EMRs
have been used to some extent on an ad hoc basis by individual instructors but there
has not been a concentrated effort to explore integration components in depth. Research
was recently undertaken to address this gap [5].
The integration of technology in education is not a new concept. Technology has
become a large part of the learning environment. From the basic use of Microsoft®
PowerPoint to more discipline-specific technologies, the use of technology in education
is now commonplace and the majority of students entering university have grown up
surrounded by IT in daily life. Looking at theoretical foundations for technology
integration, a prominent concept in the literature is the Technological Pedagogical
Content Knowledge (TPCK) framework. TPCK has been applied in many scientific
disciplines and was used in this research as the guiding conceptual framework for
exploring the integration of EMRs into undergraduate health informatics education.
The result was a unique application of TPCK for this specific integration context.
1. What is TPCK?
Technological Pedagogical Content Knowledge represents a form of teacher

knowledge which is an extension of Shulman’s Pedagogical Content Knowledge
(PCK) idea proposed in the 1980s [4]. PCK emphasized a need to understand how
pedagogy fits with content for teaching. Given that technology now plays a large role
in teaching as well, Mishra and Koehler added a third construct or knowledge base of
technology to form TPCK[4] (also referred to as TPACK). TPCK is illustrated as a
Venn diagram (see Figure 1) which shows several types of knowledge are involved.
First is the knowledge of each individual construct: content, pedagogy, and technology.
Next is the knowledge of each pair of constructs. In addition to the original idea of
PCK, there is TPK and TCK. Technological pedagogical knowledge (TPK) requires an
understanding of how pedagogy changes according to particular technologies in
relation to the benefits and limitations of the technology within learning activities. This
requires the teacher to consider which technology tools are best suited to specific
activities, even those not specifically designed for learning. Technological content
knowledge (TCK) refers to knowledge about how content is influenced and constrained
by technology and vice versa. This requires consideration of which technology is best
suited to specific content being taught. Finally, TPCK in the center is the knowledge
that represents the interaction of all three constructs together. Surrounding this is a
circle label “contexts” which plays a large role in any integration endeavor. The
external factors and circumstances that influence integration will also have an impact
on TPCK (e.g. the classroom environment and curriculum).
J. Bassi et al. / Application of the TPCK Framework in Integrating an Educational EMR 51
Figure 1. Original TPCK (left) (Source: http://tpack.org/) and TPCK in context of research (boxes at right).
2. TPCK in the Research Context
Each integration context is unique. In exploring EMR use for health informatics
education, the three knowledge bases required were defined as follows.
2.1. Technology
In the real-world, EMRs have many functionalities to support healthcare. Common

components include documentation support, decision support features such as
reminders and alerts, administrative support, test and imaging ordering, results
management, prescribing and medication management, and reporting [6]. From a
design perspective, components include the back-end database, programming, and
interface design. For this research, a modified version of an EMR called Digital Health
Designs EMR® was the technological artifact being explored for use in education. In
addition to being a fully functional EMR, it has added features for learning such as
different users modes for instructors and students that allow instructors to control
fictional patient cases within the system [7].
2.2. Pedagogy
Pedagogy refers to the practice of teaching. Instructors employ a variety of teaching

activities, methods, strategies or approaches according to their chosen pedagogy. The
specific teaching and learning activities can be categorized into one of the following
[8]:
Instructor-centered: instructor primarily passes on information to students
Interactive: learning through communication between instructor and
student and among students
Individualized: student works at their own pace individually
Experiential: learning takes place in natural or simulated settings
Assessment spans all four categories, with the assessment method reflecting the
nature of the activity. For example, an instructor-centered assessment method may be a
52 J. Bassi et al. / Application of the TPCK Framework in Integrating an Educational EMR
written test whereas an interactive activity may be assessed through an oral

presentation.
2.3. Content
The content piece reflects the competencies or knowledge that students are expected to
possess. For health informatics, several groups have established sets of competencies
for the health informatics professional that outline both the knowledge and skills
graduating students should have. In Canada, COACH produced the Health Informatics
Professionalism (HIP™) Competency Framework [9]. In this framework, the core body
of knowledge for health informatics comes from three source practices: health sciences,
information sciences, and management sciences. Within these three areas are seven
subcategories: clinical and health sciences; Canadian health system; information
technology; information management; project management; organizational and
behavioral management; and analysis and evaluation. The specific competencies for a
health informatics professional are organized under each of these subcategories.
3. Resulting Application of TPCK
This research utilized TPCK, both in study design and the resulting output. Previous
authors have expressed that technology should not drive pedagogy [10,11] and in
specifically referring to TPCK, Harris and Hofer[12] express that TPCK shouldn’t be
“technocentric”, that is, focused on the technology. They have advocated for the
exploration of technology to best suit learning needs. While in this research the
technology was preselected, the goal was not to fit the health informatics curriculum
around the educational EMR, but rather to determine where it fits best into the
curriculum.
In exploring EMR integration into health informatics education, the TPCK
framework was used to frame the research questions as well as instrument design i.e.
interview and focus group questions. Given that each construct could be defined for the
health informatics context, the next step was to look at each of the pieces in relation to
each other to effectively uncover all the considerations (i.e. types of knowledge)
required for effective integration in this context (see boxes at right in Figure 1). For
example, what learning activities (pedagogy) could employ the educational EMR
(technology) to give the instructor TPK? Or what topics (content) covered in health
informatics courses are directly related to EMRs (technology) to give the instructor
TCK? Instructors and students were recruited to provide answers to these questions (in
interviews and focus groups) using TPCK as a guide.
The result of this research is a proposed framework for integration that builds on
and extends the original TPCK framework [5]. The items identified in the research
within each knowledge area (technology, pedagogy, content) are included with two
additional pieces. The first added piece is a continuum for when to integrate the
technology into the education. From the research, points of integration emerged which
ranged from a single activity to integration across multiple programs including health
informatics in a multidisciplinary approach. The second added piece is contextual
considerations that emerged which “open-up” the context circle surrounding TPCK.
These include course, student, instructor, technical, and system considerations as well
as overall learning pedagogy.
J. Bassi et al. / Application of the TPCK Framework in Integrating an Educational EMR 53
Conclusion
In exploring the integration of educational EMRs into health informatics education,

TPCK forms a useful approach to guide educators. As the creators of TPCK state,
“there is no general solution to a teaching problem for every context, every subject
matter, every technology, or every classroom” (p. 20)[13]. However, TPCK provided a
good basis for exploration of potential solutions and in turn, the extended framework
developed through this research for health informatics education now provides a
mechanism for health informatics programs to explore integration of educational EMRs
into their own curriculum, regardless of location and the specific system involved.
Acknowledgements
This work was supported by a University of Victoria Graduate Scholarship and a

Pacific Century Graduate Scholarship from the Province of British Columbia granted
through the Ministry of Advanced Education.
References
[1] COACH Canada’s Health Informatics Association (COACH). Health Informatics Professional Core
Competencies Version 1.0. [Internet]. 2007. [cited 2009 Mar 17]. Available from:
http://www.coachorg.com/career_development/professionalism/core_competencies.htm
[2] Nagle LM. Informatics: emerging concepts and issues. Nurs Leadersh. 2007; 20(1): 30-32.
[3] Physician Information Technology Office (PITO). About PITO. [Internet]. 2009. [cited 2012 Aug 25].
Available from http://www.pito.bc.ca/about-pito/
[4] Mishra P, Koehler, MJ. Technological pedagogical content knowledge: A framework for teacher
knowledge. Teacher’s College Record. 2006; 108(6): 1017-1054.
[5] Bassi J. Towards the development of a framework for integration of an electronic medical record into
an undergraduate health informatics curriculum. [M.Sc. Thesis]. University of Victoria. 2011.
[6] Carter JH (Ed.). Electronic Medical Records: A Guide for Physicians and Administrators. Philadelphia:
American College of Physicians-American Society of Internal Medicine, 2001.
[7] Joe RS, Kushniruk AW, Borycki EM, Armstrong B, Otto T, Ho K. Bringing electronic patient records
into health professional education: Software architecture and implementation. Stud Health Technol
Inform. 2004; 150: 888-892.
[8] Weston C, Cranton PA. Selecting instructional strategies. The Journal of Higher Education. 1986;
57(3): 259-288.
[9] COACH Canada’s Health Informatics Association (COACH). Health Informatics Professional Core
Competencies Version 2.0. [Internet]. 2009. [cited 2009 Jun 20]. Available from
http://www.coachorg.com/career_development/professionalism/core_competencies.htm
[10] Lai K. ICT Supporting the Learning Process: The Premise, Reality, and Promise, In J Voogt & G
Knezek (Eds.), International Handbook of Information Technology in Primary and Secondary
Education. New York: Springer Science + Business Media; 2008: 215-230.
[11] Jefferies P. ICT in supporting collaborative learning: Pedagogy and practice. Learning, Media and
Technology. 2003; 28(1): 35-48.
[12] Harris J, Hofer M. Instructional Planning Activity Types as Vehicles for Curriculum-based TPACK
Development, In CD Maddux (Ed.), Research Highlights in Technology and Teacher Education 2009.
Chesapeake: Society for Information Technology in Teacher Education (SITE); 2009: 99-108.
[13] Koehler MJ, Mishra P. Introducing TPCK. In AACTE Committee on Innovation and Technology (Ed.),
Handbook of Technological Pedagogical Content Knowledge (TPCK) for Educators. New York:
Routledge; 2008: 3-29.
Telecare, Telemedicine and Telelaboratory
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-57
TeMaD System: Telecare for Managing

Diabetes in Saudi Arabia
Khulud ALKADIa,b1, Abdul ROUDSARIa,c
a
Centre for Heath Informatics, City University, London, UK
b
King Abdulaziz Medical City, National Guard Health Affairs, Riyadh, Saudi Arabia
c
School of Health Information Science, University of Victoria, Canada
Abstract. This paper briefly describes the main characteristics of the TeMaD
system, developed for the Saudi National Guard Hospital in Riyadh. TeMaD
attempts to improve current healthcare services for diabetic patients, and assists
healthcare givers in disease management. It strengthens communication channels
between patients and their healthcare givers, possibly leading to better health.
Keywords. Saudi Arabia, Telecare, TeMaD, Diabetes, KAMC, National Guard
Introduction
Diabetes is rapidly spreading in Saudi Arabia, affecting almost one third of the
population [1]. The healthcare sector is struggling to cope with the growing numbers of
diabetics and associated healthcare costs, nor is it prepared to deal with the expected
rise in chronic diseases due to an aging population [2]. Reasons fuelling this problem
include poor dietary habits, inactivity, an aging population and associated social factors.
The Telecare for Managing Diabetes (TeMaD) system has been developed to help
improve healthcare services offered at King Abdulaziz Medical City (KAMC) in the
capital Riyadh. It presents a telecare system that is tailored for patients in Saudi Arabia
in order to overcome specific obstacles sometimes linked to local culture and traditions
[3]. TeMaD is also a highly available system offering a number of practical
communication channels between the patients and their healthcare givers.
This paper reviews the main characteristics of TeMaD, and examines the impact it
has had on diabetic patients, during a three month trial period between 2009 and 2010.
1. Review of Current Healthcare Services Offered at KAMC Diabetic Clinic
The KAMC Diabetes clinic provides healthcare services through Physicians,

Diabetic Educators (DEs), and Nurses. DEs are primarily responsible for educating
patients about the disease, and for long-term monitoring of blood glucose levels,
medications, and other associated factors such as diet and physical activity regiments.
More than 1500 diabetics were registered at the clinic during 2008/2009, with only
four DEs. Since then, these numbers have risen considerably. Unfortunately the
1
58 K. Alkadi and A. Roudsari / TeMaD System: Telecare for Managing Diabetes in Saudi Arabia
number of DEs remains the same, leading to a strain on hospital resources. Furthermore,
blood glucose levels are recorded and monitored in an inefficient manner. Patients log
their readings in a small booklet, kept with the patient at all times. During the hospital
visit (may vary from once a month to once a week depending on the patient), the DE
reviews all recorded readings and discusses any anomalies with the patient. Based on
this information, the DE determines the next step. The DE has no access to these
readings between hospital visits, which may put high risk patients in danger, delaying
required medical intervention when necessary.
2. Common Obstacles in Saudi faced by Patients
In addition to existing obstacles commonly found in other countries, we find the

following to play a major role:
2.1. Transportation
Saudi Arabia lacks an adequate public transportation system. In addition, Saudi law
prohibits females from driving. This combination has resulted in ‘Transportation’ being
amongst the highest obstacles in receiving healthcare services as rated by KAMC
patients during a survey conducted in 2010, mostly affecting female patients.
2.2. Cultural Factors
Saudi has a unique culture which combines social, traditional, and religious factors. It
is common for patients to insist on receiving care from same-sex healthcare givers.
Society’s strong endorsement of gender-segregation may also contribute to the problem.
This may be difficult to satisfy, especially in rural centres with limited resources.
3. TeMaD Design
TeMaD targets diabetic patients at KAMC Clinic between 16-70 years. It allows
patients and healthcare givers at the clinic to communicate using 3 different modes:
Internet, SMS, and Dedicated Landline. These modes make TeMaD an accessible
solution, allowing patients from different age groups to select the most suitable one.
Let us review the TeMaD interfaces for both Patients (Users) and DEs (Administrators).
3.1. TeMaD Patient Modes
TeMaD offers patients three different modes of access. Each one targets a different age
group and offers different functionality, making TeMaD more accessible.
3.1.1. TeMaD Internet mode

The online mode offers patients a user-friendly interface with clear and simple design
and structure making data entry easy (see Figure 1). It offers the most functionality,
allowing users to monitor their blood glucose levels over a period of time using
graphical representations. It is expected to draw younger patients.
K. Alkadi and A. Roudsari / TeMaD System: Telecare for Managing Diabetes in Saudi Arabia 59
Figure 1. TeMaD Patient Web Interface
3.1.2. TeMaD SMS mode

This mode enables patients to send daily readings (up to seven) to the TeMaD system
through SMS using a local number. This mode is easy to use, and accessible to any
patient owning a mobile. It is expected to draw middle-aged patients.
3.1.3. TeMaD Landline mode

A DE manages a dedicated landline, responding to patient calls and directly logging
their readings and any other necessary information onto TeMaD. This mode was added
for two primary reasons: direct verbal communication between DE and patient may
enable capture of essential information, and to also encourage older patients, not
particularly interested in using technology, to use TeMaD.
3.2. TeMaD DE Mode
The DE uses the TeMaD Web Interface to manage patients’ diabetes. The interface is
user-friendly (see Figure 2), and enables DEs to perform various tasks, such as:
Review existing readings uploaded by patients via Web Interface and/or SMS
Log patient readings collected during a landline call
Examine patient performance levels over a period of time (daily, weekly,
monthly, 3-month)
Communicate with patient (SMS sent via TeMaD)
4. TeMaD Study
TeMaD was used to manage the diabetes of 79 participating patients during the period
December 2009 and June 2010. Random sampling is used, with ages between 16 to 70
years. Only 52 patients completed the trial period while 27 patients dropped out for
various reasons.
Figure 2. TeMaD DE Web Interface

Due to limitations in time and budget, the sample size is small; however, it does reflect
the total population of patients at the clinic, in gender and age group ratios. The HbA1c
test is used to determine the impact of TeMaD. The HbA1c test is chosen because it
measures blood sugar levels over a longer period, hence better assessment of a new
treatment plan. Each patient used TeMaD for 12 weeks, since blood cells may survive
8-12 weeks before renewal. This ensures an accurate follow-up HbA1c measurement is
taken after the trial period. Data was later examined to determine the impact of TeMaD
as an alternative method for diabetes management. The Before/After test was used,
comparing HbA1c levels of our sample group before and after intervention, where:
Pre-measurement: the value of the blood sugar level of a participating patient
obtained through an HbA1c test, before using TeMaD.
Intervention: using TeMaD for a period of 12 weeks.
Post-measurements: the value of the blood sugar level of a participating patient
obtained through an HbA1c test, after using TeMaD.
The two measurements will be compared to evaluate whether or not TeMaD had
an impact on the blood sugar levels of participating patients.
Upon completion of the study, each patient was required to complete a
questionnaire. The primary goal is to reduce patient HbA1c levels by 5% or more. The
following formula was used:
Reading 1 – Reading 2 = Hba1c change
(Hba1c change / Reading 1) X 100 = % of HbA1c level increase/decrease
where: Reading 1: is the Hba1c level taken before starting TeMaD, and
Reading 2: is the Hba1c level taken after completing TeMaD, and
Hba1c change: is the difference between Reading 1 and Reading 2.
K. Alkadi and A. Roudsari / TeMaD System: Telecare for Managing Diabetes in Saudi Arabia 61
5. Results
5.1. Relevant TeMaD application results
The gender ratio of the total number of participating patients is 46% male and 54%
female, closely resembling the ratio of patients at the clinic between 2009 and 2010,
being 42% male and 58% female. This indicates that our sample size closely reflects
the clinic environment in gender ratio.
Statistics showed that 69% of TeMaD patients fall under the 16-29 age group,
suggesting that younger patients are more likely to use alternative methods, especially
those methods relying on technology.
In addition, only 12% of participating patients lived in rural areas. However, the
urban area of Riyadh spans across 2435km2, with KAMC located at the eastern
outskirts of the city. With almost 5 million inhabitants, peak hours can result in some
patients requiring more than two hours to commute back and forth to the hospital.
TeMaD better suited many patients saving them both time and money.
Most importantly, the primary goal of the study was achieved, with TeMaD
succeeding in reducing HbA1c levels by more than 5% for 83% of participating
patients, with a greater impact on females (see Figure 3). The average HbA1c level of
patients was reduced from 9.2% down to an average of 8.4%.
Furthermore, out of the total number of participating patients recording drops in
their HbA1c levels, approximately 37% were able to lower it by 10% or more,
exceeding our expectations. HbA1c levels recorded three months after patients resumed
using traditional healthcare services indicate that progress made earlier through TeMaD
was reversed with an increased in the HbA1c levels of 42%of revisited patients.
5.2. Relevant TeMaD questionnaire results
Most females rated ‘Transportation’ as a major obstacle in receiving quality

healthcare services, while 29% do not have permanent access to a private car.
Almost 62% of patients preferred dealing with the same-sex DE. This may be
contributed to the sensitive nature of the Saudi society and the adherence to
cultural and traditional factors which promote segregation between the genders.
Statistics indicate that 84.6% of patients found TeMaD to be a more convenient
way of dealing with their diabetes than traditional services at the clinic, while
almost 98% stated that TeMaD assisted in better management of their diabetes.
90.4% of patients stated that TeMaD made it easier to access the clinic DEs.
Figure 3: Post TeMaD HbA1c levels

6. Discussion
TeMaD has improved HbA1c levels of 83% of participating patients reducing the
average from 9.2% to 8.4%. Three months post-trial, tested HbA1c levels showed a rise
in 42% of revisited patients, indicating that TeMaD intervention contributed to
improved patient health. TeMaD had a greater impact on female patients between 16-
29 years, improved monitoring techniques, and patient–caregiver communication
channels. TeMaD may not be suitable for older patients, due to technology resistance.
Also, TeMaD is not recommended for newly diagnosed patients, since it minimises
personal contact with DE, sometimes essential to assess patient psychological aspects.
TeMaD has been useful to DEs in management of patient disease, time, and resources.
7. Conclusion
TeMaD is a telecare solution designed to help meet the growing needs of the Saudi
healthcare system. Although it is developed specifically for diabetes, it can be used to
manage other chronic diseases. TeMaD is tailored to address factors of the local culture
that may affect how healthcare services are offered. More research is required to
examine how technology can assist in overcoming these factors while maintaining local
culture. TeMaD is a highly available system which improved communication between
patients and healthcare givers and led to higher compliance with the diabetic treatment
plan. TeMaD had a positive impact on diabetes management indicating the need to
incorporate such solutions into existing healthcare practices.
8. References
[1] Al-Watan Newspaper, 28% are Diabetic in Saudi, November 16, 2007, 2604: 5.
[2] Gulf Cooperation Council Health Care: Challenges and Opportunities, McKinsey analysis based on
Global Insight data, 2006.
[3] Alkadi, K., Roudsari, A., Applying a methodological approach for a telecare solution in Saudi Arabia,
Stud Health Technol Inform, 2011; 164:243-7.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-63
Telecardiology on Vancouver Island:

Imagination to Implementation
Trent HORWOOD, Mya AYLOTT, Margarita LOYOLA1, Jill HENDERSON, Gary
FRANK, and Mary Lyn FYFE
Vancouver Island Health Authority, British Columbia, Canada
Abstract. In 2011, there were more than 97,000 people living with Heart Failure
in British Columbia (BC) with a total of 17,592 within VIHA. To increase patient
accessibility to specialist care, the Vancouver Island Health Authority (VIHA)
implemented a telecardiology program that utilizes digital stethoscopes, telehealth
technology and collaboration to deliver cardiac care remotely. The program has
successfully completed 20 consultations to date in 6 communities within the
VIHA. This article outlines processes and outcomes of enabling the existing
VIHA cardiology program with the use of telehealth technologies.
Keywords. telehealth, heart failure, cardiology, Vancouver Island,

videoconferencing, digital stethoscope, Vancouver Island Health Authority
(VIHA), Provincial Health Services Authority (PHSA), telecardiology
Introduction
Telehealth refers to the use of communications and information technology to deliver

health and health care services and information over large and small distances. It is
about transmitting voice, data, images and information rather than moving patients,
health practitioners or educators [1]. In 2006, the Vancouver Island Health Authority
(VIHA), an organization in British Columbia (BC) responsible for delivering care to
750,000 people, embraced telehealth as a modality to increase patient accessibility to
specialty care. VIHA telehealth has enabled several specialties (i.e. oncology, thoracic
surgery, renal medicine, mental health, genetics and stroke) and 17 communities
(Figure 1). The service has facilitated over 7,000 physician–patient telehealth
consultations, saving over 2,700,000 km of patient travel.
Heart Failure (HF) is a chronic heart condition where the heart is weakened or
damaged due to disease and/or stress (i.e. myocardial infarction, heart disease,
hypertension and heart valve diseases). In 2011, the Ministry of Health identified
97,214 people living with heart failure in BC, with a total of 17,592 within VIHA [2].
In response to this, VIHA developed a telecardiology service to increase HF patient
accessibility to cardiac care.
1
64 T. Horwood et al. / Telecardiology on Vancouver Island: Imagination to Implementation
Figure 1. VIHA communities equipped with Telehealth
1. Background
The Heart Function Clinic (HFC) provides HF patients with medical and nursing
assessments, education and support, exercise guidance, diagnostic interpretation (i.e.
electrocardiograms and blood tests), medication reconciliation and adjustment.
Currently, there are over 700 VIHA patients registered in the HFC. Many Vancouver
Island residents live in communities that do not have a HFC, and are asked to travel to
other communities to receive care (Table 1) [3]. On average, 7 patients per week
travel to Victoria, Nanaimo or Campbell River to visit a HFC. These patients absorb
the time, expense, risk and physical burden of travelling to the nearest centre.
Table 1. Communities without Heart Function Clinics: populations and distances (round trip) to HFCs
Community Total Distance to Distance to Distance to Campbell
Population Victoria HFC Nanaimo HFC River HFC
(Kms) (Kms) (Kms)
Duncan 57,761 130 54 204
Parksville 46,375 300 36 119
Port Alberni 31,442* 390 81 148
Tofino 632 203 270
Courtenay 65,340 440 106 45
Gold River 2,356* 710 239 92
Port Hardy 12,239* 1004 384 237
Port McNeill 926 347 200
Port Alice 1024 396 249
*Population Combined
2. Program Development
Recognizing the need for increased accessibility of service to patients with complex
cardiac conditions, VIHA in conjunction with representatives from BC health
authorities developed a provincial committee to foresee the implementation of a
standard digital stethoscope that could be used throughout the province. Digital
stethoscopes provide “…reliable and valid screening for congenital heart disease” [4],
T. Horwood et al. / Telecardiology on Vancouver Island: Imagination to Implementation 65
and can be use safely over internet-based software [5]. The intent of the committee was
to ensure compatibility, reliability and knowledge transfer through lessons learned.
2.1. Technical Solution
The electronic Littmann 3M 32000 stethoscope was selected as the provincial standard
for telehealth programs within BC. The electronic stethoscope connects to a computer
using Bluetooth, and links the provider stethoscope with that of the patient at the
remote site. Once connected, providers can utilize the electronic stethoscope to assess
the heart and lung sounds of remote patients.
2.2. Program Participants and Selection
Participation in telecardiology is entirely voluntary. In total, three VIHA cardiologists

volunteered to participate. HF patients meeting the following criteria were given the
option to have their consultation in their local community via telecardiology or to travel
to the nearest HFC for a face-to-face consultation:
lived in or near community where telecardiology is offered;

had their initial visit in person at the HFC; and
agreed to participate in telecardiology.
2.3. Requirements
The telehealth team conducted a thorough analysis of the current processes at the HFC.
The following items were identified as essential when conducting a session: capability
to capture a patient’s vital signs, ability for specialist to hear patient’s heart beat, and
ability to see and hear the patient in real time. The following workflow processes were
also examined: patient inclusion criteria, scheduling, visit types, pre-visit patient
preparation, diagnostic testing, patient assessment workflow, report generation, and
documentation.
3. Program Development
To ensure program success, a dedicated project team including a project manager,

project analyst and telehealth nurse was assembled. The team focused on examining
stakeholder engagement, patient load, resource availability, rollout, risk assessment and
developed a project plan. The team also ensured site readiness in terms of both
infrastructure and human resources (i.e. equipment deployment, testing, and
infrastructure management; and clinical support at the patient site).
3.1. Stakeholder Engagement and training
Human resource engagement was an essential component for the success of the
implementation of telecardiology. Engagement can “…be a primary antecedent to
successfully implementing an organizational change initiative” [6]. All stakeholders,
including executive support, physicians, assistants, and persons facilitating telehealth at
66 T. Horwood et al. / Telecardiology on Vancouver Island: Imagination to Implementation
patient sites, were involved throughout the process to encourage communication,

collaboration and program acceptance.
The use of the electronic stethoscope during a telehealth consultation created the
need for increased training, resources and support than the typical telehealth
consultation. Software, equipment and process training were delivered by the
telehealth team in person and remotely over telehealth. Administrative assistants were
trained in the scheduling and registration processes and nurses and physicians were
trained to facilitate and prepare for the telecardiology session (i.e. flow and operating
the videoconferencing equipment, stethoscope and software). Competency checklists,
completed by the clinicians, were used to track proficiency and gaps in training.
3.2. Support Plan
The telehealth team developed a thorough multi-tier support plan for the various stages
of the project. The plan allowed for stakeholders to communicate issues or questions
through phone or email. If the issue could not be resolved immediately by the receiver,
the issue would be classified and directed to someone with specialized knowledge in
that area.
4. Findings
In the first 6 months of telecardiology, 20 patients from the following communities

participated: Campbell River, Port Alberni, Courtenay, Parksville, Port Hardy and Salt
Spring Island. So far, the service has avoided over 5,800 km of patient travel and
1,600 tones of carbon emissions.
The telehealth team has received positive feedback from participating physicians,
patients and clinicians. Dr. Swiggum (cardiologist) commented that “telecardiology
enhances the collaborative approach to delivering cardiac care to the patient” and that
telecardiology is “… like having the patient sitting next to you”. Several patients have
reported substantial cost, time and travel savings.
5. Next Steps
The success of the pilot and initial sessions has prompted the telecardiology service to
continue and expand. Other Health Authorities are expected to use the knowledge
gained from this project to implement similar use of the digital stethoscope. At VIHA,
the telecardiology model will expand to the Atrial Fibrillation Clinic. The clinic serves
up to 2,200 BC patients annually. Since there are only a few electrophysiology
cardiologists in the province the service is expected to save thousands of kilometers of
patient and provider travel.
VIHA is exploring the use of the digital stethoscope as well as other digital
medical peripherals (i.e. dermatoscope, otoscope and spirometer) to facilitate telehealth
for other disciplines such as respirology, speech language pathology, and dermatology.
T. Horwood et al. / Telecardiology on Vancouver Island: Imagination to Implementation 67
References
[1] J. Picot. Sector Competitiveness Frameworks Series: Telehealth Industry Part 1 –Overview and
Prospects. Industry Canada: Industry Sector Health Industries, Canada Health Infoway definition, 2004.
[2] Medical Service Economic Analysis, Medical Services and Health Human Resources Division, Ministry
of Health, 2011
[3] Local Health Area Profiles. Vancouver Island Health Authority. [cited 2012], Available from:
http://www.viha.ca/mho/stats/lha_profiles.
[4] J. Belmont, L. Mattioli, R. Ardinger, A.C. Thomas, Evaluation of Remote Stethoscopy for Pediatric
Telecardiology. Telemedicine Journal 1(2) (1995), 133-49.
[5] G. Fragasso, M.D. Benedictis, A. Palloshi, M. Moltrasio, A. Cappelletti, M. Carlino, et al, Validation of
heart and lung teleauscultation on an Internet-based system. Am J Cardiol 92(9) (2003):1138-9.
[6] Dicke, Colin, Employee Engagement & Change Management, Cornell University, 2007.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-68
#`*

\ ¥

^ * **
#"#¥}}¤#}^ #<>"¦}¥\\#¥}¦
¤¨}}}
¢}Z! <¢¦}!

\+

+

*Z
*

|
*
|

`
< \ ¥
`

*`

*

!
¢\ ` *< "\
}

`
< }

\
\
!`*<**

**

!
* * ** |

* \ }`*

|
| \ !
**

` < \|\ *
! |\
`* \
*
}

**

<

\ } }* * }**

*Z
*
|
`

< |

`
*

`
< ¤!} `
<<
`
< | **
*

` Z

< <}"
}
" ¦ }
}

`*
*`
*
`
!!
* *!

< <
|

¥< *

||\

< <

}
}*

*}
| *

! !

!
<*
|

|\*

|*`

`

*

< *}

* #<
D. Tuden et al. / Developing a Strategy for Studying Critical Thinking in a Nurse Telehealth Setting 69

` \ |
*\ |

|

`} |

*
`

| ** `

}

`
`

!< \

*
* < *!

*
*

** *
`
`* \
\
}|

*
<
|
¯ ¥¯

° \ ±° ±}`| <}

\ <**

**

! `* * ** |

* \ }
`*

|
| \
!
**
` <

**
|*`
!\ |\<
/!/<!=">@<J!/QX
\

`
` *`

*`

!
¢\!*}|!
}*

<
*
`

! | `

*
`

*

¥

}
#

}
^

< }
* !
*

}
*
`

! ¢\<

}

*\|
`*
`
*
*
}

<
| *`
*
|

*** <
} *
*
*
* \| ` <

* `
*}}

!`
<

`*

*\
<}
* `

!

!
*`
***
!

< \
*
***

*

<
*
*
}

} }

\

<
* }

Z
*
|

** <

*

`
}

*
\

`

<
\

*

* }
*`
**

\
<
* *
**|
\ |
| \ !

* Z
*`

Z

! ¢\<
*

70 D. Tuden et al. / Developing a Strategy for Studying Critical Thinking in a Nurse Telehealth Setting
*
``

* <

`
! !

` * *

\
\ *

\
*

<

*
|\ * |

*
`

}

`
"`
¨ < *

*
<

*
! !
< *} }
}
*
} *`

< < *
}*
} ^
!
|

<

**
\| !
¦ }

!

|\
\
<
}
}

*} `
*
\

<
!

*
`*
\ < }

!
!

<
YZXZZ XZ!/<Q
}

**

}
\*
!} Z
*
}
<
!`}

` *

**

`
||

`*
`** `
``<

|

!
** `
} `

` ` ** } °|

** \
\
* }| !
** `*

\
±

¥
^ \ #`*


Z
\ *

!
\ < } |\ *

`* \
*

* <
|
| !

|* !

!
** \ <
]^(
<'+<_--
!-!!
^ *
``
!
\*
|\
*<

|

} `| |
\
|
\< ! * `| \
* *

| * |\ *
< }|\ *!

\

!
*
**

\ <

|\
` ° \ ±|
|\|
|

<

°{ ±°
{ ±|
! |\ * !

< \ * !

!
!
*
|\< |
}
! *`* | \ }|
|
|
| |
|
< |*|\ * }
|`|!
* §
| *
^| * < *
`|

} \ |
|
`
\ < `*
`` `|

\|\
\ |
`*!
! * |\ *< |

*

|\ *

*
`

<
\ <
\ |

`* *

`| } |\ *

}
|

} `|} `|
§

|
`*< **

} | \

!

!

!

!

!
`!

+"
<
j
<
"

<
< <

+< !!!

+"
<

!

<!

!!!

{<
?
<< < {<

+<

< !

" "

!

<
<{<
<

*

|\ *

`` \

* <
"[XX!/<

*{

*
\

!
\

}

`
`

72 D. Tuden et al. / Developing a Strategy for Studying Critical Thinking in a Nurse Telehealth Setting
\ } \ ! < }
`
`!

¢\<

` ||\

!**

<

* !
°

± | *`
<

`

}! ** }! *

| *`
*

**`

<
|\|

°
±}
*

*
} |
*
} ***
|``

! | `

}
}

} < \

!

*
<*}
*
<

**
\
\

*

!
*
|

¥

*
*

°* ** ¦| ±<
`
|
!
|\ * *

\

< <

`

*` **

}
*
}
`
} !\

` | <

\
* !
{!
< *

} |

*
` \ }
** \
\

<

¤
\

¤ \ \

|

* <
*
! \

\

!}

! *
}°
`
\ } *} | *
*
! \

!

±<

\
\

! `
< |`}
° \
²
!
\ |
* !

\| ±< |\ *
`*

*
*

`* \
\
}
!
*`

!! !
* `

<
!
| ! !

|\<
} *
* |

`*! | <`\

}

**
|

<

!

*|
*

<
} |
* |\
\
*
***
`
<< * < !
| *`
\ !
|\
*
*< } * *

`

|\ !\
³
|


* ` `

*

{<
|\
* *
<< !
} `|} ! `
`* \ }
|

*

! ` |\ |

!{ `
<
|

\|
\ |

<
\ ]<
` | ! ! **

\ { <

`

|
|
* ** *
< *
*
|!
|
!

< |\ *
| !

`* !

*
\|

|
*` !

< | !
`| !
!

!
!
<\ |

*

|*
!
**
\ <
*

|\|*

\ ³ *}

} ! *

} ! *
|
|
`
< *
|\ !
}| *`
<

!` ¤

! `
*
|

|**
\ <
**

`* \ }
* | ! °* ±

*
! ! *

!\
}

* |

` \| *
<
"
* ** ¦| } < * < `!
*
|||<!<
¥|^

^
¦| !
<
*
!< \ |
*
|\ `|<

¤
³
<¥

*
* `
\ <%|<}³
<

IOS Press, 2013
doi:10.3233/978-1-61499-203-5-74
A “Realist Review” Approach to e-Health:

The Case of Type 2 Diabetes in Youth
Marilynne A. HEBERT a,1
a
Department of Community Health Sciences, Faculty of Medicine, University of
Calgary
Abstract. As e-health technology becomes more ubiquitous in our health and

health care environments, a flexible, robust understanding of what works and
under what circumstances is needed. Traditional meta-analyses tell us how
frequently a technology has worked for previous populations, but not why. Realist
Reviews can contribute to understanding why interventions work and by extension
how results of past studies can be applied to emerging health challenges. The
utility of such a method is considered in e-health interventions to address the
serious growing challenge of Type 2 diabetes and metabolic syndrome in young
people.
Keywords. Diabetes Mellitus, Health Care Research, eHealth
Introduction
As we adopt e-health technologies into our health and health care environments, we
need a flexible, robust understanding of what works and where. Without this, the many
studies and evaluations of e-health technologies, in a variety of contexts, leads to
challenges in accumulating evidence for clear-decision-making. When a new health
challenge emerges, or e-health solutions become available, it may not be evident how
previous research applies, suggesting more research is needed.
There is no simple answer as e-health technologies are complex interventions
acting on complex social systems and therefore “are not 'magic bullets' that will always
hit their target” [1]. Rather their success is dependent on context and implementation.
This paper explores a “Realist Review” approach proposed by Pawson, et al. [2] to
understand the mechanisms for why e-health technologies work and in what contexts,
in order to better adapt future technologies in response to emerging health challenges.
1. Background
1.1. Introduction - Type 2 Diabetes in Youth as an Emerging Health Challenge
An increase in Type 2 diabetes and metabolic syndrome in young people is presenting

a new health challenge. In 2009, within Alberta alone, nearly 3,000 children and
1
Corresponding Author: Marilynne A. Hebert, Department of Community Health Sciences, University
of Calgary, 3D41 - 3rd Floor TRW, 3280 University Drive NW, Calgary, Alberta T2N 4Z6
M.A. Hebert / A “Realist Review” Approach to e-Health: The Case of Type 2 Diabetes in Youth 75
adolescents were living with diabetes and over 400 new cases were identified in the
under-20-year-old population [3].
As well, obesity rates tripled among 12-17 year olds between 1979 and 2004 [4-5].
Compared to children of normal weight, overweight and obese children experience
higher blood pressure, more abnormalities in serum lipid levels and higher levels of
insulin resistance. These conditions are all hallmarks of early metabolic disease,
diabetes and susceptibility to cardiovascular diseases [5].
1.2. Use of e-Health Technologies in Diabetes Management
A literature search in CINAHL and MEDLINE databases was conducted to examine

trends in e-health technologies used to support diabetes care. Search terms including
telehealth, e-health, telemedicine and mhealth were combined with diabetes and
resulted in 67 relevant articles.
Many factors contribute to optimal diabetes self-management, including timely
information on blood glucose levels, nutrition, physical activity, medications, medical
examinations and laboratory test results. Successful management is demonstrated
through changes in three areas: behavioral (adherence to diabetes control
recommendations), psychological (self efficacy) and physiological (HbA1C levels).
Technologies can contribute to this success through enabling patient–provider
interactions; collecting, summarizing, and responding to data; or providing education
and reminders. Examples of each of these illustrate the diversity in this literature.
1.2.1. Enable Patient-Provider Interaction
Coaching calls have been used primarily with families and children with Type 1
diabetes where adherence issues were a problem. Type 2 diabetes has been called a
“life style” disease and coaching calls have also been effective in helping seniors stay
active as well as regular telephone calls to assist adults with self-management and
education.
1.2.2. Collect, Summarize, and Respond to Data
A variety of interventions have been used to automatically collect and send information
to people with diabetes via Short Message Service (or SMS), telephone and websites.
Daily text messages have been used to help youth with Type 1 diabetes to enhance self-
efficacy, stabilize insulin injections and improve overall self-management. Messages
have also been customized using patient biochemical profiles and clinical status on file.
1.2.3. Provide Education and Reminders
Patients have frequently used phone calls and video-conferencing to connect with
diabetes educators. Another version of this interaction has been to customize education
as a result of submitting personal data. Examples include patients calling an automated
system weekly to receive tailored feedback and education on self care as well as
uploading step count data from pedometers to receive feedback and motivational
messages.
76 M.A. Hebert / A “Realist Review” Approach to e-Health: The Case of Type 2 Diabetes in Youth
1.2.4. Combining Interaction, Monitoring and Feedback
There is a growing interest in technologies that may work particularly well with a
younger population. For example, using a Smart phone as both a glucose monitor and
insulin pump [6], cell phone voice and text message interventions [7], or a Personal
Health Application may address many of the factors in optimal Type 2 diabetes
management [8]. A promising mobile phone-based system includes an integrated suite
of applications including blood glucose meter, a tailor-made step counter, software for
recording food habits and providing feedback on how users performed in relation to
their own personal goals [9].
2. Realist Review Approach
2.1. Brief Introduction to Realist Reviews
Traditional systematic review methods determine how many times an intervention has
produced the expected change. They typically do not provide information on why the
intervention worked or did not work when it was applied in different contexts,
deployed by different stakeholders, or used for different purposes. In contrast, Realist
Reviews ask how interventions bring about their effects and the limits of when and
where they work [1-2].
A Realist Review focuses on explaining the relationship between the mechanism
(M) by which an intervention works, the context (C) in which the intervention is
applied, and the outcomes (O) which are produced (Figure 1). A review cycle has four
stages [2, p.84-85]:
a. develop a theory around what would explain regularities in implementing the
intervention (including M,C,O);
b. hypothesize what might work for who and in what circumstances;
c. observe what works and for who using multi-methods to collect data and
analyses using M,C,O, which allows patterns to emerge;
d. specify program generalizations that identify what works for whom and in
what circumstances.
The goal is to look for evidence of “empirical uniformities,” i.e. regularities that occur
in the field, and in doing so establish limits to the intervention as well as inform theory.
Figure 1: Generative Causation [2]

M.A. Hebert / A “Realist Review” Approach to e-Health: The Case of Type 2 Diabetes in Youth 77
2.2. Realist Reviews Applied to e-Health Used for Type 2 Diabetes in Youth
The usefulness of a Realist Review is illustrated through examining examples of e-

health technologies currently being used to support diabetes management and
identifying regularities (Table 1). Two key aspects of managing Type 2 diabetes in
youth are important in informing a theory for e-health intervention:
a. mechanism – like adults with Type 2 diabetes, they need longer-term lifestyle
management strategies. In Type 2 diabetes accumulating health damage is not
visible or seen as life threatening so behavior change may not be readily
connected to outcome. It has apparently less urgency than Type 1 diabetes
suggesting the intervention must be more widely accessible.
b. context – similar to youth with Type 1 diabetes, they need strong family and
peer support. Adolescents need additional coping strategies as well as support
from their families, the education team, and friends their own age to manage
their diabetes [10]. Structural family factors (e.g. single parent families,
supportive or non-supportive environments) are important contextual factors.
Along with being a teenager, other influences include socio-economic status,
language and minority populations [11] that are important to consider in
establishing the boundaries of what works and where.
With respect to the intervention itself, youth may consider themselves savvy
technology users, but this does not always extend to “e-health literacy” (knowledge and
skills needed to interact with technology to address health information needs), which
may be taken for granted [12].
Regardless of the context, mechanism or e-health technology, the intervention is
expected to improve health outcomes. This is relatively easy to measure in diabetes
care as the “gold standard” is maintaining an HbA1C within normal limits. However,
intermediary outcomes contributing to this goal such as monitoring blood glucose as
well as managing dietary intake and exercise are more difficult to measure and sustain.
Relating outcomes to behavior is in an additional challenge for youth with Type 2
diabetes who are not experiencing symptoms.
Table 1 – Realist Review and “CMO” Examples
Intervention Context Mechanism Outcomes

Health coaching C1: Urban M1: peer support O1: coping skills
via telephone [13] adolescents at risk
for Type 2 diabetes
-Existing school-
based curriculum
12-month Diabetes C2: 3 individual M1: peer support O1: behavior
TeleCare program - and 10 group change
self-management sessions;
education underserved
intervention [14] community
Community-based C3: Chartered M2: access to O1: behavior
cell-phone assisted Health Family education, provider change (high drop-
diabetes self- Clinic – low support out rate: lack of
management [15] income; monetary use)
incentives
78 M.A. Hebert / A “Realist Review” Approach to e-Health: The Case of Type 2 Diabetes in Youth
3. Conclusions
Many studies in diabetes management suggest e-health technologies can be effective.

However, little or no information is provided on the limits of these technologies – when
or where they work or do not work; and who they work or do not work for. These are
important questions to answer in order to determine the “empirical uniformities” that
can guide further theory development.
Youth who are overweight, pre-diabetic or diagnosed as Type 2 diabetics are at
much higher risk of cardiovascular damages much earlier in their lives than adults
likewise diagnosed. A strategy to manage this upcoming health challenge must draw on
multiple theories and technologies to address lifestyle choices in different contexts: at
school, out with friends and at home. In order to get the mechanisms right, youth
should be involved in the design, testing and implementation of the interventions. A
Realist Review approach can contribute to understanding the strengths and limits of e-
health technologies, to develop theory and inform decisions for the future.
References
[1] Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review
designed for complex policy interventions. J.Health Serv.Res.Policy 2005 Jul;10(Suppl 1):21-34
[2] Pawson R & Tilley N. Realistic Evaluation, 1997; Sage.
[3] Alberta Diabetes Atlas 2011. Alberta Diabetes Surveillance System accessed July 5, 2012 at
http://www.albertadiabetes.ca/
[4] Shaw J. Epidemiology of childhood type 2 diabetes and obesity. Pediatric Diabetes, 2007: 8; 7.
[5] Alberta Health Report. Accessed July 5, 2012 at: http://www.health.alberta.ca/health-info/diabetes.html
[6] Brookes L. Why can’t a diabetes device be as cool as an iPod? Posted 04/07/2010. Accessed on-line
July 16, 2012 at: http://www.diabetesweb.org.uk/2010_Ipod%20designs%20for%20DM.pdf
[7] Krishna S, Austin Boren S, and Balas EA. Healthcare via cell phones: A systematic review.
Telemedicine and e-Health, April 2009: 15(3); 231-240.
[8] Fonda SJ, Kedziora RJ, Vigersky RA and Bursell SE Combining iGoogle and personal health records to
create a prototype personal health application for diabetes self-management. Telemedicine and e-Health,
2010: 16(4); 1-10.
[9] Årsand E, Tatara N, Østengen G, and Hartvigsen G. Mobile phone-based self-management tools for
Type 2 diabetes: The Few Touch Application. J Diabetes Sci Technol 2010;4(2):328-336
[10] Huus K and Enskär K. Adolescents' experience of living with diabetes. Paediatric Nursing, 2007;
19(3): 29-31.
[11] Arora S, Peters AL, Agy C, and Menchine M. A mobile health intervention for inner city patients with
poorly controlled diabetes: Proof-of-concept of the TExT-MED Program. Diabetes Technology &
Therapeutics, 2012; 14(6): 492-496.
[12] Norman, C. eHealth literacy 2.0: Problems and opportunities with an evolving concept. J Med Internet
Research. 2011 Oct-Dec; 13(4): e125. Published online 2011 December 23. doi: 10.2196/jmir.2035.
Accessed on-line July 16, 2012 at http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3278111/
[13] Jefferson V, Jaser SS, Lindemann E, Galasso P, Beale A, Holl MG, and Grey M. Coping skills training
in a telephone health coaching program for youth at risk for Type 2 diabetes. Journal of Pediatric
Health Care, 2011; 25(3): 153-161.
[14] Davis RM, Hitch AD, Salaam MM, Herman WH, Zimmer-Galler IE and Mayer-Davis EJ. TeleHealth
improves diabetes self-management in an underserved community. Diabetes Care, 2010; 33(8); 1712–
1717.
[15] Katz R, Mesfin T and Barr K. Lessons from a community-based mHealth diabetes self-management
program: “It's not just about the cell phone.” Journal of Health Communication: International
Perspectives, 2012; 17(sup1): 67-72.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-79
Designing the Community Multi-user

Health Kiosk
Julie M. McMILLAN a,1, Karen L. COURTNEY b, Judith T. MATTHEWS c, Asim
SMAILAGIC a and Daniel SIEWIOREKa
a
Carnegie Mellon University, Pittsburgh, PA, USA
b
c
University of Pittsburgh, Pittsburgh, PA, USA
Abstract. This paper discusses the design and development of a multi-user health
kiosk intended for independent use by underserved populations. The modular
integration of physiological sensors and psycho-social assessments provides an
extensible, customizable platform for research. We present the development of the
kiosk’s feature set and user interaction mechanisms through iterative user testing,
in addition to some technical challenges and solutions resulting from our design
choices.
Keywords. kiosk, telehealth, user-centered design
Introduction
The Community Multi-user Health Kiosk (Figure 1) was designed to give people who
are currently underserved by the healthcare system a way to monitor their health and
communicate with their healthcare provider. The Quality of Life Technology
Engineering Research Center focuses on the elderly, those with disabilities, and people
of lower socio-economic status and supports the design and development of the health
kiosk.
Rather than a fixed function device, the kiosk is a platform that can support a
variety of psycho-social assessments and peripheral monitoring tools. It consists of a
mobile desk that has a computer running the Windows-based kiosk software program, a
printer, a movable touchscreen, and a drawer that contains any peripheral medical
devices needed for the kiosk's assessments. In the first use case, the kiosk was intended
for the common areas of senior high rise buildings. Potential sites have expanded to
include primary care offices, pharmacies, public libraries, and other community
gathering places.
In contrast to many commercially available telehealth monitors, the kiosk is
designed to be multi-user, individualized, customizable, and extensible. Each user's
self-monitoring health information is conveyed to their healthcare provider via multiple
possible mechanisms. Both the assessments and the results display can be customized
by clinicians to meet the specific needs of their patients, and an internal messaging
feature offers secure communication between the two. In addition, the modular design
makes it easy to switch hardware manufacturers/models and add or remove
1
Corresponding Author. Julie McMillan, email: [email protected]
80 J.M. McMillan et al. / Designing the Community Multi-User Health Kiosk
assessments. Because this design is substantially different from commercially available

devices, this paper will describe the design and development of the multi-user kiosk.
Figure 1. Health Kiosk
1. Features
1.1. Standard assessments
"Assessment" is a term used broadly for any value measurable with the kiosk and can
include both physical as well as psycho-social measurements. In its current baseline
configuration, the kiosk has peripheral device-based, self-report and manual entry
assessments. The peripheral off-the-shelf devices include a pulse oximeter, blood
pressure monitor, seated scale, and hand dynamometer. The devices communicate with
the computer via USB or Bluetooth.
Self-report assessments are ones where a user answers questions. An example use
case would be a clinician scheduled annual depression screening. The default surveys
on the kiosk are the Geriatric Depression Scale [1] the Epworth Sleepiness Scale [2],
the Pittsburgh Sleep Quality Index [3], the Medical Outcomes Study 36 Item Short
Form Survey [4], the Morisky Medication-Taking Adherence Scale [5], and the Cost-
Related Nonadherence to Medications Survey [6]. Our community-based usability
testing has validated the reliability in translating these instruments to the kiosk.
To accommodate these various response formats, the kiosk supports a number of
response options: multiple choice, sliding scales, date and/or time entry, and free text
responses. The instrument engine also supports branches in the survey - that is, making
questions dependent on previous answers. The kiosk can automatically score
instruments with standard scoring metrics like summing. Automatic scoring of other
instruments with non-generic scoring algorithms is possible with minor software
modifications. Adding a new self-report assessment consists of formatting the content
according to a straight-forward XML specification. No programming knowledge is
necessary. In all cases, every user that sees, changes, or deletes an assessment result is
logged by an audit trail to ensure accountability and security.
J.M. McMillan et al. / Designing the Community Multi-User Health Kiosk 81
1.2. Customization
To increase user engagement, customization has been an ongoing theme in the kiosk
development. In addition to minor personalization features, such as welcoming each
user by name, clinicians can select assessments for individual patients and set patient-
level guideline parameters for assessment results. For example, each assessment can be
made available or unavailable globally (all kiosks), at selected kiosks, or for selected
individual users. The kiosk prompts the user to complete the assessment at the
clinician's specified interval. The kiosk can notify health care providers for alert values
via multiple contact strategies including text message, email, or fax.
A user's view of their own results is one of the clearest examples of software
tailored to the patient. Results are visible to the user in three ways –the current
assessment results, a summary of all a session's results, or as a historical view
encompassing the last 3, 6, 9, or 12 months. Both numerical and categorical results are
color-coded for easy comprehension; desirable values are green and less desirable are
yellow, orange, or red. In the default settings, the color-coding stems from population
guidelines, but a clinician can customize this to emphasize a "personal best" for a
patient (Figures 2 and 3).
Figure 2. Historical view of categorical result. Figure 3. Immediate view of numerical result.
1.3. Extensibility
Designed as an expandable software platform, adding a new device-based assessment

to the kiosk is often simple enough for an average programmer to do in less than a day.
One recent extension is a trio of assessments based on the Microsoft Kinect sensor to
conduct several common mobility tests.
2. Design and Development
From the beginning of the project, we defined our user base as broadly as possible to
include older adults, clinicians, and health care providers. Our emphasis on user-
centered design governed a number of initial decisions about the structure and content
of the health kiosk. For instance, concerns about participant falls led to use of a seated
scale rather than a traditional step-up model. This choice prompted the design of a
unique push-through drawer mechanism so wheelchair users could access the kiosk
82 J.M. McMillan et al. / Designing the Community Multi-User Health Kiosk
devices without moving the seated scale. Clinician concerns about blood-borne
pathogens resulted in a design choice to have participants type in blood glucose values
rather than perform testing on a kiosk-based glucometer. The target population for the
kiosk also led to some user interaction challenges. Many of our older users were not
familiar with computers or touchscreens (Figure 4). In early prototypes that labeled
buttons with the verb "Press", we observed people pressing but not releasing. Most
buttons are now labeled "Tap." Many common software metaphors had to be
eliminated or made more explicit.
Figure 4. Revised welcome screen. Figure 5. Revised blood pressure instruction.
Instructions for device-based assessments, in particular, underwent numerous

revisions (Figure 5). CogTool [7], an application that produces a predictive user
performance model based on screen design, was used to analyze our user interface
design and supports our observation that a single step per screen is better for
inexperienced computer/kiosk users and is not a major barrier for advanced users.
Some assessments were switched to audio instructions where appropriate. When
possible, we supplement the instructions with direct feedback. After watching some
participants put the blood pressure cuff on upside down, we added a mercury tilt switch
with an open source Arduino microcontroller. This enabled the kiosk to guide
participants through placing the cuff correctly.
3. Technical Challenges
The benefits of using off-the-shelf devices include easy replacement of hardware; low
cost; and the potential usability of consumer-oriented products. The disadvantages are a
lack of standardization and, potentially, poor reliability. To mitigate these issues, our
general strategy was intensive upfront development of a modular and fault-tolerant
architecture, component testing, and user-driven hardware and software modifications
to minimize error. Table 1 lists some of the complications we encountered in the
development process, their effects, and solutions.
For example, hardware defects were an unanticipated problem, and especially
difficult when the defect was subtle. Our first pulse oximeter occasionally displayed a
dramatically different result on the device screen than appeared on the kiosk, with
disastrous results for user confidence. After ruling out other potential sources of error,
J.M. McMillan et al. / Designing the Community Multi-User Health Kiosk 83
we concluded that the hardware was at fault and changed the brand of oximeter, which
was quick and painless due to the modular nature of the kiosk software.
Issue Effect Solution

No device drivers No access to device's data Write custom driver, using
provided APIs or packet sniffing
Use scripting language to run 3rd

party software in background
Hardware design flaws Confusing or unusable devices lead Hardware customization or
to user frustration and errors substitution
Hardware defects Incorrect results, distrust from Component testing and
users verification, substitution
No hardware No generic way to collect data or Code to a generic interface,
standardization know in advance what data will customize implementations as
look like needed; minimize hard-coding
Poor architecture leads to Lack of fault-tolerance in 3rd party Anticipate noise, errors, and the
cascading failures hardware/software compounds unexpected and handle gracefully
errors
Table 1. Issues encountered during design and their resolution
4. Discussion
Lab and community-based user testing with over 40 individuals has led to a system that,
while continuing to evolve, is very usable for people in our target demographic. We
have also demonstrated that the initial investment in a modular, robust architecture
pays ample dividends when requirements change or the inevitable hardware problems
occur. Building upon this work, we are conducting pilot work for kiosk-based health
interventions and working with health organizations on kiosk data integration.
References
[1] Yesavage JA, Brink TL, Rose TL, Lum O, Huang V, Adey, M, Leirer VO. Development and validation
of a geriatric depression screening scale: A preliminary report. J Psychiatr Res. 1982; 17: 37-49.
[2] Johns MW. A new method for measuring daytime sleepiness: The Epworth sleepiness scale. Sleep.
1991; 14: 540-545.
[3] Buysse DJ, Reynolds CF, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh Sleep Quality Index
(PSQI): A new instrument for psychiatric research and practice. Psychiatry Res. 1989; 28: 193-213.
[4] Hays RD. The Medical Outcomes Study (MOS) Measures of Patient Adherence. RAND Corporation.
1994.
[5] Morisky DE, Ang A, Krousel-Wood M, Ward HJ. Predictive validity of a medication adherence
measure in an outpatient setting. J Clin Hypertens (Greenwich). 2008; 10: 348-354.
[6] Soumerai SB, Pierre-Jacques M, Zhang F, Ross-Degnan D, Adams AD, Gurwitz J, Adler G, Safran
DG. Cost-related medication nonadherence among elderly and disabled medicare beneficiaries: A
national survey 1 year before the medicare drug benefit. Arch Intern Med. 2006; 166: 1829-1835.
[7] John, B., Prevas, K., Salvucci, D., & Koedinger, K. (2004) Predictive Human Performance Modeling
Made Easy. Proceedings of the SIGCHI conference on Human factors in computing systems (2004),
455-462.
Public Health Informatics
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-87
Public Reporting of Hospital Infection

Rates: Ranking the States on Credibility
and User Friendliness
Ava AMINI a, David W. BIRNBAUMb, Bernard BLACK1, c and David A.
HYMANa, d
a
Northwestern University, School of Law ,b University of British Columbia School of
Population & Public Health, University of Victoria School of Health Information
Science, and Healthcare Associated Infections Program Manager at the Washington
State Health Department, c Northwestern University, School of Law and Kellogg
d
School of Management, University of Illinois at Champaign-Urbana, School of Law
and School of Medicine
Abstract. Health-care associated infections (“HAIs”) kill about 100,000 people

annually; most are preventable, but many hospitals have not aggressively
addressed the problem. In response, twenty-five states and the U.S. Department of
Health and Human Services require public reporting of hospital infection rates for
at least some types of infections, and other states and private entities are
implementing such reporting. The websites and related reports vary widely in ease
of access, ease of use, usefulness of information, timeliness of updates, and
credibility. We report on work in progress, in which we assess the quality and
suitability of different state websites and reports for different target audiences
(ordinary consumers; physicians, and infection control professionals) and the
extent to which they meet best practices for online communication, including
Stanford’s “Fogg” Guidelines for Web Credibility and user-friendliness metrics
developed by other researchers. We find wide variation in quality, and substantial
correlation between measures of website credibility and user-friendliness. We
identify ways to improve usability, usefulness, and tailoring for information to
different target audiences. Our analysis suggests that the “one website (and report
format) fits all users” model may not work well in delivering complex, technical
information to users with widely varying needs and sophistication.
Keywords. health-care quality; health-care public reporting; health-care associated

infections; performance measurement; website usability
Introduction
Health-care associated infections (HAIs) are a huge public health problem. Of roughly
39 million annual hospital admissions in the U.S., about 1.7 million (4.5%) result in
HAIs and 100,000 HAIs result in death [1]. The direct healthcare costs of HAIs are
estimated at $45 billion, and total annual costs likely exceed $100 billion [2, 3]. In
response, twenty-five states and the federal government currently require hospitals to
publicly report infection rates for at least some types of infections; other states have
1
88 A. Amini et al. / Public Reporting of Hospital Infection Rates
adopted laws that will require reporting in the near future; and private entities have
created their own public reports. The effectiveness of these reporting requirements
in informing consumer choice and reducing infections is unknown [4]. Moreover, and
the focus of this study, “we do not just want to know if public reporting works
(efficacy); we want to know who it works for and in what situations (effectiveness)”
[5].
HAI public reporting presents an ideal environment in which to examine the
effectiveness of public delivery of health care information. Each state maintains a
website; most also provide downloadable reports, but each uses its own unique
reporting format. There is wide variation in websites, and reports, across any metric
one might consider: how easy the websites and reports are to find, which infections
are reported; how infections or infection rates are measured; what information
about them is reported; time periods covered; level of technical difficulty,
readability levels, and so on.
We report here on work in progress in which we evaluate how effective each
state’s website and public reports are at communicating information. We focus on
website credibility, user-friendliness, and usefulness of information for three target
audiences with different needs and ability to understand technical information,
including typical healthcare consumers; physicians (and other more sophisticated
users); and infection control professionals.
1. Methodology
To assess website credibility we use ten criteria based on psychological research on

persuasive technologies [6]. The specific factors are as follows:
• Is it easy to verify the accuracy of specific statements, by clicking through to
supporting information/data?
• Demonstration that a "real" organization is responsible?
• Expertise in the area is highlighted?
• Evidence of trustworthiness?
• Easy to contact those responsible?
• Is site professionally laid out?
• Easy to use and useful?
• Updated recently?
• Limited promotional content?
• Any [reasonably apparent] mistakes?
Five of these criteria are yes/no questions; we give each 3 points for a “yes.” Five
call for more qualitative assessments; we create a detailed 1-5 scale for each, specifying
the attributes needed for each score. We sum the scores, to develop an overall “Fogg
score” (5~35). The weights are arbitrary but our results are not sensitive to the
weighting.
To assess “user-friendliness”, we adapt measures suggested in research by other
scholars of public reporting of health care performance [7-9]. We identified five factors
(and are in the process of developing a more refined list of factors):
• Provides good introduction to HAIs
• Helps consumers to integrate information from multiple indicators
• Uses both numbers and graphs/symbols to convey numeric or statistical
A. Amini et al. / Public Reporting of Hospital Infection Rates 89
information
• Straightforward to find results for a particular hospital
• Website and report length and complexity is appropriate for consumer
audience
As with the Fogg guidelines, we developed a detailed description of the attributes
needed to receive each score. Each factor was scored on a 1-5 scale, for an overall 5~25
scale. We applied these factors separately to the website and to the formal reports on
state’s websites. In this paper, we focus on the formal reports.
Finally, for “usefulness,” we developed and then assessed a number of factors,
including:
• Was the state website easy to find?
• Within the website, was pertinent information easy to find?
• Are the explanations clear?
• Was it easy to find information on a particular hospital?
• Types of infections covered
• Type of rate reported (“raw” [crude], risk-adjusted, risk-stratified, or a
combination)
• Were number of infections reported?
• Is statistical significance reported?
• Is historical data available?
• If yes, is the format consistent over time?
• Can users compare hospitals online, how many, and using what selection
criteria?
• Are similar hospitals compared to each other?
• Overall, was the website understandable and useful?
We assess usefulness for three target audiences. One important audience is average
consumers. We assumed that these users have roughly a high school reading level and
limited understanding of statistics and medical terminology. A second audience is
physicians, who can counsel patients, make decisions on which hospital to admit their
patients to, and push hospitals for change. This audience is likely to have better than a
college graduate reading skills, moderate understanding of statistics and healthcare
terminology, but limited knowledge about particular HAIs. A third audience is
infection control professionals, who can use the reports to benchmark their hospital
against peers. These persons are likely to have a college graduate reading level; be
familiar with measurement and coding issues for HAIs, but have limited understanding
of statistical analyses. At many hospitals, infection control is assigned to registered
nurses. Only at major hospitals, often with academic affiliations, is the head of
infection control likely to be an epidemiologist with advanced statistical training.
Thus, we judged that a typical infection control professional would know more about
HAIs but less about statistics than a typical physician or other sophisticated user.
Manifestly, we must exercise judgment in deciding what criteria to measure, how to
measure them, and what target audiences to assume; we do so based on the expertise
developed in our previous research on these subjects. [10-11]
Thus, each state is scored on five metrics: website credibility, user friendliness of
website, user friendliness of public reports; usefulness of website for each target
audience, and usefulness of public reports for each target audience. Initial scoring was
by a law student at Northwestern. We are in the process of assessing inter-rater
reliability. A spreadsheet including all criteria and results will be publicly posted on
Professor Black’s website at Northwestern University.
2. Results
We present here partial results from our analysis, focusing on Fogg website credibility
scores and user friendliness scores for the public reports from the perspective of an
ordinary consumer. Scores are based on the reports and websites as they appeared in
August, 2012. We also plan to assess changes in websites and reports over time. Higher
scores reflect better performance on the specified measures. Fogg scores have a
theoretical range from 5 to 40; in practice, they ranged from 24 (Vermont) to 39
(California). Public report usability scores have a theoretical range from 5 to 25; in
practice, they ranged from 9 (Rhode Island and Washington) to 25 (Massachusetts).
The close-to-maximum scores on each measure for some states reflect not perfection,
but our effort to define criteria that made a top score reasonably achievable.
Figure 1 indicates the scores on both measures for each state that currently has
public reporting of HAIs. Figure 1 also includes the Canadian province of Ontario. We
plan to expand our analysis to include the Hospital Compare website run by the
Department of Health and Human Services, and the websites operated by the Leapfrog
Group and Consumers Union.
It also provides a correlation line between the two measures. The Pearson
correlation coefficient is strongly positive at 0.58 (t = 3.46). There is thus a strong
tendency for states to do either well or poorly on both measures.
One might expect larger states, which usually have greater resources, to do better,
but this was not universally true. For example, New Hampshire did well on both
measures, while Florida did poorly on both measures.
Finally, rankings may change dramatically if states redesign their website. For
example, Washington’s current Fogg Score is 27 but an earlier version of Washington’s
website had a Fogg score of 36. The mean (median) Fogg score was 31.2 (30.5) for our
entire sample of 25 states and Ontario, so the redesign dropped Washington’s score
from substantially above-average to below-average.
3. Discussion
Public reporting of HAIs is widespread, but little attention has been paid to how to
design websites and public reports to reach the multiple target audiences. We report on
work in progress in which we systematically quantify the effectiveness of existing
websites and public reports, across a variety of general metrics (credibility, user
friendliness, and usefulness) for three different target audiences. For ordinary
consumers, we find substantial correlation between website credibility and user-
friendliness of each state’s public reports.
One issue that is missing from current discussions of credibility and user
friendliness, but substantially complicates any analysis of these issues, is the variety of
potential target audiences. HAI data can be quite technical, and assessing its value
requires an understanding of statistics and confidence intervals. To be readable and
understandable for a typical consumer, some website and reports have been deliberately
“dumbed down” to a level that is likely to prove inappropriate and frustrating for more
A. Amini et al. / Public Reporting of Hospital Infection Rates 91
sophisticated users.
Another issue that is missing from current discussions is “usefulness.” Assessing
the usefulness of HAI information requires in-depth knowledge of the goals of
reporting and the information being reported. For example, a time-series of data is
usually more reliable than a snapshot of a short time period. Reports that tell users
only whether a hospital is above, average, average, or below average (with 95% or
more graded as average) are easy to understand, but not very useful. Some states
provide more granular information, but most do not. A comparison of the CLABSI
(central line-associated bloodstream infection) rates at a small community hospital to
the rate at major tertiary hospitals may mislead, rather than inform. And so on.
Finally, many websites and reports seem to focus on a single target audience, even
though more tailored information would be useful. Serving multiple audiences through
a single website and report format is challenging. In the future, we will analyze the
extent to which states have recognized and addressed this problem.
Figure 1. Fogg Website Credibility Score vs. User- Friendliness Score for Public Reports
Fogg website credibility scores and HAI report user friendliness (for ordinary consumers) scores for 25 states
plus Ontario with public HAI reporting (as of August 2012). See text for scoring method.
References
[1] R.M. Klevens, J.R. Edwards, C.L. Richards, T.C. Horan, R.P. Gaynes, D.A. Pollock, and D.M. Cardo,
Estimating Health Care-Associated Infections and Deaths in U.S. Hospitals, 2002, Public Health
Reports 122 (2007),160-166
[2] R.D.II. Scott, The Direct Medical Costs of Healthcare-Associated Infections in U.S. Hospitals and the
Benefits of Prevention, [updated 2009], Available from:
www.cdc.gov/ncidod/dhqp/pdf/Scott_CostPaper.pdf.
[3] M.R. Eber, R. Laxminarayan, E.N. Perencevich, A. Malani, Clinical and Economic Outcomes
Attributable to Health Care-Associated Sepsis and Pneumonia, Archives of Internal Medicine 170
(2010), 347-353.
[4] B. Black, Public Reporting of HAI Rates: What We (Mostly Don’t) Know, Clinical Governance 17(2)
(2012), 124-133.
[5] Agency for Healthcare Research and Quality (AHRQ), Public Reporting as a Quality
Improvement Strategy, Evidence Report/Technology Assessment No. 208. [updated
2012], Available from:
http://www.effectivehealthcare.ahrq.gov/ehc/products/343/1198/Evidencereport208_CQG-
PublicReporting_ExecutiveSummary_20120724.pdf.
[6] B.J. Fogg, C. Soohoo, D. Danielson, L. Marable, J. Stanford and E.R. Tauber, How do People Evaluate a
Website’s Credibility? Results from a Large Study [updated 2002], Available from:
http://www.consumerwebwatch.org/pdfs/stanfordPTL.pdf.
[7] K.M. Mazor, K.S. Dodd, A Qualitative Study of Consumers’ Views on Public Reporting of Health
Care- Associated Infections, American Journal of Medical Quality 24 (2009), 412-418.
[8] K.M. Mazor, K.S. Dodd, L. Kunches, Communicating Hospital Infection Data to the Public: A Study of
Consumer Responses and Preferences, American Journal of Medical Quality 24 (2009), 108-115.
[9] N. Bardach, J.H. Hibbard, R.A. Dudley, Users of Public Reports of Hospital Quality: Who, What, Why,
and How? AHRQ Publication No 12-0016-EF, [updated 2011], available from:
http://www.ahrq.gov/qual/value/pubreportusers/.
[10] D. Birnbaum, M.J. Cummings, K. Guyton, J. Schlotter, A. Kushniruk, Designing Public Web
Information Systems with Quality in Mind, Clinical Governance 15 (2010), 272-278.
[11] D. Birnbaum, R. Zarate, A. Marfin, SIR, You’ve Led Me Astray! Infection Control and Hospital
Epidemiology 32 (2011), 276-282.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-93
Methodological Approaches to Comparing

Information about Bicycle Accidents
Internationally: A Case Study Involving
Canada and Germany
Christian JUHRA a,1 , Britta WIESKÖTTER a, Paule BELLWOOD b, Ariane von
BELOW c, Murray FYFE d, Sonia SALKELD e, Elizabeth BORYCKI b, and Andre
KUSHNIRUK b
a
University Hospital Münster, Dept. of Traumasurgery, Germany
b
University of Victoria, School of Health Information Science, Canada
c
BASt – Federal Highway Research Institute, Germany
d
Vancouver Island Health Authority, Canada
e
BCAS and Corporate Services , Emergency and Health Services Commission, Canada
Abstract. The use of bicycles as a mean of healthy and eco-friendly transportation

is currently actively promoted in many industrialized countries. However, the
number of severe bicycle accidents rose significantly in Germany and Canada in
2011. In order to identify risk factors for bicycle accidents and possible means of
prevention, a study was initiated that analyses bicycle accidents from selected
regions in both countries. Due to different healthcare systems and regulations, the
data must be selected in different ways in each country before it can be analyzed.
Data is collected by means of questionnaires in Germany and using hybrid
electronic-paper records in Canada. Using this method, all relevant data can be
collected in both countries.
Keywords. Bicycle Accidents, Data Collection, Trauma Registry
Introduction
Cycling is a healthy and eco-friendly way of transportation. The use of bicycles is

currently actively promoted in many industrialized countries. However, cyclists belong
to the group of the so-called “vulnerable road users”. According to the German
FederalStatistical Office (Destatis), 65,573 bicycle accidents occurred in 2010, and 381
cyclists died in these accidents in Germany. After years of declining accident numbers,
the number of bicycle accidents rose significantly in 2011, when 76,351 cyclists were
injured and 399 died [1]. Fatality rates of cyclists in British Columbia also rose from 9
in 2009 to 11 in 2011 (mountain bike accidents were excluded) [2]. In the year of 2009-
2010 (fiscal year), 4,324 people were hospitalized in Canada due to bicycle accidents.
Compared with the other provinces, the largest number of bicycle crashes occurred in
1
Corresponding Author: Dr. med. Christian Juhra, MBA, University Hospital Münster, Department of
Trauma Surgery, Waldeyerstr. 1, D-48149 Münster, Germany; Email: [email protected]
94 C. Juhra et al. / Methodological Approaches to Comparing Information
British Columbia (along with the Yukon Territories) (22.1 per 100,000 population, age-
adjusted) [3]. A high rate of underreporting was revealed by a study in Münster,
Germany. The accident rates were up to 3 times higher than the officially reported rates
[4]. Head injuries were the main cause of hospitalization in Germany and Canada, and
one of the leading causes of death among cyclists. To get a better overview of the real
situation of bicycle accidents and related injury patterns, a prospective study funded by
the German Federal Highway Research Institute was initiated in May 2012. In this
paper, we will describe the methodology by which accident data from the two countries
with different healthcare systems and transport regulations can be collected.
1. Vancouver Island and Germany – The Bicycle Accident Perspective
Vancouver Island covers an area of 31,285 km2 (approximately 10% of the area of the
Federal Republic of Germany), and has a population of 748,937. Almost half of those
(367,572) reside in Victoria, the capital of British Columbia. Victoria has been named
repeatedly by the Canada’s Cycling Magazine “Pedal” as the “Bicycle Capital of
Canada” as a result of its bicycle-friendly infrastructure and the many biking (hiking)
trails available. Due to its geographical location, outdoor activities are very popular on
the island. This is also reflected by a high number of bicycle accidents.
In Germany, the accident data are collected within the Trauma Network North-
West (TNNW). The TNNW is a consortium of 37 trauma care hospitals in the regions
of North Rhine-Westphalia/Southern Lower Saxony. The TNNW represents an area of
approximately 14,000km2 and a population of about 4 million comprised from the
states of North Rhine-Westphalia, Lower Saxony, as well as part of the Netherlands,
and is one of the largest trauma networks in Germany. In contrast to the more urban-
centered trauma networks, such as that of Berlin, the TNNW also includes a large
number of rural regions.
Assuming that the area of the TNNW is representative of the Federal Republic of
Germany, approximately 3,000 bicycle accidents happen in this area per year. To
include the issue of under-reporting (i.e. approximately only 2/3 of actual cases are
reported), approximately 9,000 bicycle accidents involving personal injury occur in the
area of the TNNW per year. Based on the results of the Münster bicycle study, 11.1%
of all injured persons were hospitalized, which means that the expected number of
hospitalized patients was around 1,000.
2. Collecting the data – A Tale of Two Worlds
2.1. Goals of the study
The Münster bicycle accident study led by Münster University Hospital has shown that,
due to the high underreporting, the currently available statistics cannot provide reliable
information about the actual situation of bicycle accidents. However, the Münster
bicycle accident study was limited to the city of Münster, and is, therefore, not a
representative sample for Germany.
In order to allow for a detailed analysis of bicycle accidents, the data of all patients
requiring hospital medical care in the region of the TNNW and on Vancouver Island
C. Juhra et al. / Methodological Approaches to Comparing Information 95
due to bicycle accidents will be collected for a period of one year. The study aims to
provide answers to the following questions:
• Where did the accident occur (city/country)?

• When did the accident occur?
• Are there any common accident attributes?
• Do these accident attributes vary regionally?
• What injury patterns occur in a bike accident?
• What are the economic consequences of bicycle accidents?
• Does the type of bike (especially pedelecs/eBikes) have an impact?
• How often were protective aids used (i.e., helmet, reflective vest, etc.)?
• Can intoxication have an effect on the injury rate?
The study must thus combine data concerning medical issues and data concerning
issues related to the type and circumstances of the accident (Figure 1).
2.2. Data Collection in Germany
Currently, no standardized electronic data set exists for emergency departments. There
is a great variety ways in which physician and nurses document the reason for
admission, the health status and complaints of the patient, etc. in German hospitals.
This ranges from free text on paper to highly standardized electronic forms, which,
unfortunately, is often an exception to the rule. The police collect data on accidents and
hardly any detailed data on type of accident can be found in patient records. Due to
privacy and other legal issues, these databases cannot be linked. The nationwide
German Trauma Registry includes only the severely (life-threatening) injured patients
and does not contain detailed data on the type of the accident. Currently, no
standardized emergency dataset or electronic health record exists.
Thus data collection in Germany is heavily dependent on questionnaires.
Analogous to the established methodology of the Münster bicycle accident study,
accident data are collected by patient questionnaires and hospital chart reviews.
Questionnaires are distributed to the patients in the clinics of the TNNW. In addition,
the patient data can be collected on the spot, and can later be entered by a student
assistant (SA) into an anonymous database. 23 hospitals in all regions of the TNNW
have declared their commitment to take part in the study.
A FileMaker Advance Pro 12 ® database is used to collect the data. Each SA runs
a stand-alone application which can export the data to the main database. These stand-
alone applications are available in English and German.
The data will be entered into an anonymous database. Each record will be
identified by accident time, patient age, accident site, and hospital name. In cases
where patients will be relocated to another hospital within the TNNW, the records will
have to be updated accordingly. This method of collecting data in Germany has been
approved by the Ethics Committees of the University of Münster and Munich.
96 C. Juhra et al. / Methodological Approaches to Comparing Information
2.3. Data Collection in Canada
Due to the low density of population on Vancouver Island, injured cyclists can visit not
only hospitals, but also clinics throughout the island. As a result, using questionnaires
for data collection on Vancouver Island would not be feasible.
As compared to Germany, a standardized electronic ambulance record and dataset
exists in British Columbia, in which the circumstances of the accident (e.g. protective
gear, weather etc.) are also documented. In addition, every trauma patient admitted to a
hospital is entered into the BC Trauma Registry, which also includes data on accident
type and pre-clinical emergency care. In addition to this, the Vancouver Island Health
Authority (VIHA) has an Island wide hybrid, electronic patient record system with both
paper and electronic components. Patients who receive care following a bicycle
accident at VIHA have nearly all of their data residing in databases that are part of the
electronic patient record.
The School of Health Information Science of the University of Victoria will access
the data provided by the electronic data sources. The data will need to be accessed and
linked from database repositories located in different parts of the province. This
requires application to local ethics bodies and meeting legislative and privacy
requirements in the province of BC.
Figure 1. Graphical overview of data collection methods in Germany and Canada.

C. Juhra et al. / Methodological Approaches to Comparing Information 97
3. Discussion
In order to improve the safety of cycling and promote the use of bicycles, a deep and
thorough understanding of bicycle accidents is crucial. Unfortunately, most bicycle
accidents are not officially reported, so that using the official statistics does not show
the complete picture.
Despite the large number of accidents and fatalities, research on bicycle accidents
remains scarce. Most research focuses on severe or fatal accidents [5], while minor
injuries are often not investigated.
Different healthcare systems, transportation systems and regulations, and different
ways of documenting data are a challenge for collecting accident and medical data in
different countries. However, by identifying the needed variables and the possible
sources of information, ways can be found to collect the same data in different
countries.
An electronic health record can be a valuable source for health service research.
Using existing electronically available data avoids the need for paper-based
questionnaires with their drawbacks (single-source approach). eHealth can thus also be
useful to generate data for injury prevention.
The data collection of the described bicycle study will be finished at April 30 th,
2013. Preliminary results will be presented at the end of 2012. The final results will be
published in a BASt-report in German language and presented in Canada in the fall of
2013. We will be integrating our study of bicycle accidents in both countries with our
prior work in simulation modeling and other studies. [6].
References
[1] German Federal Statistical Office (Statistisches Bundesamt, Deutschland) www.destatis.de

[2] Office of the Chief Coroner, BC
[3] http://www.cihi.ca/CIHI-ext-portal/internet/en/Document/types+of+care/RELEASE_21JULY11
[4] Juhra et al.; Bicycle accidents - Do we only see the tip of the iceberg? A prospective multi-centre study
in a large German city combining medical and police data.; Injury. 2011 Nov 18. [Epub ahead of print]
[5] Manson J et al, Major trauma and urban cyclists: physiological status and injury profile. Emerg Med J.
2012 Feb 22. [Epub ahead of print]
[6] Juhra C, Borycki EM, Kushniruk AW, Anderson J, Anderson M. Making eco-friendly transportation
safer: Developing computer based simulations to assess the impacts of bicycle accident prevention
interventions on healthcare utilization. Stud Health Technol Inform. 2011, 164: 183-7.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-98
The Development of a Standardized

Software Platform to Support Provincial
Population-Based Cancer Outcomes Units
for Multiple Tumour Sites: OaSIS -
Outcomes and Surveillance Integration
System

Jonn WUa,1, Cheryl HOa, Janessa LASKINa, David GAVINa, Paul MAKa, Keith
DUNCANa, John FRENCHa, Colleen McGAHANa, Sherry REIDa, Stephen CHIAa,
Heidi CHEUNGb
a
BC Cancer Agency, Vancouver, British Columbia
b
BC Cancer Research Centre, Vancouver, British Columbia
Abstract. Understanding the impact of treatment policies on patient outcomes is

essential in improving all aspects of patient care. The BC Cancer Agency is a
provincial program that provides cancer care on a population basis for 4.5 million
residents. The Lung and Head & Neck Tumour Groups planned to create a generic
yet comprehensive software infrastructure that could be used by all Tumour
Groups: the Outcomes and Surveillance Integration System (OaSIS). The primary
goal was the development of an integrated database that will amalgamate existing
provincial data warehouses of varying datasets and provide the infrastructure to
support additional routes of data entry, including clinicians from multiple-
disciplines, quality of life and survivorship data from patients, and three
dimensional dosimetric information archived from the radiotherapy planning and
delivery systems. The primary goal is to be able to capture any data point related
to patient characteristics, disease factors, treatment details and survivorship, from
the point of diagnosis onwards. Through existing and novel data-mining
techniques, OaSIS will support unique population based research activities by
promoting collaborative interactions between the research centre, clinical activities
at the cancer treatment centres and other institutions. This will also facilitate
initiatives to improve patient outcomes, decision support in achieving operational
efficiencies and an environment that supports knowledge generation.
Keywords. Clinical research informatics, population based outcomes research,

cancer therapies, clinical decision support, data integration, data mining.
1
Corresponding Author: Jonn Wu, BCCA, 600 West 10th Avenue, Vancouver, BC V5Z 4E6, Canada; E-
mail: [email protected].
J. Wu et al. / The Development of a Standardized Software Platform 99
Introduction
Understanding the impact of treatment policies on patient outcomes is essential in

improving all aspects of patient care. Within the British Columbia Cancer Agency
(BCCA), the development of several Outcomes Units (ie. Breast, Ovarian Units) have
influenced treatment protocols and patient outcomes1, 2. Although many Tumour
Groups are eager to develop their own databases, the associated expertise and expenses
are prohibitive. Rather than develop databases for each tumour group, the authors
planned to create a generic yet comprehensive software infrastructure that could be
used by all groups: the Outcomes and Surveillance Integrated System (OaSIS).
The BCCA provides province-wide population based cancer control strategies for
the residents of British Columbia and the Yukon. Diagnostic services, consultations,
chemotherapy, radiation therapy and supportive care are delivered through five
regional cancer centres and over forty community clinics. All treatments in BC are
delivered by standardized protocols established by Provincial Tumour Groups; clinical
cases are reveiwed at multidisciplinary Tumour Boards that are video-linked to all
cancer centres and participating community hospitals. The BCCA has been very
uniquely successful in the accumulation and electronic storage of disparate population
based health information ie. the provincial cancer registry. The Pharmacy Program
maintains a provincial data mart that tracks every chemotherapy course dispensed in
BC; the radiotherapy planning and delivery system is on a central server for all five
cancer centres - every radiotherapy course in BC is archived in one database. Until
now, these data warehouses have not been linked to each other or to clinical endpoints.
1. Limitations of Current Clinical Research Databases
Although some BCCA Outcomes Units have been very successful, most use software
programs with limited functionality (ie. Microsoft Excel, Access). Frequently, data is
collected on paper or are extracted from the chart retrospectively, then manually
entered. This process is labor intensive, expensive and error-prone; ideally, data should
be entered once, and used multiple times. These programs often cannot support
multiple users across many sites, multiple modules/disciplines, customized role-based
security access, clinician/patient data entry, or direct data linkages.
2. OaSIS – Required Functions
In order to support large-scale, multi-disciplinary outcomes research on a provincial

basis, the OaSIS authors leveraged modern information technologies to develop a
customized integration system. The primary goal is to be able to capture data related to
patient characteristics, disease/treatment details and side effects. Data will be directly
imported from existing data warehouses wherever possible. These data points will be
augmented by additional fields which will be both patient and clinician generated. The
ability to support simultaneous users in multiple centres with role-based security
permissions requires a robust, customizable and scalable solution; OaSIS is built on the
Microsoft Internet Information Services platform using SQL 2008 Server (Enterprise
Edition), Visual Studio and .Net Client Server Framework.
100 J. Wu et al. / The Development of a Standardized Software Platform
Because data entry is a challenging and expensive barrier for any outcomes
project, there will be three processes to accommodate data entry:
2.1. Data Entry By Health Record Administrators
This process will support the use of paper forms (ie. Staging and Follow-up).
Document distribution and data entry will be facilitated by the Data Quality & Registry
department. This is the traditional workflow used by most Outcomes Units currently,
and requires ongoing operational costs.
2.2. Direct Clinician/Patient Entry
Clinicians (Oral Oncology, Pathology and Oncology Nutrition) and patients (side
effects) will be able to enter data directly into OaSIS. This will omit the use of paper
forms, thereby reducing costs and errors associated with distribution and entry. It is
well recognized that patient generated quality of life data is more accurate and less
prone to bias 3, 4.
2.3. Direct Data Linkages
The OaSIS system will support direct data linkages to existing data warehouses; this
provides the most efficient and accurate method for data collection. The system is
currently connected to the Cancer Agency Information System (CAIS, clinical
information), provincial cancer registry, and the provincial chemotherapy warehouse.
Additional linkages are in progress ie. direct access to dosimetric radiotherapy data.
3. OaSIS Implementation Road Map
The OaSIS road map includes four phases.
3.1. Phase One: Clinical Integration
In September 2008, the Lung and Head & Neck Tumour Groups invited
multidisciplinary leaders to develop their own modules ie. discipline-specific data
elements. Data dictionaries are mandatory for each module, and each data element is
limited to discrete variables. These multidisciplinary modules include:
• Patient Demographics (including Tumour Stage, Risk Factors)
• Follow-Up (Clinical Outcomes)
• Pathology (Histological Subtyping, Molecular Markers)
• Chemotherapy
• Radiotherapy (non-dosimetric datasets)
• Oral Oncology
• Oncology Nutrition
• Patient and Family Counselling (PSCAN Psychological Screening)
• Quality of Life Tools (Patient and Physician Generated Assessments)
Each module is a combination of imported data and additional data collected by
clinicians. The Patient Demographics module imports information from the registry (ie.
tumour site, histology) and CAIS (ie. BCCA ID). Additional factors are recorded on
paper (ie. tumour stage, smoking history, performance status) and entered later.
The Pathology and Nutrition modules primarily consist of clinician generated data.
Pathologists enter molecular markers, and histological sub-typing at the time of central
review, whereas nutritionists record weight, eating related side effects and feeding tube
complications in “real time” at the time of the patient encounter by the clinician.
The Chemotherapy module is unique since all of the data is imported from the
pharmacy system; this includes every chemotherapeutic drug that was ordered and
dispensed by the BCCA, including dose, and how, when and where it was
administered. Over 99% of all chemotherapeutic agents dispensed in BC is captured in
this system. Additional functionality includes filters (ie. dates, drug names, number of
cycles and days per cycle) used to automatically generate chemotherapy “courses”.
After an initial testing phase, prospective data entry was initiated in May, 2011. As
of October, 2012, OaSIS is supporting 12 unique research projects and includes:
• Patient Demographics: 7,374 records
• Follow-Up: 1,934 events (loco-regional and distant recurrence)
• Chemotherapy: 2,538 courses
• Oncology Nutrition: 189 assessments
• PSCAN Psychological Screening: 10,245 assessments
The Pathology and Oral Oncology Modules will launch in 2013.
3.2. Phase Two: Radiotherapy Dosimetric Repository
The Radiotherapy (RT) Module introduced in Phase One extracts basic and generic RT
data from the Cancer Agency Information System (CAIS). These fields include RT
start and finish dates, total radiation dose, treatment technique, and treatment region.
Although these continue to serve as useful end points, they do not provide any spatial
information specific to a particular treatment course. To fully understand the effects of
radiotherapy dose on irradiated organs and target structures, investigators require
access to the full three-dimensional dose matrix ie. how much dose was delivered to a
particular part of a structure; additionally, the detailed coordinates of relevant structures
also need to be archived with their respective radiographic image sets (CT, MRI, and/or
PET/CT). For example, is radiation dose to a certain part of the tongue result in worse
dysgeusia than another part of the tongue? The second Phase will create a three-
dimensional dosimetric repository, or a “dose bank”, linked to clinical outcomes.
Although investigators have proposed the concept of a dosimetric library, they are
typically not linked to comprehensive population based clinical outcomes. Our ability
to correlate and analyze clinical endpoints, on a large scale, with three-dimensional
dosimetric data sets will provide investigators with unprecedented research potential.
This platform will allow investigators to answer important questions related to
radiation response and normal tissue durability. Initial work at the BCCA on a much
smaller scale (<100 patients) regarding salivary function and partial volume dosimetry
have already made a clinical impact5.
Software development was initiated in September, 2011, in conjunction with the
Faculty of Medical Physics, UBC. This included initial architectural design and review
of functional requirements ie. prior to any dosimetric analysis of a structure, the
module must be able to identify it correctly. Identification will be based on parsing the
name, and double checking its geographic location within the body. For example, if a
102 J. Wu et al. / The Development of a Standardized Software Platform
structure is labelled “L paro”, the module will correctly identify it as the “left parotid
gland” by parsing the name, and ensuring the structure exists in the correct geographic
location in the body. Automatic dose calculations within sub-segments of a structure is
now being developed. This work will result in improved understanding of the effect of
radiation dose to certain parts of normal structures, which will lead to fewer side effects
in irradiated patients.
3.3. Phase Three: Research Module, Patient Portal
Due to the uniquely disparate nature of the data elements in OaSIS, the Research
Module will include data mining tools using a combination of existing and custom
developed querying applications. This work will be done in conjunction with the
Faculty of Computer Science, UBC.
A multidisciplinary team will develop patient generated surveys which will capture
information during a patient’s cancer journey via a web-based Patient Portal. As cancer
therapies improve, quality of life becomes an increasingly significant endpoint.
3.4. Phase Four: Research Integration and Additional Tumour Sites
The research integration phase will see the addition of other tumour groups who wish
to participate ie. the Testes Module will be added in early 2013. As well, linkages will
be created to other data repositories, including the BCCA Tumour Tissue Repository.
4. Summary
The OaSIS platform will provide a powerful and customizable, integrated

multidisciplinary outcomes database for the BC Cancer Agency Tumour Groups.
Unlike existing outcomes databases which capture uncomplicated data fields, the
OaSIS system will leverage unique and unprecedented connectivity with existing
provincial data warehouses, including the cancer registry, provincial chemotherapy and
radiotherapy data repositories. The correlation of clinical outcomes and other patient
and treatment related parameters in a central database will allow for unique analyses
that will result in improvements in patient care, clinical decision support and treatment
delivery.
References
=? Köbel M, Kalloger SE, Huntsman DG, Santos JL, Swenerton KD, Seidman JD, Gilks CB; Cheryl Brown
Ovarian Cancer Outcomes Unit of the British Columbia Cancer Agency, Vancouver BC. Differences in
tumor type in low-stage versus high-stage ovarian carcinomas. Int J Gynecol Pathol. 2010
May;29(3):203-11.
[2] Panades M, Olivotto IA, Speers CH, Shenkier T, Olivotto TA, Weir L, Allan SJ, Truong PT. Evolving
treatment strategies for inflammatory breast cancer: a population-based survival analysis. J Clin Oncol.
2005 Mar 20;23(9):1941-50..
=? Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, Lenderking WR, Cella D, Basch E;
ISPOR ePRO Task Force. Recommendations on evidence needed to support measurement equivalence
between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good
Research Practices Task Force report. Value Health. 2009 Jun;12(4):419-29. Epub 2008 Nov 11.
[4] Hróbjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, Ravaud P, Brorson S.
Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both
blinded and non-blinded outcome assessors. BMJ. 2012 Feb 27;344:e1119.
[5] Moiseenko V, Wu J, Hovan A, Saleh Z, Apte A, Deasy JO, Harrow S, Rabuka C, Muggli A, Thompson
A. Treatment planning constraints to avoid xerostomia in head-and-neck radiotherapy: an independent
test of QUANTEC criteria using a prospectively collected dataset. Int J Radiat Oncol Biol Phys. 2012
Mar 1;82(3):1108-14. Epub 2011 Jun 2.
Clinical Decision Support Systems
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-107
Neuroanatomical Basis for Recognition

Primed Decision Making
Darren HUDSON a,1
a
Division of Critical Care Medicine, University of Alberta
Abstract: Effective decision making under time constraints is often

overlooked in medical decision making. . The recognition primed decision
making (RPDM) model was developed by Gary Klein based on previous
recognized situations to develop a satisfactory solution to the current problem.
Bayes Theorem is the most popular decision making model in medicine but is
limited by the need for adequate time to consider all probabilities. Unlike other
decision making models, there is a potential neurobiological basis for RPDM.
This model has significant implication for health informatics and medical
education.
Keywords: decision making, neuroanatomy, medical education, health

informatics
Introduction
Human decision making continues to be debated in psychology. In medicine, decision-

making is not merely academic; it affects lives. Many models purport to explain how
medical decision-making occurs. One of the largest revolutions in medicine in the late
1990s was the dissemination of evidence-based medicine, that formalized decisions
based on the available literature. Other forms of decision-making were derided as less
scientific.
The long-standing debate over the variety of medical decision-making models
neglect one aspect; time limitations on arriving at the correct decision. Additionally,
many decisions need to be made with incomplete information and concurrent with
treatment. Zakay and Wooler [1] demonstrated that effective decision-making ability
fell precipitously as time pressure increased. Even when decision-making skills were
taught prior to the experiment, the ability to make correct decisions declined with
available time.
These findings support the notion that teaching decision-making strategies are
helpful when time permits option consideration. However, in fast-paced situations,
formal decision-making procedures and instruction are essentially useless. This means
that alternative models for decision-making under time-pressured situations need to be
developed.
1
Corresponding Author: Darren Hudson, email: [email protected].
108 D. Hudson / Neuroanatomical Basis for Recognition Primed Decision Making
Recognition Primed Decision Making
The recognition primed decision making (RPDM) model was developed from cognitive
psychology in the mid-1980s by Klein et al. [2]. As a model, it has described decision-
making for time sensitive situations, employed by fire-fighters, tank commanders,
naval officers, and airline pilots [3]. However, there is a paucity of medical literature
using this model. The only published work in medicine involved an emergency
situation in an ophthalmology clinic [4] where the author described a successful
resuscitation based on early recognition and anticipation of patient deterioration.
RPDM was originally developed using fire command officers [5]. In post-incident
interviews, the researchers found that fire commanders followed a series of steps to
make decisions based both on available information and prior experience. The
investigators developed a model wherein the person presented with a situation made an
initial assessment and decided whether it fit a familiar pattern [5]. If the situation was
recognized, then the level of familiarity was graded by comparing it to previous cases
and cues present at that time. The individual then considered whether the intended
goal was feasible in the current circumstance, considered issues in the contemporary
case that needed to be accounted for, and finally considered possible actions.
If the current situation was not recognized, RPDM was unlikely to be effective,
further information was sought, and the situation reassessed in order to generate a
recognizable pattern [5]. Failing this, the decision-maker had the option to choose
other strategies to seek an answer. However, in the case where the situation was
recognized and a reasonable match found from past experience, the decision-maker
could mentally simulate the solution and try to anticipate problems that may occur. If
this simulation suggested a satisfactory outcome, it was enacted. If the simulation
suggested difficulties, then the situation required reassessment and plan modifications.
All these steps occurred automatically with little conscious effort. [5].
Several features make RPDM distinctive from other decision-making strategies
[5]. As a decision-maker develops more experience, only feasible options are
considered, and long lists of random options - many of which will not work - are
obviated. RPDM, then, renders faster, more efficient decisions. As well, decision-
making is not intended to find the best possible solution, but to find the first
satisfactory option for the current situation. Better options may present themselves
subsequently, but they do not delay action. This “satisficing” means that the goal in
the decision-making is to find the most reasonable, but not necessarily the best option,
to accomplish the goal. As opposed to other decision-making models, no formal
instruction in RPDM is required. The process is intuitive and built on experience.
Finally, the use of mental simulation also allows decision-makers to put the task into
context and allows for preliminary testing to ensure feasibility of the option.
Neuroanatomical Basis for Recognition Primed Decision Making
A recent hypothesis has suggested that the main function of the cerebral cortex is to act
as a memory storage unit and recognition/prediction engine [6]. Consider the example
of a conversation in a noisy room. The only way to follow the conversation is to be
able to predict and fill in the unheard gaps. The memory of words and common
sentences with an ability to predict the next phrase makes this possible.
D. Hudson / Neuroanatomical Basis for Recognition Primed Decision Making 109
At only 4 mm thick, the cerebral cortex is a new evolutionary invention found only
in vertebrates. Mapping studies over the entire cortex have demonstrated different
functional regions throughout the brain. However, the microscopic structure of each
region is the same throughout the brain [7]. Thus, different regions are functionally
distinct because of their connections, not because of their structure.
The cortex can be divided into six layers - layer VI is the deepest, I is the most
superficial [6]. Inputs enter the cerebral cortex via VI and travel upwards for
processing. The different layers are structured such that cells in one layer will receive
multiple inputs from the cells below and send inputs to higher levels and laterally
through interneurons. Inputs arriving at the cortex will tend to stay within the same
region as they progress up the layers. Thus the cortex can be considered being divided
into columns. Sensory input sequences (vision, touch, sounds etc.) ascend up the
cortex through each cortical layer and interact with descending signals from higher
memory levels [6]. These higher memory level signals are representations of complex,
often multi-sensory sources used to predict incoming sensory input.
The cortex as a prediction engine has been modelled successfully by Rao and
Ballard [8] using an artificial retina attached to electronic simulation of a neuronal
column. In the model, a prediction of expected input based on previous input flows
downward from higher level neurons and interfaces with an input signal. The
difference between prediction and input signal generates an error signal which travels
back up the simulated cortical column through successive layers of interaction with the
prediction algorithm to modify the expected input. In this simple model, the simulator
successfully anticipated and predicted incoming inputs. Interlayer feedback interacted
with the incoming signal and was an efficient way to encode the visual image.
If the cortex can be considered a prediction/error recognition engine, then RPDM
has a neuroanatomical foundation. When faced with a situation, sensory stimulation
enters the cortex via level VI. These inputs ascend individual columns towards level I.
When the signal is recognized, associative interneurons connect to related columns.
These columns are now anticipating an incoming signal. When it arrives, the column
performs sequential error checking of the signal. If the incoming signal matches then it
is recognized and the process repeats with other associative interneurons. Consider the
example of hearing a series of odd numbers from one to nine. By the time the number
five is heard, the sequence is clear and interneurons have sent signals to the columns
involved in processing the numbers seven and nine. When seven is heard, there is a no
error signal in that column and it is considered to be recognized. If the next number is
ten, an error signal is generated which requires processing in higher cortical layers and
the sequence is now not recognized.
Applications in Health Informatics
RPDM is relevant to health informatics, most obviously in education. Currently,

undergraduate and postgraduate medical and nursing education teach decision-making
using evidence-based medicine or Bayes theorem. There is no focus on rapid decision
making strategies. However, the challenge with RPDM is that it cannot be formally
taught. This method requires experience. This means that educators must recognize
the value of clinical experience and ensure sufficient exposure to permit expertise
development. Educators also need to ensure learners understand the potential for
110 D. Hudson / Neuroanatomical Basis for Recognition Primed Decision Making
misuse of this decision-making model and associated potential cognitive errors as has
been pointed out by Kahneman [9].
RPDM also allows us to understand the phenomenon of the experienced nurse
calling for help before the crisis occurs. As a patient deteriorates, a series of stimuli are
generated. These inputs are processed in the lower cortical layers and recognised. The
associative interneurons then to connect to alarm centres that tell the nurse that there is
something wrong and to call for help. Often the nurse will be unable to explain the
reason for the recognition because it occurred in the lower layers and did not generate
error signals that required higher level processing. This re-enforces the importance of
experience and exposure in education. Reading a text about a patient deterioration will
not generate the same recognition pathways as experiencing it.
Conclusion
In this paper, recognition primed decision making as a model for medical decision
making has been described. Originally studied in non-medical situations, it represents
an important addition to understanding how experienced clinicians decide. Unlike other
decision making models, there appears to be a neuroanatomical mechanism for RPDM.
This is based on the hypothesis that the cerebral cortex functions as a memory storage
and recognition/anticipation engine. The implications for education and health
informatics are significant.
References
[1] Zakay D, Wooler S. Time pressure, training and decision effectiveness. Ergonomics.
1984;27(Compendex):273-84.
[2] Klein GA, Calderwood R, Clinton-Cirocco A. Rapid Decision Making on the Fire Ground. United
States: 1988.
[3] Schraagen JM. Naturalistic decision making and macrocognition. Aldershot, England ; Burlington, Vt.:
Ashgate; 2008.
[4] Bond S, Cooper S. Modelling emergency decisions: recognition-primed decision making. The literature
in relation to an ophthalmic critical incident. J Clin Nurs. 2006;15(8):1023-32. Epub 2006/08/02.
[5] Klein GA, Calderwood R. DECISION-MODELS - SOME LESSONS FROM THE FIELD. IEEE
TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS. 1991;21(5):1018-26.
[6] Hawkins J, Blakeslee S. On intelligence. 1st ed. New York: Times Books; 2004. 261 p. p.
[7] Creutzfeldt OD. Generality of the functional structure of the neocortex. Naturwissenschaften.
1977;64(10):507-17.
[8] Rao RPN, Ballard DH. Predictive coding in the visual cortex: a functional interpretation of some extra-
classical receptive-field effects. Nature Neuroscience. 1999;2(1):79.
[9] Kahneman D. Thinking, fast and slow. New York: Farrar, Straus and Giroux; 2011..
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-111
Use of Knowledge Discovery Techniques

to Understand Nurse Practitioner Practice
Patterns and Their Integration into a
Healthcare System
Esther SANGSTER-GORMLEYa,1, Mu-Hsing KUOb, Elizabeth M. BORYCKIa,b and
Rita SCHREIBERa
a
School of Nursing, University of Victoria, Victoria, British Columbia, Canada
b
School of Health Information Science, University of Victoria, Victoria, British
Columbia, Canada
Abstract. The objective of this study is to assess the feasibility of applying

knowledge discovery techniques to identifying nurse practitioner practice patterns
and enacted scope of practice. For the research, we plan to use data extracted
from a Ministry of Health database. The data items are focused around: nurse
practitioner demographics, health authorities, and encounter types. This analysis
produces patterns that indicate relationships between the demographics, scope of
practice and practice settings of nurse practitioners working in British Columbia.
Keywords. Knowledge Discovery, Data Mining, Nurse Practitioner, Practice

Patterns, Nurse Practitioner Integration
Introduction
Since 2005, when legislation enabling the nurse practitioner (NP) role was enacted in
British Columbia (BC), integrating the role into clinical practice settings has been a
priority. To date, $62.6 million has been invested to support and resource the
integration of NPs, however, no evaluation of their impact on the BC health system has
been conducted. At this time, it is imperative that we: a) begin to determine the changes
that occur for patients and the implications to the health care system when NPs become
part of the care process, b) assess the impact of adding an NP to the functioning of
collaborative health care teams, and c) continue to monitor the practice settings and
scope of practice of NPs working in BC. A mixed method, multi-year study is currently
being undertaken to evaluate the integration of NPs into the BC healthcare system, with
the intent to establish an evaluation framework. The overall study is designed to
address the following broad questions: (1) What are the practice settings and the scope
of practice of NPs working in BC? (2) What changes result for patients and, what are
the implications for the health care system when NPs become part of the care process?
(3) What is the impact of adding an NP on the functioning of collaborative health care
teams?
1
112 E. Sangster-Gormley et al. / Use of Knowledge Discovery Techniques
In this paper we report on our planned use of knowledge discovery techniques to

analyze NP data from a Ministry of Health (MoH) database. This work will lead to a
better understanding of NP practice patterns and the integration of NPs into the health
care system in BC and will be used to answer the questions outlined above.
1. Review of Relevant Literature
Although the NP role is new to BC, it has existed in other Canadian provinces and the
United States for more than 40 years. The role has been studied more than any other
role in health care [1]. The evidence accumulated over these years has consistently
demonstrated that NPs provide safe, effective care [2]. Researchers consistently find
that patients are satisfied with the care received from NPs, trust NPs [3], feel that NPs
take their problems seriously and discuss their concerns, have expert communication
skills and are approachable [4].
Sidani, et al. [5] found that, when other variables were controlled, patients cared for
by an NP in acute care were satisfied with their care, and also had higher physical and
social functioning than those cared for by other providers. Others compared NPs to
physicians and found that NPs had longer consultation times and patients received
more information and counseling [6]. In addition, NPs placed more emphasis on health
promotion and self care management of chronic and acute conditions than did their
physician colleagues [7]. Researchers have consistently found that NPs provide patient-
centred care, spend time with patients, and emphasize health promotion and chronic
disease self-care and, when compared to physicians, the physical outcomes of care are
similar [6, 8]. Nonetheless, there is a need to expand our evaluative efforts to
understand the value-added impact of NPs in the BC context. It is time to evaluate the
consequences or improvements in the overall health and wellbeing of patients that
might be attributed to such NP practice activities.
At the time the NP role was implemented in BC, the MoH developed encounter
codes to be used by NPs to track their activities. Encounter codes are unique numbers,
similar to ICD-10 codes, assigned to specific types of patient care services performed
by NPs. NPs are expected to submit an encounter code to the MoH at the end of each
patient encounter. Encounter code data are stored in Population Data BC databases.
2. The Knowledge Discovery Process (KDD)
Data mining is "an integral part of knowledge discovery in database (KDD), which is
the overall process of converting raw data into useful information" [9]. In other words,
it is the process of automatically discovering useful information in large data
repositories (e.g. data warehouses). The KDD process consists of four major
transformation phases. They include the: (1) problem definition phase, (2) data pre-
processing phase, (3) pattern recognition phase and (4) pattern validation phase (see
Figure 1) [10] .
In the problem definition phase we are attempting to answer the broad question:
What are the practice settings and scope of practice of NPs working in BC?
E. Sangster-Gormley et al. / Use of Knowledge Discovery Techniques 113
Figure 1: A generic process of knowledge discovery

In the data pre-processing phase raw data is transformed into an appropriate
format for subsequent analysis. Furthermore, real-world data tend to be incomplete,
noisy and inconsistent. Data cleaning routines attempt to fill in missing values, smooth
out and correct inconsistencies in the data. In the pattern recognition phase one
chooses the proper data mining algorithm(s) to discover the patterns. In this study, we
propose the Apriori algorithm [11] to execute association analysis on the NP patient
diagnosis, health authorities and encounter types. The algorithm employs an iterative
approach known as level-wise search, where k-itemsets are used to explore (k+1)-
itemsets. Then, association rules are generated, which is an implication of the form
XµY, where X and Y are disjoint subsets of all the possible data items. The strength
of the association rule can be measured by its support and confidence as follows:
Support determines how often the data items in a rule are present together in a
transaction in a given data set and is simply the count of transactions that contain X and
Y. Support is used to determine if a rule is of interest since high support indicates that
the rule occurs often in the data. Confidence is used to determine the reliability of the
inference made by the rule and is an estimate of the conditional probability of Y given
X. It says “If X is present in a transaction, how likely is it that Y is also present”. The
generated rules suggest a strong co-occurrence relationship between the given data item
subsets. Finally, in the pattern validation phase we undertake the process of assessing
how well the mining models perform against real data. It is important that the mining
team validates the discovered patterns before deploying them into a production
environment. There are several approaches for validating the patterns [12]. For
example,
Use statistical validity to determine whether there are problems in the data or in
the model - A number of statistical methods may be used to evaluate the data
mining model quality or pattern accuracy, such as Cross Validation and Receiver
Operating Characteristic (ROC) curves.
Separate the data into training and testing sets to test the accuracy of patterns - It
is common for the data mining algorithms to find patterns in the training set which
are not present in the general data set. To deal with the so called "over-fitting"
issue, the validation uses a test data set which the data mining algorithm was not
114 E. Sangster-Gormley et al. / Use of Knowledge Discovery Techniques
trained on. The learnt patterns are applied to the data set and the resulting
output is compared to the desired output.
Ask domain experts to review the results of the data mining to determine
whether the discovered patterns have meaning in the targeted scenario – In this
study, the research team composed of domain experts in the fields of NP
education/research (ESG, RS) and nursing/health informatics (EB, MSK) will be
involved in the validation process to interpret the accuracy of the patterns.
3. A Hypothetical Case Study
We are planning to use data extracted from the MoH database from BC. In this paper,
since our human research ethics approval is pending, we use a test dataset with 25
records to test the feasibility of the proposed method. The data are encoded as numbers
for data analysis (see Figure 2). In the dataset, there are 4 health authorities (named
VIHA, VCH, FHA and NHA). Each health authority has 3 NPs whose age is between
20 and 59. The encounter types used by NPs are coded using unique number; for
example, 36360 is a prenatal visit, 36365 is the insertion of an intrauterine device,
36413 is repair of a minor laceration and 36255 is newborn care.
Applying the KDD techniques to this simple dataset has yielded some interesting
patterns/information as follows:
Pattern 1: Male NPs use more encounter types than female NPs (4-types/8-
encounters (50%) vs. 3-types/17- encounters (18%)).
Pattern 2: Older NPs (age group 4) use more encounter types than other age group
of NPs (age group 1~4: 1/2(50%), 4/7(57%), 3/12(25%) and 3/4(75%)).
Pattern 3: NPs in the Northern Health Authority (NHA) use more encounter types
than those of other health authorities (health authority 1~4: 3/7(43%), 3/9(33%),
3/5(60%) and 3/4(75%)).
Pattern 4: Female NPs insert more intrauterine devices (36365) than male NPs.
The confidence is 86%.
Figure 2; A hypothetical data set with 25 records

E. Sangster-Gormley et al. / Use of Knowledge Discovery Techniques 115
Pattern 4 suggests additional analysis is needed. This pattern shows that female
NP’s insert more intrauterine devices than male NPs. Further analysis will be
undertaken to determine if there are other activities that female NP’s undertake more
often.
4. Conclusion
Integrating the NP role into clinical practice settings is new in BC. An evaluation of its
impact on the BC health system is important. In this study we assessed the feasibility of
applying knowledge discovery techniques to analyze NP data and to generate some
interesting patterns/information. This work will lead to a better understanding of NP
practice patterns and will be used to answer the study questions. This analysis data
involved the use of test data (as we determined the feasibility of employing data mining
techniques). Future work will involve extracting data from BC MoH databases.
Patterns generated from our analyses in this study do not reveal the real nature of the
associations between NP demographics, geographic locations and encounter types.
Future work will use MOH NP data, and domain experts will be involved in the
validation process to interpret the mined patterns. This should increase the value of our
work in that we will be able to determine how generalizable the approach is.
References
[1] Bauer JC. Nurse practitioners as an underutilized resource for health reform: Evidence-based
demonstrations of cost-effectiveness. Journal of the American Academy of Nurse Practitioners,
2010;22(4):228-231.
[2] Horrocks S, Anderson E, Salisbury C. Systematic review of whether nurse practitioners working in
primary care can provide equivalent care to doctors. Brit Med J, 2002;324(6): 819 -823.
[3] Benkert R, Hollie B, Nordstrom C, Wickson B, Bins-Emerick L. Trust, mistrust, racial identity and
patient satisfaction in urban African American primary care patients of nurse practitioners. Journal of
Nursing Scholarship,2009; 41(2): 211-219.
[4 Thrasher C, Purc-Stephenson R. Patient satisfaction with nurse practitioner care in emergency
departments in Canada. Journal of the American Academy of Nurse Practitioners, 2008;10, 231-237.
[5] Sidani S, Doran D, Porter H, Lefort S, O’Brien-Pallas L, Zahn C., et al., Outcomes of nurse
practitioners in acute care: an exploration. Internet Journal of Advanced Nursing Practice, 2007;8(1), 3.
[6] Kinnersley P, Anderson E, Parry K, Clement J, Archard L, Turton P. et al. Randomizedcontrolled
trial of nurse practitioner versus general practitioner care for patients requesting “same
day”consultations in primary care. Brit Med J, 2000; 320, 1043-1048.
[7] Charlton C, Dearing K, Berry J, Johnson M. Nurse practitioners’ communication styles and their
impact on patient outcomes: An integrative literature review. Journal of the American Academy
of Nurse Practitioners,2008; 20, 382-388.
[8] Lenz E, Mundinger M, Kane R, Hopkins S, Lin S. Primary care outcome in patients treated by nurse
practitioners and physicians: Two-year follow-up. Medical Care Research and Review,2004;61(3), 332-
351.
[9] Tan PN, Steinbach M, Kumar V. Introduction to Data Mining. Addison Wesley;2005. ISBN0-321-
32136-7
[10] Kuo MH. Databases, Data Warehousing and Data mining for Healthcare" in Healthcare
Information Technology Essentials for the IT Professional, 2012, McGraw-Hill's. (in press)
[11] Kotsiantis S, Kanellopoulos D. Association Rules Mining: A Recent Overview, GESTS International
Transactions on Computer Science and Engineering. 2006;32 (1), 71-82.
[12] Richesson RL. Clinical Research Informatics. Springer;2012. ISBN: 1848824483
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-116
Decision Support for Evidence-Based

Pharmacotherapy Detects Adherence
Problems but Does Not Impact Medication
Use
Janese M. WILLISa, Rex EDWARDSa, Kevin J. ANSTROMa, Fred S. JOHNSONa,
Guilherme DEL FIOLb, Kensaku KAWAMOTOb, Nancy M. Allen LAPOINTEa, Eric
L. EISENSTEIN1a, David F. LOBACHa
a
Duke University Medical Center, Durham, NC, USA, bUniversity of Utah, Salt Lake
City, UT, USA
Abstract. Although evidence-based pharmacotherapies are a principal component

of patient care, 30-50% of patients do not take their medications as prescribed. We
conducted a randomized trial of two clinical decision support (CDS) interventions
in 2219 patients: patient adherence reports to providers (n=744), patient adherence
reports to providers + email notices to care managers (n=736), and controls (739).
At 18-month follow-up, there were no treatment-related differences in patient
medication adherence (overall, by medication class, and by medical condition).
There also were no treatment-related differences in patient clinical and economic
outcomes. Thus, while this study’s CDS information interventions were
successfully delivered to providers and care managers, and were effective in
identifying medication adherence deficits and in increasing care manager
responses to medication adherences issues, these interventions were not able to
alter patient medication behavior.
Keywords. Clinical decision support, Population health management, Managed

care, Health information exchange, Health network, Medication adherence
Introduction
The Institute of Medicine’s (IOM) Quality Chasm report highlighted 20 priority areas
in which improvements in care delivery could significantly impact health quality.[1]
Although evidence-based pharmacotherapies (EBP) are a principal component of
patient care in many of these areas (asthma, diabetes, depression, heart failure, COPD,
hypertension, ischemic heart disease and stroke), 30% to 50% of patients do not take
their medications as prescribed.[2]
Medication adherence is a result of two processes: (1) clinicians prescribing EBP
to their patients, and (2) patients taking their EBP medications as instructed by their
clinicians.[3] However, effective approaches for assuring that these processes are
completed have not been effectively integrated into routine clinical care. This lack of
1
Corresponding Author: Eric L. Eisenstein, DBA, Duke Clinical Research Institute, DUMC Box 3865,
Durham, NC 27710. Telephone: 919-668-8984; Fax: 919-668-7057; e-mail: [email protected]..
J.M. Willis et al. / Decision Support for EBP Detects Adherence Problems 117
effective integration has led to calls for new care models that address appropriate
medication use and make it a part of the patient care process. The present study sought
to use clinical decision support (CDS) to increase EBP adherence within a vulnerable
population and to assess the impact of this CDS technology on service use and medical
costs.
1. Methods
1.1. Study Setting
The Northern Piedmont Community Care Network (NPCCN) provides care

management services for Medicaid beneficiaries residing in five adjoining counties
located in north central North Carolina.[4] This network’s community-based care
management team includes nurses, social workers, community health workers,
nutritionists, and health educators. Services provided by the care management team
include: home assessments, in-home health education and dietary instruction, assistance
scheduling and keeping clinic appointments, and support for obtaining and taking
medications. Team members routinely interact with other network partners including
physicians, nurse practitioners, nurses, and pharmacists. Clinic partners for the present
study included 7 administrative groups with 5 distinct patient appointment scheduling
systems operating at 16 clinic sites.
NPCCN is supported by a regional health information exchange (HIE) and a care
management documentation tool known as COACH, which facilitates communication
between team members collaborating in the care of network patients.[5] This system
receives patient demographic and eligibility data from the North Carolina Office of
Rural Health and Community Care, and clinical, scheduling and billing data from
partner sites. These data include encounter and pharmacy claims data from the State
Medicaid Office and billing data from 9 clinics and all 5 hospitals in the network’s
service region. Data collected by the system include: 1) administrative data
(demographics and identifiers, services used, provider associations and audit trails); 2)
care management data (care management encounters, health risk and environment
assessment, socio-economic data, special needs and care management plans); 3)
claims/billing data (encounters, problems/procedures, and medications); 4) scheduling
data (pending and missed appointments); 5) clinical data (allergies, laboratory results,
and disease-specific care plans) and 6) data on communications (messages and alerts,
referrals and notices of new information). The network is supported by a CDS tool that
uses Web service technology to receive patient data from a client application.[6] It then
processes these data according to an application-independent, pre-programmed set of
rules (e.g., clinical algorithms and guidelines), and returns patient-specific
recommendations to the client application. These recommendations are then used to
notify network personnel of key health events occurring within their patient
populations. Examples of these events include hospital and emergency department
admissions, missed outpatient appointments and the need for well-child visits.
1.2. Study Population
Subjects in the present study included NPCCN Medicaid beneficiaries with at least one
of six targeted IOM priority conditions who were continuously enrolled during the
118 J.M. Willis et al. / Decision Support for EBP Detects Adherence Problems
intervention period. The six conditions included persistent asthma, diabetes,

hypertension, congestive heart failure, ischemic heart disease and stroke. These IOM
priority conditions were identified from claims data using CDS algorithms modeled
after the Healthcare Effectiveness Data and Information Set (HEDIS) with medication
claims criteria excluded to avoid bias inherent in the selection of subjects who were
more adherent to their medications. Because medication information is required to
identify subjects with persistent asthma, we conducted chart audits on all 1,064 subject
who had an asthma diagnosis code in their claims data. These audits identified 617
cases of persistent asthma, 112 cases of intermittent asthma and 335 cases that were
indeterminate.
1.3. Study Interventions
The two study interventions included CDS alerts delivered as reports to primary care
clinics (Figure 1) and email notices sent to network care managers (Figure 2). The first
intervention included reports with a summary list of filled prescription claims along
with a numeric calculation of medication adherence and a graphical depiction of “days
covered” over a 1-year period along with recommendations addressing possible
deficiencies in medication therapy prescriptions relative to EBP guidelines. These
medication summary reports were delivered to the point of care at the time of a
patient’s scheduled visit at their primary care clinic. The second intervention included
email notices to care managers for patients with 1) low medication adherence; 2) no
primary care clinic appointments in the previous 6 months; and 3) no appointments
scheduled for the future. Table 1 summarizes the pharmacotherapeutic rules used in this
study and their criteria for activation. Our methods for developing and testing these
interventions have been previously described.[7, 8]
Figure 1. Sample Medication Summary Report

Figure 2. Sample Notice to Care Manager
Table 1. Summary List of Pharmacotherapeutic Rules and Criteria for Activation
Medication Class Activation Criteria

Statin Diabetes + Age > 40
Diabetes + Coronary Artery Disease if Age < 40
Stroke + LDL Cholesterol ¸100 (Age>17)
Stoke + Diabetes + LDL Cholesterol¸70 (Age>17)
Prior Myocardial Infarction (Age>17)
Stroke + Coronary Artery Disease (Age>17)
ACEI (ARB) Prior Myocardial Infarction or Coronary Artery Disease + Hypertension (Age>17)
Left Ventricular Systolic Dysfunction (Age>17)
Prior Myocardial Infarction or Coronary Artery Disease + Diabetes (Age>17)
Prior Myocardial Infarction or Coronary Artery Disease + Chronic Kidney Disease
(Age>17)
B-Blocker Prior Myocardial Infarction (Age>17)
Left Ventricular Systolic Dysfunction (Age>17)
Warfarin Stroke + Mech Valve (Age>17)
Stroke + Valvular Heart Disease (Age>17)
Stroke + Afib (Age>17)
Anti-Hypertensive Hypertension (Age>17)
Diabetes+ Hypertension (all Ages)
Stroke + Hypertension (Age>17)
Anti-Diabetes Diabetes (all Ages)
Inhaled Steroid or Persistent Asthma + Age>36 mo
Montelukast
B-Agonist Persistent Asthma + Age>24 mo
1.4. Study Design
NCPPN enrollees were randomly assigned using a 1:1:1 allocation by family unit using
a pseudo-random number generator. Subjects assigned to the first group received usual
care, those assigned to the second group received the first intervention (clinic reports)
and those assigned to the third group received both interventions (clinic reports and
care manager notices). In place of informed consent, potential study subjects were sent
a letter explaining the study and a response card to allow them to opt out of the study.
The Duke University Medical Center Institutional Review Board approved this study’s
protocol on December 18, 2007 (Registry Number: Pro00002524) and
ClinicalTrials.gov registration was obtained on August 27, 2009 (Identifier:
NCT00979225). The 12-month intervention period began on December 7, 2009, and
ended on December 6, 2010. Study subject follow-up ended on August 30, 2011. The
primary study outcome measure was medication adherence across all drug classes and
conditions. Table 2 contains a complete list of study outcome measures. Estimates for
study outcomes and between-treatment group comparisons were generated using
generalized estimating equation models with a working correlation matrix to account
for clustering by family unit. Primary analyses were performed for the entire study
population, following the intention-to-treat principle. Secondary analyses were
performed for the subset of patients who generated an alert or medication summary
report to focus on patients who actually received an intervention (“touched” subjects).
Table 2. Study Outcomes Measures
Measurement Focus Measures

Clinical Outcomes Medication adherence across all drug classes and conditions (all subjects and
touched subjects)
Medication adherence by drug class (all subjects and touched subjects)
Medication adherence by disease condition (all subjects and touched
subjects)
Outpatient encounters per 100 pt years (all subjects and touched subjects)
ED encounters per 100 pt years (all subjects and touched subjects)
Hospitalizations per 100 pt years (all subjects and touched subjects)
Care Coordination Care Manager contacts (all subjects and touched subjects)
Costs / Revenues Outpatient costs (all subjects and touched subjects)
ED costs (all subjects and touched subjects)
Hospitalization costs (all subjects and touched subjects)
Pharmaceutical reimbursement (all subjects and touched subjects)
Satisfaction Clinician opinions
2. Results
2.1. Study Population
A total of 2910 subjects met this study’s inclusion criteria. Of these, 155 declined to
participate by returning their opt-out letter, 497 could not participate after their clinic
sites installed new electronic health record (EHR) systems and were unable to send
patient appointment scheduling information to the network HIE, and 39 transferred to
clinical sites not included in this study. A total of 2219 subjects were included, of
which 739 were assigned to usual care, 744 to have the clinic reports intervention and
736 to have both the clinic reports and care manager notices interventions (Figure 3).
2.2. Baseline Characteristics
Patients in the control and intervention groups were well matched with regard to
baseline characteristics. Overall, study subjects were predominantly female, African-
American and young. Most subjects had a history of persistent asthma, diabetes or
hypertension. Given this population’s age, it is not surprising that relatively few
subjects had a history of congestive heart failure (CHF), ischemic heart disease (IHD)
or stroke.
Figure 3. Consort Flow Diagram
2.3. Study Interventions
Study interventions were generated for all study subjects meeting CDS criteria but were
only sent to those randomized to groups 2 or 3 (Table 3). Overall, the numbers of
medication management reports and care manager notices generated did not differ
significantly by treatment group; however, the volume of clinic reports was 5 times that
of care manager notices (5948 versus 1052) as anticipated because the criteria for
generating care manager notices were more stringent than for generating clinic reports.
Table 3. Study Intervention
Control Clinic Reports Clinic Reports + Care Manager

Notices
N=739 N=744 P=* N=736 P=**
Reports
Generated 2061 1951 0.44 1936 0.71
Sent 0 1951 1936
Notices
Generated 353 336 0.69 363 0.52
Sent 0 0 363
Note: * Clinic Report vs. Controls; ** Clinic Reports + Care Manager Notices vs. Controls
2.4. Medication Adherence
There were no treatment-related differences in medication adherence rates overall and

for specific medication classes and medical conditions (Table 4, Table 5). Adherence
rates were lowest for individuals receiving medications for persistent asthma and
highest for those receiving medications for stroke. When we examined 6-month
adherence rates following the first contact (or potential contact) with an intervention
among patients for whom an intervention was generated (but not necessarily sent),
adherence rates (not shown) increased slightly. However, there were no treatment-
related differences between groups.
Table 4. Adherence by Medication Class
Group Arm #1 Arm #2 Arm #3

(Control) (Reports) (Reports+)
Adherence Adherence Adherence
N=739 N=744 N=736
Medication Class % % P= % P=
All Classes 41.3 41.2 0.82 42.9 0.35
Statin All conditions (age >17) 54.0 61.0 0.12 56.3 0.63
ACEI (ARB) All conditions (age >17) 69.7 68.4 0.85 64.2 0.52
B-Blocker All conditions (age >17) 55.2 58.3 0.54 57.1 0.76
Warfarin All conditions (age >17) 48.3 56.1 0.54 37.8 0.26
Anti-Htn All conditions (age >17) 62.8 61.1 0.88 63.9 0.71
Anti-DM Diabetes (all ages) 54.3 53.8 0.90 57.1 0.49
Inhaled Steroid Persistent Asthma 36.9 36.4 0.73 41.8 0.23
+age>36 mo
B-Agonist Persistent Asthma + 25.4 23.9 0.85 24.7 0.84
age>36 mo
2.5. Clinical and Economic Outcomes
Six-month encounter rates per 100 subjects and medical costs per subject (total and by
type) were similar across treatment groups. As expected, the highest encounter rates
were for outpatient care and the greatest costs were for pharmaceuticals and outpatient
care.
Table 5. Adherence by Medical Condition

Adherence Adherence Adherence
isease Condition % % P= % P=
All Diseases 40.6 39.3 0.77 40.6 0.76
Persistent Asthma 27.7 29.0 0.29 27.2 0.47
Diabetes 49.6 50.3 0.83 51.3 0.62
Hypertension 56.6 53.7 0.29 55.8 0.74
CHF 55.8 62.7 0.32 53.9 0.83
IHD 52.4 51.8 0.92 61.6 0.30
Stroke 61.2 54.1 0.62 50.3 0.22
Table 6. Clinical and Economic Outcomes

Encounter Rates # # P= # P=
Outpatient 46.0 46.6 0.42 44.5 0.81
Emergency Department 0.87 0.84 0.77 0.89 0.47
Hospitalization 0.19 0.21 0.96 0.21 0.92
Total 47.0 47.7 0.49 45.6 0.81
Medical Costs $ $ P= $ P
Outpatient 4417 50 0.42 4423 0.75
Emergency Department 423 4 0.58 420 0.07
Hospitalization 154 1 0.95 184 0.93
Pharmaceuticals 5579 66 0.15 5703 0.79
Total 10.573 123 0.32 10.730 0.76
3. Discussion
This study sought to use CDS to increase EBP adherence within a vulnerable
population, and to assess the impact of this CDS technology on service use and medical
costs. We demonstrated significant technical advancement for CDS for promoting
medication adherence by intervening at both the level of the PCP clinic and the
population. Unfortunately, even though the interventions were successfully delivered,
neither the clinic-based intervention (point-of-care reports to clinicians), nor the
population-based intervention (notices to care managers) resulted in increased
medication adherence to EBP. From a positive perspective, we demonstrated that a
CDS intervention can successfully identify medication adherence issues relative to EBP
and mobilize the provider workforce. We detected that the notices sent to care
managers significantly increased the extent of contact that care managers had with
study subjects. One potential weakness of this population-level intervention was that
care manager contact was limited to encouraging patients to arrange follow-up with
their PCP clinic (where a medication management report would be available). Care
managers were permitted to inquire about possible reasons for medication
nonadherence, such as financial or transportation issues. However, they were not
permitted to give clinical advice because such an intervention exceeded their scope of
practice. Perhaps a more effectual population-level intervention would have been to
engage clinical pharmacists or advanced practice nurses to directly address
nonadherence with EBP.[9]

We can conclude from our negative clinical outcome that a printed summary of
medication adherence information based on filled claims and delivered to the point of
care was insufficient to impact overall adherence for EBP. The study was originally
powered to detect an absolute difference in medication adherence of 6.5%. These
calculations were based on a two-sided Type I error rate of 0.025, 80% power, and a
sample size of 1120 per group. With the observed sample size of approximately 740
per group and a Type I error of 0.05, this study had greater than 80% power to detect
differences larger than 7.3%. In addition to the possibility that patients were refractory
to provider-driven efforts to change their medication adherence, three additional
primary points of failure need to be considered: 1) the report content was not useful; 2)
the reports were not effectively integrated into the clinical workflow; and 3) the clinical
setting was not conducive to addressing medication nonadherence issues effectively.
The root cause for low adherence can be quite complex and fixing it may require more
intensive behavior change interventions than a brief talk with the PCP.
Based on findings from our clinic site visits, contextual evaluation, and clinician
surveys, we surmise that the failure to impact EBP adherence was not due to report
content. Our qualitative data indicate that clinicians found the medication report
content helpful for addressing medication nonadherence issues and many anecdotes
were provided in which the reports directly led to discussing medication adherence
issues with a patient. Notably, a majority of clinicians who were interviewed three
months following implementation of the reports rated the reports as helpful or very
helpful.[10] The usefulness of the medication reports is perhaps most strongly
supported by a decision of the leadership of NPCCN to fund conversion of the study
interventions to an operational system so that the reports continued to be available after
this research study ended.
While the qualitative data support the utility of the reports, they indicate a
breakdown in workflow as a possible point of failure. In spite of carefully analyzing
work practices at each clinic site in order to customize an approach for delivering the
medication reports to the point of care, we discovered from our site visits that between
15% to 50% of reports were often not available to clinicians at the time they interacted
with a patient. Consequentially, patients did not receive the full benefit of the
intervention because reports were not always available. As most stakeholders suggested
for future effectiveness, reliable report delivery can be enabled by direct integration of
the reports into existing practice IT systems. From the qualitative analysis with regard
to the third possible point of failure, we also learned that on many occasions even when
reports were available clinicians did not take time to address issues related to EBP
nonadherence. Accordingly, we postulate that the many competing demands during the
clinical encounter may hinder clinicians from addressing nonadherence issues. As
suggested in the contextual evaluation, more resources (time and personnel) and
possibly other team members (e.g. pharmacists) and venues outside of the exam room
may be more conducive to resolving medication nonadherence issues effectively than
the reports provided to clinicians at the point of care in this study.
To summarize, we point out that having information about medication
nonadherence and having tools/resources to address it are two different things and that,
in this group of high risk patients, the knowledge did not beget effective action. That
result does not mean that the knowledge is not useful, but that the path to action needs
to be developed.
Acknowledgements.
This study was funded by the Department of Health and Human Services, Agency for
Healthcare Research and Quality (R18-HS017072).
References
[1] I. O. M. Committee on Quality of Health Care in America, Crossing the Quality Chasm: A New Health
System for the 21st Century. Washington, D.C.: The National Academies Press, 2001.
[2] NEHI. (August 11, 2009). Thinking Outside the Pillbox: A System-wide Approach to Improving Patient
Medication Adherence for Chronic Disease. Available:
www.nehi.net/publications/44/thinking_outside_the_pillbox_a_systemwide_approach_to_improving_p
atient_medication_adherence_for_chronic_disease
[3 E. L. Eisenstein, et al., "Evaluating implementation fidelity in health information technology
interventions," AMIA Annu Symp Proceedings, pp. 211-215, 2007.
[4] J. Cook, et al., "Practice profile. Community collaboration to improve care and reduce health
disparities," Health Aff.(Millwood.), vol. 29, pp. 956-958, 2010.
[5] D. F. Lobach, et al., "Proactive population health management in the context of a regional health
information exchange using standards-based decision support," AMIA Annual Symposium Proceedings,
pp.473-477, 2007.
[6] K. Kawamoto and D. F. Lobach, "Design, implementation, use, and preliminary evaluation of
SEBASTIAN, a standards-based web service for clinical decision support," AMIA Symposium, pp. 380-
384, 2005.
[7] G. Del Fiol, et al., "Improving Medication Adherence in a Regional Healthcare Information Exchange
using a Scalable, Claims-Driven, and Service-Oriented Approach," AMIA Annu Symp Proc, pp. 137-41,
2010.
[8] K. Kawamoto, et al., "Development and evaluation of an improved methodology for assessing
adherence to evidence-based drug therapy guidelines using claims data," AMIA Annu Symp Proc, pp.
394-8, 2007.
[9] S.B. Calvert, et al., "Patient-focused intervention to improve long-term adherence to evidence- based
medications: a randomized trial," Am Heart J. 2012 Apr:163(4):657-65.e1.
[10] L.L. Wood, et al, "Utility of point-of-care medication management reports generated through a decision
support Web service," AMIA Annu Symp Proc, p.1312, 2010
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-126
Analysis of Continuous Oxygen Saturation

Data for Accurate Representation of Retinal
Exposure to Oxygen in the Preterm Infant
Josie CIRELLI a, Carolyn MCGREGOR a,1, Brenda GRAYDON b, Andrew JAMES b, c
a
University of Ontario Institute of Technology, Oshawa, ON, Canada
b
The Hospital for Sick Children
c
University of Toronto, Toronto, ON, Canada
Abstract. Maintaining blood oxygen saturation within the intended target range
for preterm infants receiving neonatal intensive care is challenging. Supplemental
oxygen is believed to lead to increased risk of retinopathy of prematurity and
hence managing the level of oxygen within this population is important within
their care. Current quality improvement activities use coarse hourly spot readings
to measure supplemental oxygen levels as associated with targeted ranges that vary
based on gestational age. In this research we use Artemis, a real-time online
healthcare analytics platform to ascertain if the collection of second by second data
provides a better representation of retinal exposure to oxygen than an infrequent,
intermittent spot reading. We show that Artemis is capable of producing more
accurate information from the higher frequency data, as it includes all the episodic
events in the activity of the hour, which provides a better understanding of oxygen
fluctuation ranges which affect the physiological status of the infant.
Keywords. Oxygen, SpO2, Retinopathy of Prematurity, Clinical Decision Support,

Quality Improvement
Introduction
Medical and technological advances in neonatal intensive care over the past 50 years
have increased survival rates for preterm infants whom are <37 weeks gestational age
(GA) and weigh <2500 grams. In 2009–2010, the Canadian Institute for Health
Information reported that one in 12 infants (8%) are born prematurely in Canadian
hospitals, and that more than 6% of those infants weigh <2500 grams [1].
Physiological immaturity is a precursor to many imminent complications involving the
lungs, brain, and eyes which result in the use of supplemental oxygen therapy to aid
with the pulmonary inadequacy [3]. In the preterm infant population, optimal oxygen
saturation ranges are still unknown and evidence based practices have set guidelines for
oxygen administration at 85-92% to maintain and reduce the risk of complications
while increasing the benefits of oxygen [8]. Supplemental oxygen exposure is an
established risk factor for Retinopathy of Prematurity (ROP) as the immature retina is
susceptible to injury [4, 5]. ROP is described as a vascular disorder of the retina, which
occurs after birth, among preterm infants and involves prolonged oxygen
1
Corresponding author
J. Cirelli et al. / Analysis of Continuous Oxygen Saturation Data for Accurate Representation 127
administration and fluctuation of oxygen, associated with hyperoxia and hypoxia [4].
At birth the immature retina is exposed to an excess of oxygen saturation which
decreases the sensitivity of cells due to the over exposure of oxygen therefore, reduces
the normal stimulus for development of the blood vessels and encourages abnormal
retinal vascular migration [5].
We use the Artemis platform, a real-time analytic system that captures and
analyses high fidelity physiological data streams from bedside medical devices in the
NICU. Its multidimensional approach uses time-series data and temporal data mining
to identify relationships between the data streams and specific clinical conditions that
affect the preterm neonate. The primary goal of this exploratory study was to determine
if the collection and analysis of high fidelity blood oxygen saturation levels (SpO2)
provides a better representation of the immature retina’s exposure to potentially
damaging oxygen levels than infrequent, intermittent spot recordings of SpO2.
1. Related Works
Recent research has shown that not only hyperoxia but also hypoxia and fluctuations in
retinal oxygen levels can contribute to the development of ROP [3,6,8]. A study by
Hagadorn et al., showed that SpO2 is in the intended target range for 50% of the time as
there is considerable fluctuation above and below the intended target range [8].
Hagadorn et al. have also shown that preterm infants less than 28 weeks gestational age
experience fluctuations in SpO2, but these episodic events are frequently transient
and/or have resolved by the time the nurse observes the baby, and therefore
opportunities for intervention are limited [8]. Current research is exploring how new
technologies can help to control oxygen delivery so to better understand fluctuations in
SpO2, and consequently, improve target range compliance [6, 8].
2. Methods
In this exploratory study, we used the Artemis platform to capture high fidelity data for
SpO2, to document retinal exposure to oxygen. We extracted SpO2 values both hourly
and at the frequency of 1Hz, for a preterm infant who was receiving supplemental
oxygen over a consecutive 4 day period. The representations of retinal exposure to
oxygen of these two approaches for capturing SpO2 were compared. The infant was
enrolled in a REB approved nosocomial infection research project being conducted in
the NICU at The Hospital for Sick Children, Toronto.
2.1 Data Collection
For this study, the Knowledge Extraction component of Artemis was used to extract
raw SpO2 values (collected by Masimo SET pulse oximeter) over days 27, 28, 29, and
30 of a study month, on a convenient sample data of a de-identified infant <32 weeks
GA [9]. The top of the hour spot readings were extracted as a subset of that dataset.
128 J. Cirelli et al. / Analysis of Continuous Oxygen Saturation Data for Accurate Representation
O2 saturation by hour using second by second data readings, day 27

Achieved continual data
3600
3000
2400
readings
1800
1200
600
0
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
Hours of the day 16 17 18
19 20 21
22 23
SpO2 Ranges <85% BELOW 85%-92% TARGET 93% -100% ABOVE
Figure 1. Frequency histogram for Day 27 of SpO2 from Artemis Data.
2.2 Statistical Methods
In order to understand the copious amounts of data that Artemis produces on a

continual basis, we used the univariate analysis on the raw data, this enabled us to
focus on three characteristics of the data; distribution, central tendencies and dispersion
of values around the central tendency. Frequency histograms were constructed with
hourly readings summarized into SpO2 ranges of; <85%, 85%-92%, 93%-100%
representing ‘below’, ‘target’, ‘above’ respectively. Further analysis for shift windows
was also performed.
The representation of retinal exposure to oxygen for the high fidelity data stream
was compared to that of the spot readings by analysing the dispersion of values around
the central values using the paired sample t-test with p < 0.05.
3. Results
The characteristics of the continual collection of achieved SpO2 data using day 27 are
shown in Figure 1. The availability and capture of the infants’ machine acquired
bedside SpO2 transcutaneous oxygen saturation value achieved per hour was examined.
On day 27, ten of the 24 hourly periods showed oxygen saturation average values
which fluctuated more than 5% values from the top of the hour value, within this 24
hour period, one of those hours fluctuated 12%. Similar differences were noted on the
other three days within the study.
Achieved oxygen saturation readings between two shift windows, using the
Artemis second by second data readings (Figure 2) and the top of the hour spot
readings (Figure 3) were compared and examined for all days. Using the data on day 28
seen in figures 2 and 3, we observed that the appearances of the two histograms are
dissimilar. The percent values for each SpO2 range represent a different value in both
the day and the evening shift. Both values were represented by the frequency percent
which is derived from the number of times the achieved SpO2 value was captured in the
% range divided by the total data readings captured for the shift.
O2 saturation percent frequency using Artemis 1 Hz data collection, day 28
60% 53.31
55% 45.79
50%
45% 36.38 38.97
40%
35%
30%
25% 17.83
20%
15% 7.72
10%
5%
0%
7am - 7 pm Day Shift 7pm - 7 am Evening Shift
<85% Below 85%- 92% Target 93% - 100% Above
Figure 2. SpO2 percent frequency using Artemis 1 Hz data collection, day 28
O2 saturation percent frequency using top of the hour spot readings, day 28
75% 66.67
70%
65%
60%
55%
50% 41.67
45%
40% 33.33
35% 25.00 25.00
30%
25%
20%
15% 8.33
10%
5%
0%
7am - 7pm Day Shift 7pm - 7 am Evening Shift
<85% Below 85% - 92% Target 93% - 100% Above
Figure 3. SpO2 percent frequency using top of the hour spot readings, day 28
As can be seen in figures 2 and 3, there is a significant difference in the

distribution between below, target and above for the day shift. The Artemis data
demonstrates that there was less above target range oxygenation across the shift and
more under oxygenation during the shift by 21% and 9% respectively.
Within the night shift the below target was 18% lower in the Artemis data and the
above target was 20% higher.
4. Discussion
This exploratory study has shown potential that higher fidelity oxygen saturation data
collection provides greater detail of retinal exposure to oxygen than infrequent,
intermittent spot readings. The high frequency data collection provides a better
representation of all activity occurring in the full hour as episodic events are captured
and included to represent a clear picture of SpO2 occurring during each hour. As a
result this approach provides a greater ability to understand the retinal exposure to
oxygen values and percentages with more accuracy.
130 J. Cirelli et al. / Analysis of Continuous Oxygen Saturation Data for Accurate Representation
In addition to what has been reported in this paper, Artemis provides more in-
depth information than that of a single spot reading. It allows for detailed information
to be extracted from the raw data such as; ranges, times, frequency, hourly approaches
and outliers which can be used to help expose what is occurring at a particular time, for
any particular length of time. This aids in the tracking of hypoxia and hyperoxia and
provides a better understanding of the patient’s current level of oxygenation. Using this
detailed level of data can provide a more comprehensive data set inference from than
just a single value, in a single time frame which, would lack the connectivity and
relevance of the oxygen saturation which occurred during the hour. Analyzing rapid
and wide fluctuations in an hourly dataset can provide extensive information for the
many neonatologists, residents, fellows, nurses, and respiratory therapists to draw from.
This has great potential to improve adjustments in titrating the transcutaneous blood
oxygen saturation levels at the bedside. The ability to accurately reflect the activity of
the hour and convert copious amounts of data into visual charts and graphs, provides a
better understanding of the information which can be passed on to another health
professional at the end of the shift as a clinical and educational tool.
The Artemis platform has the capability to support clinical research to quantify
the relationship between retinal exposure to oxygen and ROP more precisely and
accurately. There is also potential for the discovery of other relevant causative factors.
5. Conclusion
This paper has presented an exploratory study to compare the target range analysis for
blood oxygen saturation levels SpO2 when data is only recorded as top of the hour
readings verses analysis of the readings each second. The analysis was performed using
data collected via the Artemis platform. High fidelity blood oxygen saturation data
provides a better representation of retinal exposure to oxygen than infrequent,
intermittent spot readings. In future research publications we will report on our
research to study the association of the SpO2 values recorded in Artemis and the
association with ROP risk scores.
References
[1] CIHI, Highlights of 2009-2010 selected indicators describing the birthing process in Canada. [cited May
12, 2011], Available from: www.cihi.ca.
[2] CIHI, Too early, too small: a profile of small babies across Canada.[ cited 2009]. Available from:
https://secure.cihi.ca/free_products/too_early_too_small_en.pdf.
[3] K. Johnson, S.D. Scott, K.D. Fraser, Oxygen use for preterm infants. Factors that may influence clinical
decisions surrounding oxygen titration. Advances in Neonatal Care 11, 1 (2011), 8-14.
[4] D.K. Wallace, K.A. Veness-Meehan, W.C. Miller, Incidence of severe retinopathy of prematurity before
and after a modest reduction in target oxygen saturation levels. . American Association for Pediatric
Ophthalmology and Strabismus 11 (2) (2007), 170-174.
[5] K.W. Wright, D. Sami, L. Thompson, R. Ramanathan, R. Joseph, S. Farzavandi, A physiologic reduced
oxygen protocol decreases the incidence of threshold retinopathy of prematurity. Trans Am Ophthalmol
Soc 104 (2006), 78-84.
[6] J.S. Greenspan, J.P. Goldsmith, Oxygen therapy in preterm infants: Hitting the target. Pediatrics 118, (4)
(2006) 1740-1741.
[7] S. Vanderveen, T. Mansfield, E. Eichenwald, Lower oxygen saturation alarm limits decrease the severity
of retinopathy of prematurity. American Association for Pediatric Ophthalmology and Strabismus 10
(2006), 445-448.
[8] J.I. Hagadorn, A.M. Furey, T. Nghiem, , C.H. Schmid, D.L. Phelps, D.M. Pillers, Achieved versus
intended pulse oximeter saturation in infants born less than 28 weeks’ gestation, Pediatrics 118(4)
(2006), 1574-82.
[9] M. Blount, M. Ebling, M. Eklund, A. James, C. McGregor, D. Sow, Real-time analysis for intensive
care, IEEE Engineering in Medicine and Biology Magazine, March/April (2010), 110-118.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-132
Tailoring Decision Support to Suit User

Needs: A Diagnostic Imaging Example
Janessa GRIFFITH a,1, Elizabeth M. BORYCKI a , Andre W. KUSHNIRUK a
a
School of Health Information Science, University of Victoria,
Victoria, British Columbia, Canada
Abstract. Unnecessary diagnostic imaging (DI) examinations raise concerns for

patient safety and place stress on human and financial resources. To reduce
unnecessary DI examinations, several Canadian pilot studies have investigated
how decision support systems (DSS) could be utilized. Based on interview results
from our previous research, in addition to a literature review, themes emerged that
influenced the features and design of a DI DSS prototype. Features include having
the referring professional indicate how the results of the examination will be
utilized (i.e. for diagnosis or patient management), increasing communication
between referring physicians/nurse practitioners and radiologists, and displaying
previous DI examinations (or orders that are scheduled to take place) to avoid
duplicate orders. Presenting a patient’s cumulative radiation exposure, and having
resources for information support to guide physicians through challenging clinical
decisions are two other features included in the DSS prototype. By incorporating
physician perspectives and current literature into the design, this DSS aims to
promote the appropriate use of DI resources by supporting physicians and nurse
practitioners in their DI ordering practices.
Keywords. diagnostic imaging, medical imaging, decision support tools, decision

support system, appropriateness, radiation exposure, patient safety, order entry
system, prototype
Introduction
According to the Canadian Association of Radiologists (CAR), 30% of diagnostic

imaging (DI) in Canada may be unnecessary [1, 2]. These unnecessary DI examinations
strain financial and human resources. As well, some DI modalities (for example, x-ray,
computed tomography [CT], and positron emission tomography [PET]) expose patients
to radiation [3], making the unnecessary examinations a patient safety concern.
Unnecessary, or inappropriate, DI ordering refers to situations where the most ideal
imaging is not selected, when imaging would not change or support patient
management, when the examination is performed too early, or when a duplicate order
is placed [3]. Again, these inappropriate orders place stress on human and financial
resources and raise patient safety concerns.
In the author’s previous work, Canadian physicians (both general practitioners and
specialists) were interviewed regarding their DI ordering practices [4]. Specifically, the
researcher was interested in how physicians proceed with DI decision making, as well
1
Corresponding Author: Janessa Griffith, School of Health Information Science; Email:
[email protected]
J. Griffith et al. / Tailoring Decision Support to Suit User Needs: A Diagnostic Imaging Example 133
as what methods of information support physicians would seek during challenging

clinical scenarios. Physician participants were also asked to describe what factors
contribute to inappropriate ordering. Lastly, participants identified potential solutions
to inappropriate ordering [4]. This previous study, accompanied by a literature review,
formed the basis of a DI decision support system (DSS) prototype. DSSs are
“computerized tools that aim to improve patient care by putting best-practice
recommendations directly onto computer and hand-held devices of physicians” [5]. The
prototype was created using Microsoft’s® PowerPoint tool. The prototype is strictly
visual, and is being used as a communication tool to illustrate what features are
important to include in a real system and to elicit further feedback from future users.
For example, the prototype could be used to communicate what is expected from
computer programmers, or to demonstrate to funders how their contributions will be
spent.
A DSS prototype may be incorporated into an electronic health record and may be
used to learn about physician perceptions and reactions to the DSS design. For example,
in a DSS pilot project in New Brunswick, Canada, physician participants expressed
frustration with logging on to a separate system (i.e. the DSS). They also noted the
inconvenient physical location of the technology [6]. Thus, this DSS prototype was
integrated into an electronic health record.
1. The Prototype
In this paper we describe a DI DSS prototype. In the prototype, a tab on the horizontal
menu bar on the electronic health record supports physician navigation of the DI
portion of the system. By scrolling over this tab, physicians can access previous DI
examinations through the picture archiving and communications system, view current
guidelines, or order DI. When proceeding to order DI, physicians/nurse practitioners
are presented with a series of closed-ended questions in order to provide end-users with
the most relevant decision support. After answering these questions, the system
displays the recommendation. This feature is modeled after a DSS used at a hospital in
the US that produces a utility score to indicate the appropriateness of a DI examination
[7].
After a recommendation is produced in the DSS prototype, the physician or nurse
practitioner can accept or override the recommendation. If the recommendation is
rejected, the end-user must provide rationale in a free-text box. If the recommendation
is accepted, the physician may proceed to order the DI. Not only does the DSS suggest
the most suitable type of DI, it also recommends an urgency classification that is used
for booking the appointment. The order entry component of this DSS suggests multiple
dates and locations for the appointment. The patient will leave their consultation with a
confirmed DI examination appointment.
2. Derivation of Prototype Features
Based on findings from a previous study [see 4], and from a literature review, features
emerged that could be considered for inclusion in future DSS development. These
features included questioning the utility of the DI examination, increasing
communication between referring physicians/nurse practitioners and radiologists, and
134 J. Griffith et al. / Tailoring Decision Support to Suit User Needs: A Diagnostic Imaging Example
displaying previous DI examinations (or orders that are scheduled to take place) to
avoid duplicate orders. Other features include presenting a patient’s cumulative
radiation exposure, and having resources for information support to guide physicians
through challenging clinical scenarios. Overall, results from our previous work [see 4],
formed the basis for the features included in the DSS prototype described below.
2.1. Utility of DI Examination
To better understand DI decision making, physician participants in the author’s

previous work [4] were asked to describe their thought processes when presented with
a challenging clinical scenario (i.e. when they were unsure of whether to order DI, or
unsure which modality to use). Participants described questioning the utility of a DI
examination, whether it was to reach a diagnosis, to support patient management, or
both. This is similar to CAR’s definition of a useful DI examination: “one in which the
result—positive or negative—will alter clinical management and/or add confidence to
the physician’s diagnosis” [3]. Thus, the prototype includes an initial question, asking
the physician to select whether the DI examination is for diagnosis, patient
management, or both. This step provides the radiologist with more information, and
requires the referring physician to consider the utility of the DI examination.
2.2. Requisition Communication
In research conducted by the author [see 4], physician study participants suggested that
there is a need to increase opportunities for physician communication when completing
DI requisitions. Providing physicians with opportunities to communicate could help to
promote more appropriate DI ordering. More specifically, participants noted the
importance of providing sufficient information on the requisition; however, some
requisition formats (paper and electronic) restricted the amount of wording referring
physicians could enter. Thus, this prototype includes requisition order entry features
with a large free-text space for additional comment. Physicians can click a button if
they require more space than is provided (see Figure 1).
2.3. Duplicate Orders
In previous research work [4], physician participants were asked how they determine if
a DI examination has already been performed or scheduled to take place. Interestingly,
67% expressed they would ask the patient directly [4]. The remaining participants
described using an electronic system to determine if the examination had been
performed or scheduled, but could only do this within their jurisdiction [4]. Thus, the
prototype includes a pop-up alert to warn physicians if a duplicate order has occurred.
This alert includes the date and type of DI examination. The alert does not cancel the
requisition being placed, as some duplicate orders may be necessary.
2.4. Radiation Dose Monitoring
Certain DI examinations expose patients to radiation (for example, X-ray, CT, and
PET). For example, a CT examination of the abdomen or pelvis exposes a patient to
approximately 4.5 years worth of natural background radiation [3]. As noted previously,
J. Griffith et al. / Tailoring Decision Support to Suit User Needs: A Diagnostic Imaging Example 135
Figure 1. Requisition Communication
physicians may ask their patients directly to determine if previous imaging had been
performed. Having this information available could help physicians make more
informed decisions and promote patient safety.
2.5. Information Support
As described in the previous study [4], physician participants noted consulting a

physician (either a colleague/specialist, or radiologist), or searching the literature (using
UpToDate®, “an evidence-based, peer-reviewed medical information resource” [9], or
Google Scholar) as their top methods of obtaining information to guide challenging
clinical decisions with respect to DI. This is similar to another study where physician
participants rated consulting a radiologist or, searching the literature through
UpToDate®, a specialty journal, or Google’s search engine as their top methods of
obtaining information support [8]. Common to both of the studies, consulting another
physician or searching the literature were used for information support. To support end-
user preferences for obtaining information support, the prototype includes buttons to
directly access UpToDate®, specialty journals, and an internet search engine. The
prototype also includes a button to access a directory of physicians and their contact
information to support physicians who wish to consult another physician. Therefore,
including direct access to the top methods of information support that physicians
identified using in their work could promote more appropriate ordering.
3. Discussion and Conclusion
Inappropriate DI ordering places stress on human, and financial resources and raises
patient safety concerns. Several Canadian DI DSS pilot studies have demonstrated
there is an interest in utilizing this technology to promote more appropriate ordering.
Results from previous research and a literature review formed the basis for the
development of this DSS prototype. The prototype included features such as having the
end-user identify the utility of the DI examination as well as increasing opportunities
for communication between referring professionals (physicians and nurse practitioners)
and radiologists. In order to prevent repeated DI ordering, this DSS prototype displays
136 J. Griffith et al. / Tailoring Decision Support to Suit User Needs: A Diagnostic Imaging Example
examinations that have already been performed or are scheduled to take place. The
prototype also displays the cumulative DI-related radiation exposure a patient has
received to help physicians make more informed decisions. As well, the top reported
methods of information support used by referring physicians (i.e. using UpToDate®,
Google®, specialty journals, and contacting other physicians) were incorporated into
the prototype to support decision making involving challenging cases. As inappropriate
DI ordering raises concerns for patient safety and places a strain on human and
financial resources, investigating ways to reduce such ordering has become critical. As
described in this paper, based on interviews from our previous study, and from a
literature search, several themes arose that influenced the introduction of specific
features into this DSS prototype. Future work will focus on the usability testing of the
prototype described in this paper.
References
[1] Canadian Association of Radiologists. Do you need that scan? [updated 2005], [Cited Aug 8, 2012],
Available from: http://car.ca/uploads/news%20publications/car_cat_scan_eng.pdf.
[2] Canadian Association of Radiologists. As many as 30% of diagnostic imaging procedures deemed
unnecessary: Canadian Association of Radiologists helps physicians choose best test first, [updated
2010], [Cited Nov 3, 2010], Available from:
http://www.car.ca/uploads/news%20publications/car_news_release_20100927.pdf .
[3] Canadian Association of Radiologists, Diagnostic imaging referral guidelines: A guide for physicians,
1st ed. Saint Laurent, Quebec, 2005.
[4] Griffith JK. Diagnostic imaging ordering practices by referring physicians: A qualitative approach
[thesis]. Victoria (BC); University of Victoria; 2012.
[5] Health Council of Canada. Decisions, decisions: Family doctors as gatekeepers to prescription drugs
and diagnostic imaging in Canada, [updated 2010], [Cited 2012 Sep 5]. Available from:
http://healthcouncilcanada.ca/docs/rpts/2010/HSU/Decisions.
[6] Atlantic Health Sciences Centre, Canadian Association of Radiologists, Medicalis. Demand-side
control of diagnostic imaging through electronic clinical decision supports: A pilot using
appropriateness guidelines. [updated 2005], [Cited Oct 6, 2010]. Available from:
http://www.car.ca/uploads/nb_approp_051005_finaldraft.pdf .
[7] D. Rosenthal, J. Weilburg, T. Schultz, J. Miller, V. Nixon, K. Dreyer, J. Thrall, Radiology order entry
with decision support: Initial clinical experience, JACR 3 (2006), 799-806.
A. Bautista, J. Amorosa, A. Burgos, B. Nickel, A. Tilara, J. Yoon. Do clinicians use the American
College of Radiology Appropriateness Criteria in the management of their patients? Am J Roentgenol
192 (2009), 1581-1585.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-137
Supporting Cystic Fibrosis With ICT

Erin ROEHRERa1, Elizabeth CUMMINGSa, Paul TURNERa, Jenny HAUSERc, Helen
CAMERON-TUCKERb, Sean A BEGGSc, Nicole A MICALLEFc, Claire
WAINWRIGHTd, Joyce CHENEYd, Mel JESSUPf, Heather SADDINGTONd, Leonie
ELLISa, Haydn WALTERSb, David W REIDe
a
eHealth Services Research Group, School of Computing & Information Systems,
University of Tasmania, Australia, bMenzies Research Institute, University of
Tasmania, Australia, cRoyal Hobart Hospital, Tasmania, Australia, dQueensland
Children’s Medical Research Institute and The University of Queensland, Brisbane,
Australia, eQueensland Institute of Medical Research, Brisbane, Australia, fResearch
Centre for Clinical and Community Practice Innovation, Griffith University, Gold
Coast, Australia
Abstract. ICT use in cystic fibrosis management provides an alternative means of

information supply to individuals, families, health care professionals and other
stakeholders. The purpose of this paper is to present the evolution of a series of
projects culminating in a project that translates the previous research into practice.
In this paper the sequential nature of the projects will be detailed. The three
projects explored are the Pathways Home for Respiratory Illness Project
(Pathways Home), Enhancing Self-Efficacy for Self-Management in People with
Cystic Fibrosis and the Tasmanian Community Fund Project (myCF pilot).
Keywords. ICT supported chronic disease self management, cystic fibrosis,

incremental project development
Introduction
The use of information communication technology (ICT) in chronic disease

management has been the subject of various reviews [1-3]. Incorporation of ICT in
chronic disease self-management has the potential to provide an alternative means of
information supply to individuals, families, health care professionals and other
stakeholders [4]. Understanding the motivating and influencing factors that underlie
consumer engagement in eHealth initiatives is an important contextual requirement for
the successful running and implementation of ICT projects.
The purpose of this paper is to present the evolution of a series of inter-linked
projects culminating in translation of the previous research into practice. The drivers
for each project’s conceptualisation, methodology and development, will be detailed.
Finally, the influence of the combined results of the projects will be discussed.
Three projects, undertaken by a multi-disciplinary team, are explored, the
Pathways Home for Respiratory Illness Project (Pathways Home), Enhancing Self-
Efficacy for Self-Management in People with Cystic Fibrosis (CFA Project) and the
1
Corresponding Author: Erin Roehrer, School of Computing & Information Systems, University of
Tasmania, Private Bag 87, 7001, Hobart, Tasmania.E-mail: Erin,[email protected]
138 E. Roehrer et al. / Supporting Cystic Fibrosis with ICT
Tasmanian Community Fund Project (myCF pilot). Each project is discussed from a
health informatics perspective.
1. Pathways Home
Pathways Home for Respiratory Illness was a joint research project involving the
Schools of Medicine, Nursing and Midwifery, and Computing and Information
Systems at the University of Tasmania. The project was funded by the Commonwealth
Department of Health and Ageing, under the Australian Health Care Agreement.
A major aim of the Pathways Home project was to investigate interventions that
offered the potential to facilitate the development of self-management skills and self-
management self-efficacy in people with cystic fibrosis (CF). The project responded to
a range of challenges presented to people with CF in Tasmania including: the high birth
incidence of CF in Tasmania; a geographically dispersed CF population receiving care
through regional healthcare facilities rather than via the conventional centralised CF
Centre care model and the resultant impact on regular, physical clinic visits; and the
fact that people with CF are surviving well into adulthood.
The CF arm of the Pathways project, which also assessed ICT support for older
patients with chronic obstructive disease (COPD)[5], aimed to investigate whether the
introduction of an ICT tool in CF, specifically a mobile phone application, could be
used for daily symptom monitoring to encourage the development of self-management
self-efficacy. The study explored whether individuals with CF would use the tool and if
its use would make a difference to their self-management behaviours and quality of life.
A total of 19 participants across Tasmania participated in the investigation. These
participants were randomised into 3 groups;
Intervention 1: Given access to health-mentoring to foster self-efficacy for
self management (face-to-face and via telephone).
Intervention 2: Given access to health-mentoring to foster self-efficacy for
self management (face-to-face and via telephone) and also provision of an ICT
supported self-monitoring program (via mobile phone).
Control Group: Continued to receive normal level of care.
The project ran for 6 months with a 6-month washout. Both qualitative and
quantitative data was collected at commencement; 3 and 6 months; and 6 months post
completion. The initial evaluation of the results of Pathways Home indicated that there
was sufficient evidence to undertake further evaluation of these intervention types. The
project also demonstrated that mentoring did assist in developing self-efficacy for self-
management behaviours, but the pilot study was too small to make definitive
conclusions on the ICT arm, but the feasibility of this approach was confirmed [5].
2. Improving self-efficacy in adolescents and adults with Cystic Fibrosis
The second project was funded by the Australian Cystic Fibrosis Foundation and
extended the Pathways Home project that had been conducted in adults into
adolescents and older children with CF. The project was conducted in Queensland
through collaboration with the Royal Children’s Hospital and their CF outreach service.
This project examined two self-management related strategies, task-specific self-
E. Roehrer et al. / Supporting Cystic Fibrosis with ICT 139
efficacy and self-monitoring, designed to improve self-management behaviours and

quality of life amongst CF patients.
The project participants’ ages ranged between 12-19 and they were recruited from
the Royal Children’s Hospital and Gold Coast Hospitals outpatient and outreach clinics.
Participants were randomised to one of three groups:
Intervention 1: Health-mentoring initiated and reviewed by a mentor who
may or may not have been resident in the local area, thus allowing remote
mentoring by telephone to be assessed. The ICT application was accessed
either through a mobile phone provided by the trial or via a desktop PC and
allowed the individual to record their symptoms via a daily electronic diary.
Intervention 2: Participants undertook the self-efficacy program and allocated
a mentor, without the daily diary or use the ICT application.
Control: Participants in this arm received normal multi-disciplinary CF care.
A total of 43 participants were recruited for this project. 15 participants received
health-mentoring plus the ICT application. Participants took part in the trial for six
months, with a further six months of follow up. Data collection involved both
quantitative and qualitative assessments collected at base-line, 3, 6 and 12 months.
The electronic diary consisted of a set of questions and an additional randomly
generated question to improve data quality. Participants were able to view a summary
of their clinical data as feedback on how their symptoms were tracking. The project
mentors were also able to review their patients remotely.
The design of the CFA Project included the electronic patient dairy that could be
accessed through a mobile phone or a PC; a mobile server that captured and sent
clinical information to the participants using the mobile phone application; and a
database with a web-based interface that stored the data present in the platform.
The outcomes of the CFA Project provided sufficient positive response from the
qualitative interviews and usage statistics to indicate that further investigation with a
broader scope encompassing a more sophisticated suite of products to support the CF
community may be beneficial. Useful information was also obtained on aspects of self-
monitoring that were not viewed favourably by these younger CF patients such as the
need to enter diary data on a daily basis, which not surprisingly was not an attractive
option for young adolescents.
3. myCF
The acknowledgement of the potential beneficial supportive role of ICT in a health-

mentoring environment, and the desire to assist the Tasmanian CF community in
raising levels of self-efficacy for self management led to the development of the myCF
project, utilising web based resources that can also be accessed via a mobile phone. The
Tasmanian Community Fund provided support for this current project.
MyCF has three main aims:
1. To improve access to educational material on CF; A web based information
portal which will contain expert reviewed health information sheets and links
to other relevant sites.
2. To increase Community Support; Availability of community support from
peers and families located in Tasmania through a secure online chat room.
140 E. Roehrer et al. / Supporting Cystic Fibrosis with ICT
3. To develop a Health-Mentor System: Introduction of a health-mentor system

of trained health professionals to encourage the development of self-
monitoring and increase self-awareness about their condition.
The myCF project is an evolution of the Pathways Home and Australian Cystic
Fibrosis Foundation projects and incorporates the daily symptom diary into a web
portal, allowing integration of educational, enhanced communication and self-
monitoring facilities into the one ‘virtual’ location. The myCF website design continues
to undergo iterative development in order to deliver a clean, accessible site for users.
The combined platform of a web-based delivery, incorporating a mobile phone
application as an additional input device, integrates ICT in a way that allows
individuals with busy lives to access the portal daily if they wish to do so, but
frequency of use is dictated entirely by the individual. Thus, the mobile phone allows
daily symptom input with minimal interruption to everyday routines.
The project pilot was staggered and consisted of an active 6 week trial of the
symptom monitoring diary of the myCF website, conducted between June 2011 and
September 2011. Participants of the pilot were 15 Tasmanian individuals with CF. The
potential project participants were identified through attendance at one of the
Tasmanian CF clinics. Participants were randomly recruited from different age groups;
paediatric, adolescent and adults to allow for a more comprehensive assessment of how
each age group would interact and use the myCF website. Data collection involved
both quantitative and qualitative assessments collected at pre and post pilot periods.
The myCF project endeavours to facilitate the implementation of an ICT support
model for clinical care in Tasmania for CF. Throughout the planning stage of the
myCF project, diary baseline questions were developed and loaded into a database to
provide the reference base for the self-monitoring aspect of the site. The core questions
developed as part of the Pathways Home project have been enhanced and added to. The
participant can select the questions that are relevant to their disease manifestations, i.e.
predominantly bowel or respiratory-related and they receive quantifiable data feedback
in graphical form based on their subjective diary entries. This re-work of the diary
questions for the myCF project pilot ensures applicability for the individual with CF.
The myCF project pilot study demonstrated that participants perceived the single
platform containing educational, mentoring and symptom-monitoring elements to be
beneficial. Pilot study participants also indicated that the use of a symptom-monitoring
diary would be of far more benefit if performed in conjunction with health-mentoring
or peer support.
4. Discussion
The succession of three projects building on each projects’ findings enables the
research team to ensure the myCF project takes its place as a translational research
project in the model of CF care in Tasmania. The feedback from the series of projects
suggested that not only was online symptom monitoring feasible but that it is most
likely to be beneficial when combined with access to health-mentoring opportunities
and educational resources. The progression from single platform offerings to that of a
platform (myCF) with multiple functions was aided by knowledge from the previous
projects. The myCF project pilot additionally combined the web platform and ability to
be mobile in order to overcome some of the limitations highlighted by participants
from the sole use of the mobile phone or PC.
E. Roehrer et al. / Supporting Cystic Fibrosis with ICT 141
The myCF project pilot was trialled by representatives of the three generations;
older children, adolescents and adults. This application allowed for a holistic
representation of the CF community, removing the adult only or adolescent only focus
of the previous two projects. The recruitment strategy reflects the applied nature of
research into practice of the myCF project pilot.
The evolution of Pathways Home, CFA Project and myCF represents a natural
progression and the challenge now is to determine which components of the myCF
project pilot are of most benefit and why. By monitoring usage of myCF we will be
able to explore how patients and their families interact with particular components of
the ICT tool and gain better understanding of how myCF may be used to support
chronic disease self-management.
The size of the sample populations and at times difficulty with ensuring
sustainable use of the IT interventions were some of the limitations of these projects.
This paper does not explore the findings of each project in detail, rather presents an
overview of each project to highlight the method of building on the previous projects.
5. Conclusions
ICT use in chronic disease support has increased in use from a health consumer’s point
of view. This paper follows the journey of three projects involving clinicians and
eHealth researchers that explored health mentoring supported in some cases by ICT
with the aim of improving self-management and self-monitoring activities for those
with chronic disease, in particular, cystic fibrosis.
The potential for ICT use in self-monitoring, particularly using mobile devices, has
been highlighted by all three projects. The myCF pilot combines an on-line symptom
diary with educational material and peer support with the aim of improving care for
people with cystic fibrosis who live remote from specialist CF services such as the
patient population residing on the island state of Tasmania.
References
[1] K. Akesson., B. Saveman & G. Nilsson. Health care consumers’ experiences of information
communication technology – A summary of the literature, International Journal of Medical Informatics.
2007; 76: 633-645.
[2] D. Dorr., L.M. Bonner., A.N. Cohen., R.S. Shoai., R. Perrin., E. Chaney & A.S. Young. Informatics
Systems to Promote Improved Care for Chronic Illness: A Literature Review, Journal of the American
Medical Informatics Association. 2007; 14: 156-163.
[3] M.R. Solomon. Information Technology to Support Self-Management in Chronic Care, Disease
Management and Health Outcomes. 2008; 16: 391-401.
[4] B. Celler., N. Lovell & J. Basilakis. Using information technology to improve the management of
chronic disease, MJA. 2003; 179: 242-246.
[5] E.A. Cummings, J. Hauser, H.L. Cameron-Tucker, P. Fitzpatrick, M. Jessup, H. Walters, D.W. Reid & P.
Turner. Enhancing Self-Efficacy for Self-Management in People with Cystic Fibrosis. Studies in Health
Technology and Informatics. 2011; 169(1): 33-37.
Human Computer Interaction
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-145
Information Accountability and Usability:

Are There Any Connections?
Tony SAHAMAa,1, Andre KUSHNIRUKb and Shigeki KUWATAc
a.
Information Security Discipline, School of Electrical Engineering & Computer
Science, Faculty of Science and Technology, Queensland University of Technology
(QUT), Brisbane, QLD, Australia, b.School of Health Information Science, Faculty of
Human and Social Development, University of Victoria, Victoria, BC, Canada,
c.
Department of Medical Informatics, National Cerebral and Cardiovascular Centre
Hospital (NCVC), Suita, Osaka, Japan
Abstract. Availability of health information is rapidly increasing and the

expansion and proliferation of health information is inevitable. The Electronic
Healthcare Record, Electronic Medical Record and Personal Health Record are at
the core of this trend and are required for appropriate and practicable exchange and
sharing of health information. However, it is becoming increasingly recognized
that it is essential to preserve patient privacy and information security when
utilising sensitive information for clinical, management and administrative
processes. Furthermore, the usability of emerging healthcare applications is also
becoming a growing concern. This paper proposes a novel approach for integrating
consideration of information accountability with a perspective from usability
engineering that can be applied when developing healthcare information
technology applications. A social networking user case in the healthcare
information exchange will be presented in the context of our approach.
Keywords. Information Accountability, Usability Testing, Healthcare Information

Technology (HIT), Health Informatics, Information Governance, Electronic
Healthcare Record (EHR), Electronic Medical Record (EMR)
Introduction
The creation of health information silos, and the generation of thousands of Electronic
Healthcare Record (EHR), Electronic Medical Record (EMR) and Personal Health
Record (PHR) systems around the globe are seemingly unstoppable. Furthermore, the
explosion of information sharing using public social networking and related links is
accelerating in an ever-increasing rate. Recent statistics show that around 3.7 billion
people around the globe are using the Internet [1] (this is >50% of world population),
while 40 billion photos are deposited [2] in Facebook. An important aspect of capturing
the positive impact of such developments is to integrate those social activities with
health information exchange (HIE), a powerful tool [3] yet to be realised fully. Health
information sharing in a general context is not new, however, in the digital world, this
increasing social interaction demands further scrutiny for several reasons. Health
1
Corresponding Author: Dr. Tony Sahama, QUT. Email: [email protected]
146 T. Sahama et al. / Information Accountability and Usability: Are There Any Connections?
information sharing in clinical settings [4] is timely, and supporting informed

qualitative clinical decision-making processes in digital medicine is needed.
Research evidence shows that Healthcare Information Technology (HIT) can make
a positive impact on reducing healthcare costs, for example real-time remote
diagnostics and health monitoring (e.g., telemedicine) by using smart mobile devices
and Internet technologies. Although HIT is becoming part of the critical infrastructure
for improved digital health scenarios, the evolving process of adoption and use is slow
among practitioners and patients, with both groups indicating concerns about
information security and privacy. One of the key inhibitors is patient liability when
using HIT for medical and health related decision-making processes, whilst another
main barrier is physician dissatisfaction with these systems. This dissatisfaction is often
related to time commitments for unfamiliar and sometimes unreliable (e.g., the
technology not being user friendly) computer applications and the time taken to learn
such new processes, protocols and how to use new information systems. Furthermore,
adoption of HIT leads to demands for social value, like user acceptability where there
are compelling reasons why the system must be used. However, a number of issues
continue to arise, including cost factors (e.g., time commitments, potential loss of
productivity and efficiencies of using the system) and poor levels of usability (e.g., user
friendliness), which have been cited as being some of the biggest obstacles [5]. These
factors often do not receive enough consideration when HIT applications are developed
and integrated with healthcare systems that aim to improve the healthcare processes in
long run. This is predicted to change as physicians and medical practitioners become
more comfortable with computers and web-based healthcare offerings and in
conjunction with telemedicine vendors devising more convenient technology and
helpful applications. However, in order to support these endeavours, it is important and
timely to evaluate factors such as HIT usability and information accountability when
developing healthcare applications, which will form the focus of this paper.
1. Information Accountability
Information accountability focuses on the concept of monitoring use of personal

information and holding the users of that information accountable if that information is
misused [6][7]. Information accountability is not a new concept but rather this
phenomenon had been used for other systems like accounting and financial systems for
a longer time. Information accountability in the digital healthcare paradigm is bedrock
for effective clinical governance and a catalyst for healthcare information technology
[7]. In general, accountability for processes can form a good building block for further
work, such as measuring actual outcomes of care [6, 7].
2. Usability
Broadly speaking, usability can be defined as a measure of ease of use and usefulness
of an information system in terms of its: (1) effectiveness, (2) efficiency, (3)
enjoyability, (4) learnability, and (5) safety [8]. These attributes or dimensions of
usability are useful in focusing attention on key aspects of the use of systems in the
design and evaluation of health information systems. For example, to lead to uptake
and adoption by end users, health information systems need to both effective and
T. Sahama et al. / Information Accountability and Usability: Are There Any Connections? 147
efficient. Learnability is also essential, as is safety in order to ensure that information

systems in healthcare do not inadvertently cause users to make medical errors.
Furthermore, systems must be satisfying for end users. Poor usability has been cited as
being one of the main reasons for lack of end user adoption of systems and health
professional dissatisfaction with HIT [9]. To ensure that systems are usable, a variety
of methods from the field of usability engineering have been applied to HIT and
continual evaluation of the usability of systems under development has been
recommended through iterative cycles of design and testing [9].
3. What is the connection between Information Accountability and Usability

Testing?
Both information accountability and usability have a number of similarities when

considered in comparison. Firstly, both can be considered as being key requirements
for development of effective HIT. Along these lines, both can be considered as being
“non-functional requirements”, i.e. critical requirements that need to be considered for
system success that are neither functional requirements nor technical requirements [10].
In order to lead to improved chances of HIT success and adoption, information
accountability and the key aspects of usability (i.e. system effectiveness, efficiency,
learnability and safety) need to be considered. Furthermore, both high levels of
information accountability and good usability can be considered as being “soft goals”
to be achieved through successive planning and iterative analysis [10]. Both are
ultimately required to lead to uptake and adoption of HIT by both patients and health
professionals. In this regard, methods for usability testing that focus on user experience
can be expanded to include consideration of user perceptions, comments and thoughts
about the accountability of information contained in health information systems they
are interacting with during usability testing.
4. Healthcare Information Technology, Information Accountability and Usability

Testing
In considering the broad definition of usability given above, it is argued in this paper
that adding information accountability to the list of attributes/dimensions of the concept
of “usability” could practically lead to improved systems and consequently more
effective user adoption of HIT. From a practical point of view, studies can be designed
to assess the following from the perspective of end users interacting with HIT: (1)
effectiveness, (2) efficiency, (3) enjoyability, (4) learnability, (5) safety, and (6)
information accountability (this non-functional requirement is a critical consideration
for HIT applications that adequately protect information privacy and security). This
will be essential as systems that have been deemed to have met requirements of the first
5 attributes above may still not be accepted by end users without explicit consideration
of making information accountable to all classes of end users. Nowhere is this more
important than in designing systems intended for end users who are patients or citizens,
as described in the next section.
5. Use cases and Scenarios
Our preliminary experiences to date in integrating the concept of information

accountability with usability have involved planning for evaluation of PHRs and social
media. Issues related to both usability (as it has been described above [8]) merged with
a perspective from information accountability are being explored [7]. In terms of
modeling system requirements for PHRs, we are working on including information
accountability as an essential non-functional requirement (along with the other
“classic” usability dimensions discussed above).
The approach we are working on is more towards a patient, public partnership
(depicted in the use case scenario in Figure 1) by identifying partner participations as
social interactions. Figure 1 depicts a general use case for the public with several social
networking accounts. The approach is to develop protocols to integrate them all within
one profile (e.g., MPM: Multiple Profile Manager) when using health information
sharing and exchange (e.g, HIE). This scenario is already active without public
knowledge of information sharing in the digital world (e.g., sharing through
interconnected EHRs, EMRs, and PHRs). While, this social interaction is value added
to health information exchange (HIE), without considering appropriate information
accountability (e.g., authentication, authorization and synchronization) and without
applying usability testing to obtain user input and perceptions about accountability, the
acceptance of such HIT applications (e.g., the Multiple Profile Manager, -MPM [11-
13]) and the sustainability of the approach will be jeopardized.
Along these lines, we are also including assessment of the end user’s perceptions
about needs for information accountability within the design of upcoming usability
testing of several PHRs being implemented in Canada and in Australia.
Figure 1: User Case scenario in a multiple social networking set-ups [12][13]

T. Sahama et al. / Information Accountability and Usability: Are There Any Connections? 149
6. Discussion
Healthcare is an information intensive, complex, large-scale, adaptive, distributed and

evolving system [14]. With advancement of technologies in particular information and
communication technologies (ICT), digitalization of healthcare processes and protocols
are developing in an alarming rate. While this advancement represents positive growth
for the digital economy, information privacy and security are still open ended questions
and there is a long way to go to for assessing end users needs. A simple approach
would be to empower the patient with usability and information accountability
protocols where patient become a partner in the healthcare decision making processes
as well as HIE processes. This attempt might be debatable for some clinical settings
however establishment of practicable usability testing and implementation of active
and accurate information accountability protocols would help the sustainability of HIT
and thereby lead to a reduction in mounting healthcare costs.
We have proposed a novel approach to consideration of HIT and HIE where there is
integration of usability analysis with analysis of information accountability needs, with
both being considered as essential non-functional requirements for patient centric HIT
applications that must not be ignored. Further studies assessing end users’ perception
and the need for information accountability are being planned within the design of
upcoming usability testing of several PHRs.
References
[1] Internet World Stat: http://www.internetworldstats.com/stats.htm Retrieved on 30 July 2012

[2] The Economist 2010: "Data, data everywhere". The Economist. 25 February 2010.
http://www.economist.com/node/15557443.
[3] Shapiro, J.S., Kannry, J., Kushniruk, A.W. and Kuperman, G. (2007). Emergency Physicians’
Perceptions of Health Information Exchange, Journal of American Medical Informatics, 14:700-705
[4] Adler-Milstein, J. and Jha, A.K. (2012). Sharing Clinical Data Electronically: A Critical Challenge for
Fixing the Health Care System, JAMA, 307(16): 1695-1696
[5] Alagoz, F., Valdez, A.C., Wilkowska, W., Ziefle, M., Dorner, S. and Holzinger, A. (2010). From Cloud
Computing to Mobile Internet, From User Focus to Culture, and Hedonism: The Crucible of Mobile
Health Care and Wellness Application, IEEE Pervasive Computing and Applications (ICPCA), 2010
5th International Conference onDate of Conference: 1-3 Dec. 2010, 38—45
[6] Allen, P. (2000). Clinical governance in primary care- Accountability for clinical governance: developing
collective responsibility for quality in primary care, BMJ, 321:608—611
[7] Gajanayake, Randike, Iannella, Renato, & Sahama, Tony R. (2011) Sharing with care: an information
accountability perspective. IEEE Internet Computing, 15(4), pp. 31-38
[8] Preece J, Rogers Y, Sharp H, Benyon D, Holland S, Carey T. Human–computer interaction. New York:
Addison-Wesley Publishing Company; 1994.
[9] Kushniruk, A. W., Patel, V.L. (2004). Cognitive and usability engineering approaches to the evaluation
of clinical information systems. Journal of Biomedical Informatics, 37(1), 56-76.
[10] Cysneiros, L., Kushniruk, A.W. (2003). Bringing usability to the early stages of software development.
Proceedings of the 11th IEEE International Requirements Engineering Conference 2003, 359-361.
[11] Liang, J., Sahama, T. and Iannella, R. (2012). Online Multiple Profile Manager for eHealth Information
sharing, “Connect, Communicate & Collaborate” – The IEEE International Conference on
Communication, June 10—15 Ottawa, Canada.
[12] Liang, J., Sahama, T. and Iannella, R. (2012). Impact of Social Networking Applications for Health
Information Management for Patients and Physicians. ‘Quality of Life Through Quality of Information’
Proceedings of European Federation of Medical Informatics (Eds), J. Mantas et al., ISO Press. Page
803—807.
[13] Liang, J., Iannella, R. and Sahama, T. (2011). Synchronised Integrated Online e-Health Profile. Studies
in Health Technology and Informatics, 168; 96—103.
[14] Leap, L.L. and Berwick, D.M. (2005). “Five Years After to Err is Human: What Have We Learned?”,
JAMA, 293(19):2384—90.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-151
A Review Of Healthcare Information

System Usability & Safety
Simon MINSHALL a,1
a
School of Health Information Science, University of Victoria, Victoria, BC, Canada
Abstract. Healthcare information systems have been designed to increase the

efficiency and safety of healthcare processes. Systems such as electronic health
records and pervasive computing devices have been shown to improve the safety
of healthcare. However, increasing research has indicated that the design of such
systems, in particular the user interface, may be related to increased incidence of
other types of error. In this review, the relationship between human factors and
usability will be considered in the context of designing safe and effective
healthcare applications, with a focus on hand-held computing devices. Medline
was searched for the specific terms listed below and restricted to the date ranges
2006-01-01 through to 2011-03-03: (error AND technology AND human factors);
(error AND (CPOE OR (Computerized AND provider AND order AND entry)));
(Technology AND Induced AND Error). The returned list of papers was screened
by examining titles and abstracts to select candidate papers for further review.
The initial search yield was 239 papers. On reviewing the title and abstract, 186
were rejected and 51 papers remained for analysis. New technology, such as
CPOE, offers improvements over traditional paper tools and it is shown to have a
positive effect on patient safety. New technology also creates the opportunity for
new errors to occur and lead to the coining of the term “technology-induced error”.
The magnitude of the usability-testing needs is larger than it may seem.
Keywords. Usability, technology-induced error, simulations, health information

systems, patient safety, CPOE, EHR, health informatics, biomedical informatics
1. Introduction
Information systems have been designed to increase the efficiency and safety of
healthcare processes. The predominant paper technology is inadequate for meeting the
needs of modern healthcare [1]. Mobile devices, such as smartphones and tablets, are a
pervasive technology. They can facilitate messaging, time management, education and
telephony. They also have the ability to easily retrieve and run health-related software
‘apps’ from online repositories. Consequently, mobile devices are now capable of
many assistive roles, particularly the ability to display and manipulate medical
information in clinical settings. In Canada, health information systems (HIS) adoption
rates lag behind those of other countries [2]. Considering the low HIS adoption rates
and the rapid mobile device adoption rates, it is not unreasonable to suggest that
adoption of these pervasive devices will outpace the adoption of HIS systems – if they
have not already done so. The issue for safety is the evidence that new technologies,
despite their clear advantages over paper technology, can be the root cause for new
1
The author may be contacted at [email protected]
152 S. Minshall / A Review of Healthcare Information System Usability & Safety
kinds of errors. Research in this area has indicated that the design of new technology,
in particular the user interface (UI), may be related to increased incidence of new types
of error [3], [4]. In this paper, the relationship between human factors and usability
will be considered in the context of designing safe and effective healthcare applications,
with a focus on computing devices.
2. Methods
Medline was searched for the specific terms listed below and restricted to the date
ranges 2006-01-01 through to 2011-03-03:
1. (error AND technology AND human factors)
2. (error AND (CPOE OR
(Computerized AND provider AND order AND entry)))
3. (Technology AND Induced AND Error)
3. Findings
3.1. Search results
Table 1. The Medline search resulted in 239 papers. After reviewing the title and abstract, 186
papers were rejected and 51 papers remained for review.
Inclusion Criteria Title Exclusion n Abstract Exclusion n
User Errors Not data-related 37 Wrong device 20
Data entry Other tools 27 Not medical app 17
Data access Tablet is medication 16 Not clinician 9
Computing Surgical devices 13 Review off topic 17
HCI Not medical 5 Inaccessible 4
User interfaces Education 3 Duplicates 4
Usability Workflow 3 Education 13
In English
Remaining 239 135 51
3.2. Categorized selected papers
Table 2. The remaining papers were classified into the following sets. Countries abbreviated as per ISO
3166 standard
Paper Type n Research n Technology n Country n
Critical Reviews 1 Case Studies 4 CPOE 27 US 21
Lit. Reviews 10 Interventions 10 Devices 2 CA 10
Methods 10 Observations 3 HIS 18 NL 9
Research 30 Prospective 1 Paper 3 AU 4
Retrospective 4 Workflow 1 DK 2
Simulations 6 IL, FR 2
Surveys 2 IR, IT 2
LB 1
Column Totals 51 30 51 51
S. Minshall / A Review of Healthcare Information System Usability & Safety 153
3.3. CPOE themed papers
CPOE themed papers in this review demonstrated that this specific IT implementation
caused the following: a positive effect on safety, particularly when coupled with a
decision support system (DSS) [5], increased compliance and improved legibility [6].
A review by Reckmann reported that between 1998 and 2007, the evidence that
CPOE interventions reduced errors is limited and not compelling [7]. Another review
by Weant [8] reported that errors increased during a paper to CPOE transition. Further,
inappropriate alerting, alert fatigue, inappropriate dose defaults, workflow disruption
and multitasking effects experienced and measured after CPOE implementation, all
contributed to a reduction of patient safety. Errors caused by features unique to the IT
implementation can be considered technologically induced errors. Many of the papers
call for further investigation and ask new research questions: What cognitive processes
contribute to alert overriding? Can heuristics be used to detect prescription errors?
Can frequently overridden alerts be safely disabled?
3.4. Paper-based fieldwork on CPOE interventions
Paper itself is a common theme in this search, primarily because CPOE

implementations or interventions are done to replace the traditional paper-based
workflow. Paper-themed research using qualitative semi-structured interviews [9]
identified 11 distinct reasons for using paper, they are: efficiency, ease of use, memory,
preference, awareness, task specificity, task complexity, data organization, time
management, trust and data security. The study discusses these findings in detail and
concludes that in some cases paper was a valuable tool, i.e., as a cognitive short-term
memory aid, and in some cases an impediment to safety, i.e., when paper was used to
circumvent the intended use of the EHR, a potential path to error.
Miller, describing an interesting example of an in situ study, uncovers a basic truth
missing from many of the technology-implementation papers in this review. For five
days at the bedside of five terminally ill patients, researchers recorded clinical
conversations. The participants were the clinicians caring for these. The researchers
were investigating the merits of paper as a clinical technology prior to a CPOE
installation to test two hypotheses A) that paper-based tools are used in different ways
by doctors, nurses and other professionals, and B) that different paper artefacts (forms)
support different decision-making processes. The statistically significant results of the
investigation supported the hypotheses and led to two design principles 1) that effective
information systems must support interdisciplinary interactions, and 2) that they must
support decision processes, information flows and practice in addition to data entry.
Miller’s paper argues that the IT solution should not be used to replace paper-use as a
record-keeping system; rather, and more importantly, that they replace the functions of
the paper-based tools provide to clinicians while they go about their daily activities, i.e.,
paper forms and worksheets are seen as tools of collaboration not merely tools for data.
Therefore, IT systems aimed at replacing paper-based systems need to consider the
users’ needs for collaborative enablers and not just data storage and retrieval [10]. This
notion was also discussed in [6] where the authors concluded that the workflow
impediments introduced by the paper-replacing CPOE system caused error-inducing
conditions which led to safety concerns and as a result, new systems must address the
communication needs that integrate the work of nurses and physicians. Hysong et al,
contributed to the body of methods by proposing a protocol consisting of qualitative
methods and a human factors approach to evaluate systems and processes that lead to
diagnostic and medication error. The resulting protocol describes methods adapted
from human factors and psychology to analyze the ways in which providers currently
use CPOE to communicate and identify barriers to effective electronic communication
[11]. Fieldwork by the foregoing researchers has provided valuable insight into the
nature of teamwork paper-use and the introduction of new technology.
3.5. Simulation methods and effect of realism
Many more studies advocated and used simulation methods in their investigations.
Lilholt, in a paper responding to published concerns about applications being
unsuitable for clinical use, devised and tested a usability evaluation to investigate
problems in an EHR system by combining methods from laboratory tests and field
studies [12] and applying them in a realistic ward-simulator using an actor to play the
part of a patient. Lilholt concluded that some of the usability issues encountered only
manifested due to the realistic setting, suggesting that if the scenario were not realistic,
some usability issues would not have been revealed. Papers reporting high quality
simulations seemed to have fewer stated limitations seemed more effective and perhaps
left less room for doubt or criticism due to the effects of a simulated environment.
Although simulations may seem limited by their implicit lack of realism, their safety-
related use in other industries, notably aviation and nuclear power control, is
widespread.
3.6. Technology-induced error and the role of simulation
In 2009, six papers were published describing and advocating simulation-based

research methods on new technology and human error. Five of these, from the
University of Victoria Department of Health Informatics, deal specifically with
research methods to: test, analyse root causes, develop heuristics and prevent [13]
technology-induced error in healthcare. Carvalho et al, on a similar theme, found that
simulation methods may be used with success to prospectively evaluate clinical
hardware and software and, using heuristic evaluations, predict the likely error-rates of
systems before they are procured and deployed in large numbers [14]. The simulation
methods differ from Lilholt’s study in that much simpler methods are advocated and no
highly realistic environments are necessary. For example, a researcher could gather
data by video recording and screen-recordings of a clinician sitting at an ordinary desk.
The coding and analysis may be done elsewhere, even in a different country separating
the low-skilled data collection from the expert analysis. Another example given is a
cognitive walk-through where software can be assessed by an expert using a set of
research-derived heuristics and evidence-based metrics [15]. This body of research
recommends that more evidence-based usability studies be done on the ever-growing
list of available electronic medical devices, hardware and software. All technology-
induced error studies reviewed here also advocate the utility of using mixed-methods
primarily to investigate the human factor of the new errors manifested in the use of new
health IT systems.
S. Minshall / A Review of Healthcare Information System Usability & Safety 155
3.7. Usability testing to predict unsafe technologies
Usability tests are performed to analyze a system that consists of three parts: hardware,
software and the users. Papers from studies of human factors – beyond the scope of this
review, but still relevant – show that older and younger adults use the same tools in
different ways, suggesting that life experience and risk-taking play a part in the
cognitive processes of decision-making [16]. People vary in ability, skill, familiarity,
competence, confidence and style. According to the reviewed papers, many clinical
tool devices and systems do not offer any adaptation to users’ variability. User
interfaces are criticized for inflexible login modes, poor medication models, rigid
controls, inflexible processes, ordering formats, rigid design and rigid workflows [17].
Examining a tool for usability can reveal these issues and potentially predict the
likelihood for and magnitude of technology-induced error.
The need for more granular usability studies is evident in the reviewed papers.
Buchel found “that devices which are already on the market differ drastically in their
usability [… and] that unsatisfying usability bears a high risk in health care.” [18].
That one device proves to be usable, does not mean that another similar device or its
software, is automatically similarly usable. One cannot state with confidence that
because a particular EHR app is deemed ‘usable’ on one tablet computer with a touch
screen as its main input device, will be just as usable on a similar tablet that uses a
stylus. Ideally, every device-software combination should be vetted for its error-
inducing potential as part of the procurement process. New technology arrives
regularly in the form of mobile hardware and cross-platform software. Improvements
and major changes to both are also frequent and can come with radical user interface
changes that fundamentally alters the way the device-software is used – rendering any
previous usability analysis, published or unpublished, as superfluous.
The variety of mobile devices is increasing. The need for either more usability
assessments or tighter restrictions on their use in the clinical setting will be needed to
measure and control patient safety issues.
4. Conclusion
New technology, such as CPOE, offers improvements over traditional paper tools and
it is shown to have a positive effect on patient safety. New technology also created the
opportunity for new errors to occur and led to the coining of the term technology-
induced error. The human factors of device and technology use, in particular
transitioning from the team-based use of traditional tools discussed in this review, show
that this area is under-investigated.
Fieldwork is essential to assess the validity of simulation studies and to gain
insight on the working practices of teams. Likewise, realistic simulations are useful to
demonstrate the minutiae of device use when an actor-patient is present. The bulk of
actual usability work is better done using simulation methods. The bulk of actual
simulations for usability research can be done using low-cost methods. This is
important because of the great volume of testing needed due to the ubiquity of mobile
devices and access to clinical and clinical-style software available to owners of these
devices. Ideally, each device-software pair should be assessed for its error-inducing
properties. To cope with the increasingly rapid growth of this area of technology more
usability testers and facilities will be required. Further research-methods studies and
evidence-based metrics are needed to generate more efficient and simpler usability
protocols. The scope of this usability-testing problem is larger than it may seem given
the current proliferation and global distribution of new, sophisticated mobile devices
and un-vetted access to health-related software.
References
[1] Shortliffe E, Blois M. The computer meets medicine and biology: emergence of a discipline.
Biomedical Informatics, 2006; 1: 3–45.
[2] CHI Annual Report 2009-2010. Canada Health Infoway, May 2010: 1–48.
[3] Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimme SE, Strom BL. Role of computerized
physician order entry systems in facilitating medication errors. JAMA, 2005; 293(10): 1197–1203.
[4] Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability:
the relationship between usability problems and prescription errors when using a handheld application,
Int J Med Inform, 2005;74(7): 519–526.
[5] Kuperman GL, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, Classen DC, Bates DW.
Medication-related clinical decision support in computerized provider order entry systems: a review, J
Am Med Inform Assoc, 2007;14(1):29–40.
[6] Pirnejad H, Niazkhani Z, van der Sijs H, Berg M, Bal R. Evaluation of the impact of a CPOE system on
nurse-physician communication--a mixed method study, Methods Inf Med, 2009;48(4):350–360.
[7] Reckmann MH, Westbrook JI, Koh Y, Lo C, Day RO. Does computerized provider order entry reduce
prescribing errors for hospital inpatients? A systematic review., J Am Med Inform Assoc, 2009;16(5):
613–623.
[8] Weant KA, Cook AM, Armitstead JA. Medication-error reporting and pharmacy resident experience
during implementation of computerized prescriber order entry, American Journal of Health-System
Pharmacy, 2007;64(5):526–530.
[9] Saleem JJ, Russ AL, Justice CF, Hagg H, Ebright PR, Woodbridge PA, Doebbeling BN. Exploring the
persistence of paper with the electronic health record, Int J Med Inform, 2009;78(9):618–628.
[10] Miller A, Pilcher D, Mercaldo N, Leong T, Scheinkestel C, Schildcrout J. What can paper-based
clinical information systems tell us about the design of computerized clinical information systems (CIS)
in the ICU?, Aust Crit Care, 2010;23(3):130–140.
[11] Hysong SJ, Sawhney MK, Wilson L, Sittig DF, Esquivel A, Watford M, Davis T, Espadas D, Singh H.
Improving outpatient safety through effective electronic communication: a study protocol., Implement
Sci, 2009;4:62.
[12] Lilholt LH, Pedersen SS, Madsen I, Nielsen PH, Boye N, Andersen SK, Nøhr C. Development of
methods for usability evaluations of EHR systems, Stud Health Technol Inform, 2006;124:341–346.
[13] Kushniruk AW, Borycki EM, Anderson JG, Anderson MM. Preventing technology-induced errors in
healthcare: the role of simulation, Stud Health Technol Inform, 2009;143:273–276.
[14] Carvalho CJ, Borycki EM, Kushniruk AW. Using heuristic evaluations to assess the safety of health
information systems, Stud Health Technol Inform, 2009;143:297–301.
[15] Koppel R, Leonard CE, Localio AR, Cohen A, Auten R, Strom BL. Identifying and quantifying
medication errors: evaluation of rapidly discontinued medication orders submitted to a computerized
physician order entry system, J Am Med Inform Assoc, 2008;15(4):461–465.
[16] Ezer N, Fisk AD, Rogers WA. Age-related differences in reliance behavior attributable to costs within a
human-decision aid system, Human Factors: The Journal of the Human Factors and Ergonomics
Society, 2008;50(6):853–863.
[17] Borycki EM, Kushniruk AW, Keay E, Kuo A. A framework for diagnosing and identifying where
technology-induced errors come from. Stud Health Technol Inform, 2009;148:181.
[18] Büchel D, Baumann T, Matern U. Usability of radio-frequency devices in surgery, HCI and Usability
for Medicine and Health Care, 2007: 97–104.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-157
Commercial versus In-Situ Usability

Testing of Healthcare Information Systems:
Towards “Public” Usability Testing in
Healthcare Organizations
Andre W. KUSHNIRUKa1, Elizabeth M. BORYCKIa, Joseph KANNRYb
a
School of Health Information Science, University of Victoria, Victoria, British
Columbia, Canada, b Mount Sinai Medical Center, New York, New York
Abstract. The need for improved usability in healthcare IT has been widely
recognized. In addition, methods from usability engineering, including usability
testing and usability inspection have received greater attention. Many vendors of
healthcare software are now employing usability testing methods in the design and
development of their products. However, despite this, the usability of healthcare IT
is still considered to be problematic and many healthcare organizations that have
purchased systems that have been tested at vendor testing sites are still reporting a
range of usability and safety issues. In this paper we explore the distinction
between commercial usability testing (conducted at centralized vendor usability
laboratories and limited beta test sites) and usability testing that is carried out
locally within healthcare organizations that have purchased vendor systems and
products (i.e. public “in-situ” usability testing). In this paper it will be argued that
both types of testing (i.e. commercial vendor-based testing) and in-situ testing are
needed to ensure system usability and safety.
Keywords: human computer interaction, usability, usability testing, in-situ
Introduction
The usability of healthcare information systems has been recognized as being critical to
the successful deployment, adoption and appropriate use of these technologies. Over
the past several decades a wide range of usability engineering projects, efforts and
laboratories have emerged to address these issues. However, despite these efforts, the
usability of healthcare IT (HIT) is still reported as being problematic, particularly when
commercial vendor systems, such as electronic health record (EHR) and electronic
medical record (EMR) systems are deployed in offices, clinics and hospital settings.
The reporting of a considerable number of usability problems and issues with these
systems is not just localized to Canada but is an international phenomena [1]. Many
vendors and developers of healthcare software are now applying usability engineering
methods, such as usability testing. However, despite this, the usability of HIT is still
considered to be problematic and many healthcare organizations that have purchased
systems that have been tested at vendor testing sites are continuing to report a range of
1
Corresponding author: Andre W. Kushniruk: [email protected]
158 A.W. Kushniruk et al. / Commercial Versus In-Situ Usability Testing
usability and safety issues [2]. In this paper we explore the distinction between
commercial usability testing (conducted at centralized vendor usability laboratories and
limited beta test sites) and usability testing that is carried out locally within healthcare
organizations that have purchased vendor systems and products (i.e. public “in-situ”
usability testing). It will be argued that both types of testing (i.e. commercial testing)
and in-situ testing are needed to ensure system usability and safety. It will further be
argued that on-site in-situ usability testing can provide organizations who have
purchased vendor HIT systems with critical feedback for: (a) customization of those
systems to improve usability, (b) improved user training, and (c) evidence to present
back to vendors in cases where usability is deemed to be extremely poor, leading to
potentially “unsafe” installations of HIT. It is also argued that there is a need for closer
collaboration between vendor usability efforts and efforts to improve usability that are
spearheaded at local healthcare institutions and which purchase HIT from vendors.
1. Low-Cost Rapid Usability Engineering Conducted “In-situ” in Local

Healthcare Settings
In this part of the paper we will describe the low-cost rapid usability engineering
approach we have developed and refined that can be applied within local healthcare
organizations (e.g. medical offices, clinics and hospitals) for doing usability testing.
We will refer to this approach as “in-situ”, as it is conducted in the real-life settings in
which healthcare IT is deployed (e.g. electronic health record deployment within the
hospital setting the system will be implemented in). Figure 1 shows a continuum from
vendor-based testing (conducted at vendor usability labs or beta test sites) to “public”
in-situ usability analyses carried out at the actual institutions where the healthcare IT is
being deployed. We have found that in-situ usability testing can be carried out
effectively by IT staff or health informaticians at local healthcare organizations without
requiring extensive training in the methodology, as the approach has been streamlined
for deployment for a variety of settings and locations both nationally and
internationally.
A.W. Kushniruk et al. / Commercial Versus In-Situ Usability Testing 159
The applications evaluated range from the evaluation of a medication order entry
system (using bar-coding technology) prior to its deployment in a hospital in Japan [3]
to the study of an introduction of an EMR at major American medical center (involving
in-situ testing both before and after system going live) [4].
Applying our approach to in-situ usability testing, our work is typically carried out
in real clinical settings (e.g. a physician’s office, in a clinic or even in an operating
room after hours). The approach typically involves asking between 8-10 subjects (e.g.
nurses or physicians) to interact with a system under study (e.g. an EMR) to carry out
tasks (e.g. to enter medications from a medication list). In addition, subjects may also
be asked to “think aloud” while carrying out the task, which is audio recorded. Overt
physical activities of the subjects (e.g. checking information in the computer against
paper records on their desk) are recorded using one or more low-cost portable digital
cameras. The computer screens are also recorded as a digital movie, using free screen
recording software installed on the computer running the system being tested (we are
currently using Hypercam©). When running this software (by simply plugging a
microphone into the computer) the result will be a movie of the user-computer
interaction with the audio track of the subject’s verbalizations (i.e. “thinking aloud”),
which can be played back for immediate informal review and analysis. Thus data
collected consists of digital movie files showing what users are doing on the computer
which are linked to audio recordings and which can also be linked to the external video
view of physical activities if needed. Using this approach we have carried out studies of
HIT in environments ranging from physician offices to patients’ homes. An extension
of the approach involves continuing to record users interacting with the system once it
is released for real use (obtaining of course, ethical approval).
In order to substantiate the benefits of this approach, recent work by Baylis,
Kushniruk and Borycki involved conducting a formal cost-benefit analysis of
conducting in-situ usability testing on a disease management software system to be
released throughout a healthcare system at a provincial level [5]. From this work, it was
found that in testing only a small subset of the system’s functionality the total cost
(including one-time cost of buying a laptop for testing purposes) was approximately
$8,000 with the most conservative estimate of benefit from this analysis being in the
range of $50,000 (and with the benefits considerably greater if the cost associated with
potential medical error is considered).
2. Data Collection and Recommendations for System Improvement
The analysis of the data collected from local in-situ usability testing can vary from
informal playback of recordings of user interactions (e.g. playing back digital movies
of user interactions with an EMR) at IT meetings, to more formal coding and analysis
of identified user problems and even statistical analysis. Most of our work involves an
analyst viewing the data and developing qualitative categories of problems (e.g.
navigational problems, problems users have in interpreting information presented and
other types of usability problems) and then quantifying the occurrence of the problems
(e.g. tabulating the frequency of the most common usability problems). The result of
this analysis can then fed directly back to the local system implementation team and
prioritized for remedial action (e.g. either through local customization, or feedback to
local user training). In cases where usability problems are deemed to be critical to fix at
160 A.W. Kushniruk et al. / Commercial Versus In-Situ Usability Testing
the local level, but the nature of the software (or the nature of the contract with the
vendor) precludes fixing at the local level, the data collected (and recommendations
made) need to be reported back to the commercial vendor for making more
fundamental changes to the software. Some of our work has involved applying low-
cost portable in-situ testing to identifying potential technology-induced errors that may
be caused by a system (e.g. inappropriate medication defaults in an order entry system),
or by poor user interface design [6]. It should be noted that by testing systems in their
real environment of use (as opposed to testing them in a vendor usability laboratory, or
only conducting usability testing at some beta test site), then problems and issues that
would only arise in the local setting (i.e. the hospital which purchased the system) can
be detected, analyzed and prioritized prior to system release. Along these lines, many
of the issues noted in the literature with the deployment of HIT are related to local
conditions, contexts and work environments, making this approach to usability testing
important to detect problems that would otherwise go unnoticed until a vendor system
is actually running live within a healthcare organization. This local testing is essential
for ensuring not just the usability but also for ensuring the safety of HIT deployed in
real-world contexts.
3. Experiences to Date
We have applied the methodology described above at varied locations internationally.

Our first work applying the approach involved testing an electronic record system
being deployed at a major American medical center in New York. From this work it
was calculated that the in-situ testing resulted in a ten-fold decrease in usability
problems (with a cost for conducting the analysis of under $10,000) [7]. A range of
usability problems were identified and prioritized for fixing (based on frequency of
occurrence and cost of fixing) including user interface consistency problems, problems
in matching of terms used in the system to user selected terms and problems in
representing temporal data. This study was conducted within the hospital setting over
several days. More recently, we have employed our refinement of the method to
examine the impact of clinical best practice guidelines on physician workflow using an
electronic medical record system at a second major American hospital center in New
York [4]. This involved both in-situ testing of users interacting with the guidelines both
(prior to widespread release) as well as naturalistic testing of the system after
deployment for use with real patients (using the same unobtrusive recording technology
and set up in both cases). This analysis involved three phases of work. In phase one
usability testing was conducted with 8 physicians who were asked to interact with the
guidelines embedded in a major commercial EMR. After fixing the surface level
usability problems identified from the first phase, as second phase was conducted
where a new group of physicians interacted with the guidelines while interviewing a
“digital” simulated patient. This work led to fixing of problems that occurred in the
triggering of guidelines under realistic conditions. Finally, we are currently examining
naturalistic data we have continued to collect on real use of the guidelines within the
organization. It should be noted that all three phases were conducted in the real hospital
setting, using the commercial vendor based system that was deployed at the institution.
The analysis detected usability problems that were corrected locally (prior to
widespread system release) that were not anticipated by the vendor of the EMR, with a
A.W. Kushniruk et al. / Commercial Versus In-Situ Usability Testing 161
resultant high level of user uptake of the clinical guidelines (as determined from a
subsequent full scale trial that was conducted) [4].
4. Discussion
From our work we have found that many usability problems and errors that are
encountered in real clinical contexts are not detected or corrected by usability testing
conducted at centralized vendor usability laboratories prior to system release (or at only
selected beta test site healthcare organizations). We do feel that vendor-based testing
(including vendor usability laboratory testing, as well as regional centralized
conformance testing of vendor products) is necessary. However, it has been argued in
this paper that centralized private usability testing (such as that conducted by major
EHR vendors) needs to be complemented by “public” in-situ testing to ensure system
usability and safety. Current issues and challenges include getting permission to
conduct studies in a real environment and also obtaining rooms and locations after
hours for conducting in-situ usability testing. Furthermore, there is a need for improved
communication between healthcare institutions and vendors in order to allow for the
most serious usability and safety problems identified at the local level to be fed back
for software improvement at the vendor level. It is argued that if we are to ensure that
the systems we implement are both usable and safe, then these additional types of
studies (conducted in-situ at local healthcare organizations) are necessary and that they
can be carried out in a cost-effective and beneficial way to improve the safety and
efficiency of healthcare IT.
References
[1] Viitanen J, Hypponen H, Laaveri T, Vanska J, Reponen J, Winblad. National questionnaire study on
clinical ICT system. Int J Med Inform, 2011;80:708-725.
[2] Beuscart-Zephir M, Anceaux F, Meaux J, Degroisse M, Degoulet P. Impact of CPOE on doctor-nurse
cooperation for the medication ordering and administration process. Int J Med Inform, 2005;74(7):629-641.
[3] Borycki, E., Kushniruk, A., Kuwata, S., Kannry, J. Use of simulation in the study of clinician workflow.
AMIA Annual Symposium Proceedings, 2006, 61-65.
[4] Li, A., Kannry, J., Kushniruk, A., Chrimes, D., McGinn, T., Edonyabo, D., Mann, D. Integrating usability
testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical
decision support. In press, Int J Med Inform, in press.
[5] Baylis T, Kushniruk A, Borycki E. Low-cost rapid usability testing for health information systems: Is it
worth the effort? Proceedings of MIE 2012, 363-367
[6] A.W. Kushniruk, M. Triola, E. Borycki, B. Stein, and J. Kannry, Technology induced error and usability.
Int J Med Inform, 2005, 74, 519-526.
[7] A.W. Kushniruk, V.L. Patel, J.J. Cimino, and R. Barrows, Cognitive evaluation of the user interface and
vocabulary of an outpatient information system. Proceedings of the 1996 Annual AMIA Conference, 1996,
22-26.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-162
Integrating Human Factors in an

International Research Project: Lessons
Learned from the PSIP Project
Romaric MARCILLYa,1, Marie-Catherine BEUSCART-ZEPHIRa and Régis
BEUSCARTa
a
INSERM CIC-IT, Lille ; Univ Lille Nord de France ; CHU Lille ; UDSL EA 2694 ; F-
59000 Lille, France
Abstract. This paper presents how a Human Factors (HF) engineering approach
has been applied to a European Project aiming at preventing Adverse Drug Events.
Methods used by HF experts and their main contributions are depicted. Main
lessons learned from HF involvement in this project are proposed in terms of
methods for ADE prevention interventions and project management.
Keywords: user-centered design, human engineering, adverse drug event, CDSS
Introduction
Human Factors (HF) activities are increasingly applied during the design of Health
Information Technology devices/software. Those activities consist in applying HF
expertise and methods all along the project and in involving representative end-users at
every step. Applying those methods supports a better consideration of HF issues in the
design, development and evaluation of devices. Ultimately it has a positive impact on
the acceptance of the product [1] and on the effectiveness of the product [2]. The HF
strategy is even recommended to comply with the European Medical Device Directive
[3] specifically to prevent the risk of use errors related to ergonomics features [4].
One of the most common HF strategies is the User-Centered Design process
(UCD) [5]. As described in Figure 1, this process includes four iterative HF activities
ranging from the analysis of the context of use and the specification of the users’
requirements to the design itself, the iterative evaluation of the solutions to ensure that
it meets users’ requirements and a long-term post-implementation monitoring. Besides,
to improve the performance of the user-centered part of the system development (e.g.
improving UCD process efficiency), a “process for usability” application is also
proposed [6]. This process is composed of 7 activities (Figure 2) starting with the
overall consideration of UCD process in the lifecycle (#1), then ensuring the
management of the UCD all along the process (#2), achieving all UCD tasks (#3 to #6)
and finally supporting the implementation of the product (#7).
1
Corresponding Author: [email protected]. Institut Hippocrate – CHRU de Lille, 2
avenue Oscar Lambret – 59037 Lille Cedex
R. Marcilly et al. / Integrating Human Factors in an International Research Project 163
Figure 1. The User-Centered Design process adapted from the ISO 9241-210 [5].
The paper at hand presents how the HF strategy, inspired by the HCL, has been
planned and conducted during a European Union project. Lessons learned from this
experience are presented.
1. Background: HF engineering in the PSIP project
The European Union project entitled "PSIP - Patient Safety through Intelligent
Procedures in Medication" aimed to (a) innovatively produce new knowledge on
Adverse Drug Events (ADE) (by data/semantic mining methods) and (b) investigate
different possibilities for reducing ADE (e.g. developing knowledge to be integrated
into Computerized clinical Decision Support System (CDSS) and solutions supporting
patients’ compliance) [7]. The entire project was considered as a patient-safety
intervention aiming at identifying the current barriers against ADE existing in hospital
settings and at reinforcing them through innovative technologies [8].
Table 1 describes the compliance of the PSIP project with HCL. Following the
requirements of the UCD process, HF experts worked tightly with researchers and
industrials in the shaping of the produced knowledge and in the design of the
applications displaying this knowledge to clinicians and patients.
Three teams of HF experts from three countries (Italy, Denmark, and France)
characterized by different background expertise worked together on the project in
cooperation with a Danish patient-safety unit.
Figure 2. Human-Centered Lifecycle (HCL) process description adapted from the ISO 18529 [6].
164 R. Marcilly et al. / Integrating Human Factors in an International Research Project
Table 1. Human-centered lifecycle process activities applied to the PSIP project.

HCL Corresponding activities in PSIP
activities
#1 UCD process was explicitly integrating in the project as accepted and funded by the EU
#2 Human-centered activities were defined beforehand in the project plan and HF
representatives participated in the management board.
#3 #4 #5 #6 A UCD process has been followed to design the applications developed during the project.
#7 Within the duration of the project (40 months), the most advanced application (ADE-
scorecards) has been installed in a hospital and a one-year monitoring has been performed.
2. Methods
Main contributions of each HF team are presented in Table 2. Those methods were
applied all along the project for the HF patient-safety intervention and for the
development of innovative ADE prevention tools. Therefore representative end-users
and industrials have often been engaged (e.g. through focus groups, tests) all along the
project to optimize the intervention and the design process. Finally, HF experts worked
on a list of HF recommendations for the design of medication-related CDSS. Those
principles were proposed to industrials partners to assess their appropriateness and
technical feasibility.
3. Results - Lessons learned
Integrating HF expertise all along the project and at every level proved to be an
efficient approach. Participation in the management board is essential for the HF
approach and results to be actually integrated in the decision and development process.
3.1. ADE prevention requires complementary tools for the whole healthcare team
The multi-perspective approach allowed identifying that ADE prevention requires more
than a single tool addressed to a single healthcare professional. On the contrary, a set of
functions reinforcing the existing effective barriers developed by every professional is
necessary.
Table 2. Expertise fields and main contributions of each HF team. Tasks were often carried out in
cooperation.
HF teams Expertise Main contributions
Italy Italian’s expertise was oriented toward Analysis of the existing barriers of prevention
the error prevention through barriers against ADE in hospital setting based on
analysis and came from the aviation taxonomy of errors and work systems analysis
domain. [9].
Denmark Danish expertise was related to co- Design support and product evaluation based
design and user-driven design methods respectfully on co-design process [10] (e.g.
applied to medical devices and to the focus groups, prototyping) and simulation in a
evaluation within a full-scale medical full-scale simulator [11].
simulator.
Patient safety unit Validation of knowledge and of the
interventions in terms of patient safety risk.
France French team’s skills dealt with UCD process based on work system analysis,
usability, UCD process and cognitive end-users involvement, usability inspection and
ergonomics. test supported by usability goals [12].
Table 3. Functions required to reinforce the existing barriers and corresponding developed tools.
Required functions Developed tools
Raise the ADE awareness of every ADE-scorecards tool that allows clinicians getting an overview
involved clinician (physicians, of the incidence and quality of ADE in their own wards.
pharmacists and nurses)
Provide them with information at the CDSS integrated in Clinical Information System (CIS) or
point of care Computerized Prescription Order Entry (CPOE) informing
clinicians in real time of the risk of ADE related to ordered
meds.
Allow clinicians testing orders apart Standalone Intranet CDSS available to every clinicians apart
from their workflow from their workflow to simulate an order
Provide patients with medical Mobile patient component participating with clinicians to
information to support their patients’ awareness of ADE risks due to their medications.
compliance
This set is described in Table 3. Therefore, to optimally support ADE prevention,
designed tools should be clinicians’ partners, i.e. interacting with clinicians taking into
account their actions, and also team players, i.e. supporting the cooperation between
every healthcare professional involved.
Confronted to industrials’ technical proposals, those functions were developed
through four kinds of tools (Table 3). This set of tools was designed for every
healthcare professional involved in the medication use process (physicians, pharmacist,
nurses etc.). Hence, they are not workflow-dependent and may be implemented in
different kinds of work organizations.
3.2. Safety oriented UCD process is useful to make release decisions
Following a safety-oriented UCD process allows assessing the usability level of the
different tools and whether their usability features may cause use errors. Therefore
sufficiently usable tools without usability flaws that may cause use errors can be
identified and released.
During the project, the different tools did not achieve the same usability maturity.
At the end of the project, the patient component still faced remaining usability flaws
along with knowledge display issues. Those problems may have caused use errors.
Therefore, this tool was not released and researches about how to provide patients with
medical information had to go on. As for the integrated tools, they were more difficult
to evaluate than the standalone ones because integrated CDS functions inherit the
usability features from CPOE/CIS. Thus it was not obvious distinguishing the proper
CDSS usability features from the main system ones. At the end of the project,
improvements still had to be done on them to ensure that no usability flaws would
cause use errors. Finally, the standalone CDSS and the ADE-scorecards were assessed
as sufficiently usable and use error-prone-free and were released. The ADE-scorecards
were even installed in test wards of a northern France hospital for an impact study [13].
3.3. Industrials-HF experts collaboration is a win-win cooperation
The tight cooperation between HF experts and industrials (knowledge and CIS/CPOE
vendors) in shaping of the knowledge and designing the tools convinced the industrials
of the usefulness of the HF approach: they are still working with PSIP HF experts for
the PSIP follow-up and they integrate now HF engineering in their technological
innovations projects. From the HF perspective, based on industrials’ collaboration, the
166 R. Marcilly et al. / Integrating Human Factors in an International Research Project
appropriateness and the feasibility of formulated principles for the design of CDSS
were validated [14] improving their usefulness.
4. Conclusion
The PSIP project ended-up successfully one year ago. The positive dynamic it created
amongst participants (industrials and health institutions) still continues. The ADE-
scorecards tool is well-received by users and health institutions. It is still installed in
the test wards to continue the impact evaluation in terms of patient safety on a longer
duration. Several hospitals are asking for roll out.
Acknowledgment
The research leading to these results has received funding from the European
Community’s Seventh Framework Program (FP7/2007-2013) under the Grant
Agreement n°216130-the PSIP project.
The authors would like to thank every Human Factors expert whose involvement
made completion of this project possible: C. Nøhr, M.B. Christiansen, A.M. Kanstrup,
K. Lawton, P. Skjødt, S. Jensen, C. Binzer, N. Leroy, S. Pelayo, S. Bernonville, C.
Riccioli & C. Cacciabue.
References
[1] K.A. Thursky, M. Mahemoff, User-centered design techniques for a computerised antibiotic decision
support system in an intensive care unit, Int J Med Inform 76 (2007), 760-768.
[2] J. Chan, K.G. Shojania,A.C. Easty, E.E. Etchells, Does user-centred design affect the efficiency, usability
and safety of CPOE order sets? J Am Med Inform Assoc 18 (2011), 276-281.
[3] Council Directive 2007/47/EC, Council Directive 2007/47/EC, (2007).
[4] International Electrotechnical Commission. Medical Devices - Application of usability engineering to
medical devices (Rep. No. ISO 62366), Geneva: International Electrotechnical Commission; 2008.
[5] International Standardization Organization, Ergonomics of human system interaction - Part 210: Human
centred design for interactive systems (Rep N°9241-210), International Standardization Organization,
Geneva, 2010.
[6] International Standardization Organization, Ergonomics - Ergonomics of human-system interaction -
Human-centred lifecycle process descriptions (ISO/TR 18529). International Standardization
Organization, Geneva, 2000.
[7] R. Beuscart, P. McNair, J. Brender, Patient safety through intelligent procedures in medication: the PSIP
project. Stud Health Technol Inform 148 (2009), 6-13.
[8] M.C. Beuscart-Zephir, N. Nohr, Human factors engineering for computer-supported identification and
prevention of adverse drug events. Stud Health Technol Inform 148 (2009), 14-24.
[9] P.C. Cacciabue, G. Vella, Human factors engineering in healthcare systems: the problem of human error
and accident management, Int J Med Inform 79 (2010), 1-17.
[10] A.M. Kanstrup, C. Nøhr, Gaming against medical errors: methods and results from a design game on
CPOE, Stud Health Technol Inform 148 (2009), 188-196.
[11] E. Ammenwerth, W.O. Hackl, K. Binzer, T.E. Christoffersen, S. Jensen, K. Lawton, P. Skjoet, C. Nohr,
Simulation Studies for the evaluation of health information technologies: experiences and results, HIM
Journal 41(2012), 14-21.
[12] R. Marcilly, S. Bernonville, C. Riccioli, M.C. Beuscart-Zephir, Patient safety-oriented usability testing:
a pilot study, Stud Health Technol Inform 180 (2012), 368-372.
[13] R. Marcilly, W.O. Hackl, M. Luyckx, E. Ammenwerth, Scorecards: a new method to prevent adverse
drug events? Preliminary results from a clinical field study, Stud Health Technol Inform 166 (2011),
234-245.
[14] S. Pelayo, R. Marcilly, S. Bernonville, N. Leroy, M.C. Beuscart-Zephir, Human factors based
recommendations for the design of medication related clinical decision support systems (CDSS), Stud
Health Technol Inform 169 (2011), 412-416.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-168
Usability Inspection to Improve an

Electronic Provincial Medication
Repository
Nicole A. KITSON aͳ, Morgan PRICE a,b, Michael BOWEN a, Francis LAUa
a
Health Information Science, University of Victoria, b Department of Family Practice,
University of British Columbia
Abstract: Background: Medication errors are a significant source of actual

and potential harm for patients. Community medication records have the
potential to reduce medication errors, but they can also introduce unintended
consequences when there is low fit to task (low cognitive fit). PharmaNet is a
provincially managed electronic repository that contains the records for
community-based pharmacy-dispensed medications in British Columbia. This
research explores the usability of PharmaNet, as a representative community-
based medication repository. Methods: We completed usability inspections of
PharmaNet through vendor applications. Vendor participants were asked to
complete activity-driven scenarios, which highlighted aspects of medication
management workflow. Screen recording was later reviewed. Heuristics were
applied to explore usability issues and improvement opportunities. Results and
Discussion: Usability inspection was conducted with four PharmaNet
applications. Ninety-six usability issues were identified; half of these had
potential implications for patient safety. These were primarily related to login
and logout procedures; display of patient name; display of medications; update
and display of alert information; and the changing or discontinuation of
medications. Recommendations: PharmaNet was designed primarily to support
medication dispensing and billing activities by community pharmacies, but is
also used to support care providers with monitoring and prescribing activities.
As such, some of the features do not have a strong fit for other clinical
activities. To improve fit, we recommend: having a Current Medications List
and Displaying Medication Utilization Charts.
Keywords Drug Information System; Usability; Medication Management;

Improvement
Background
Medication errors are a significant source of actual and potential harm to patients [1].
Medication errors can include the wrong dose, frequency, route, and allergy details [2],
duplicate dosing [3], and other unintended interactions [4]. Community medication
records have the potential to reduce medication errors throughout the medication
management workflow [5, 6], such as: providing medication information at the point of
prescribing and dispensing [7], and supporting medication reconciliation [8]. However,

1
Corresponding Author. Box 3050 STN CSC Health Information Science, University of Victoria,
Victoria, BC V8W 3P5; Email: [email protected].
N.A. Kitson et al. / Usability Inspection to Improve an Electronic Provincial Medication Repository 169
using infoormation systeems may also o inadvertentlly introduce unintended

u coonsequence
[9]. They can introducce new types of errors, dep pending how the informatiion system
was desiggned and its usability [3]. This can bee the case wh hen the tool has a low
cognitive fit to the task
k. Cognitive fit refers to match between thet problem-soolving task
(e.g. revieewing a patieent’s medicatiion history) anda the probllem representtation (e.g.
electronic dispensing record) [10]. Higher
H cognittive fit improv
ves performannce, as has
been seenn in medication administratiion records [1 11]. The purpose of this ressearch was
to inspect the usability of a representtative commun nity based meedication repoository, and
he fit of a disp
explore th pensing recordd for common clinical tasks at the point oof care. We
then dreww out recom mmendations to t improve design
d of meedication reppositories /
dispensingg records, withh a focus on patient
p safety and
a improving g cognitive fitt.
Methods
PharmaNeet is a provinccially managed d electronic reepository that contains the rrecords for
communitty-based pharm macy-dispensed medication ns in British Columbia.
C PharmaNet is
accessed byb communityy pharmacistss and physiciaans through ceertified 3rd paarty vendor
applications [12]. Pharm maNet vendorrs were sent an a invitation letter via emaiil. Vendors
who agreeed to particippate completed d activity-drivven scenarios using five exxisting test
patients from
f the PhharmaNet traiining environ nment (not realr patient data). We
developed d standardized d user activitiees to highlighht aspects of medication
m management
workflow that were co overed by PharmaNet’s features (determ mined by reviewing the
PharmaNeet conformancce specificatio ons and mapp ping these to medication
m m
management
workflow)). (e.g., Figu ure 1, all annotated
a waalkthroughs area available freely at
ehealth.uvvic.ca).
We completed
c usaability inspectiions remotely y via screen shharing, whichh permitted
us to view w the vendorrs’ screens on n our local computers.
c W facilitated inspection
We
sessions while
w the venddors walked thhrough the sceenarios. We to ook extensive notes, and
the screenn was recordeed for analysiis. Three anallysts attended the walkthrooughs. One
analyst reeviewed the sessions in deetail using Jak kob Nielsen’s ten usabilityy heuristics
[13]. A seeverity scale was not used d; rather speciific issues weere mapped too focus on
broader design
d issues and improveements. The other team members revviewed the
inspectionn report to ennsure accuracyy and compleeteness. Confidential usabillity reports
were prov vided back to o the particippants to inforrm potential product imprrovements.
Aggregatee findings rellated to usability issues th hat had poten ntial safety isssues were
collected and summarrized. We compared c theese findings with the coonformance
specificatiions.
Figure 1: Example walkthrough for usability inspection. Simulated patient with allergy to penicillins.
170 N.A. Kitson et al. / Usability Inspection to Improve an Electronic Provincial Medication Repository
Results
Seven PharmaNet vendors, selling one or more PharmaNet applications, were invited
to participate; two consented. Usability inspection was conducted for four PharmaNet
applications (all participating vendor products). Inspection occurred in March 2011.
Ninety-six usability issues were identified across the four applications. The largest
proportion of observed usability issues (44%) was related to error prevention
(conditions that could lead to errors, e.g., being unable to confirm an action before
proceeding). Half (49%) of the observed usability issues had potential implications for
patient safety as determined by the analysts. Usability issues with implications for
safety were primarily related to: login and logout procedures (e.g., unclear procedures;
user name not clearly displayed); the display of patient name (e.g., patient name lost in
scrolling or covered by popup windows); the display of medications (e.g., no current
medication list, current medications not labeled, potential truncation of medication
administration directions, difficult to locate dose changes); the update and display of
alert information (e.g., timing of alerts, adverse reactions linked to a specific drug code
not the active ingredient or class); and the changing or discontinuation of medications
(e.g., unclear dosage selections, lengthy or incomplete picklists).
Discussion
For login and logout procedures, it is important to ensure that the correct authorized
user is accessing the system at any given time. A poorly visible login feature can
increase reliance on the previous user to logout; similarly, a poorly displayed logout
function can make it difficult for users to remember to logout [3]. We observed ways to
mitigate these challenges, including: clear login and logout buttons that do not scroll
off screen; straightforward, logical login and logout procedures; and screen timeout.
The display of patient name should be constant and consistent to help reduce the
likelihood that the wrong patient is being reviewed [3, 14]. There can be many
interruptions in practice; therefore, the patient name should never be obscured on
screen. We observed the patient name lost in scrolling, overlapping boxes, or not
appearing on each screen. It should be consistently and clearly displayed along with
key identifiers to ensure the right patient is being accessed at all times [15].
One of the key functions of a medication repository is to display medications to a
provider. A chronologic dispensing record can be very helpful to providers [16];
however, for many of the common medication related activities, it requires the user to
perform excess cognitive tasks. For example, to determine an expected list of current
medications from a dispensing record, the user must review each dispensing item and
calculate an expected end date and then compare it to today’s date. Similar mental
calculations must be made for medication use rates. Dose changes can be difficult to
locate in a long dispensing list. The user must remember current dose and then find
previous dispensings and mentally compare prescribing instructions.
In terms of the update and display of alerts, we found good instances of clearly
displayed alerts provided at the appropriate points. However, the reverse was also
identified, for instance, drug allergies appearing after the user dispensed the medication.
It appeared that adverse reactions are linked to a Drug Identification Number and did
not appear to be linked to the active ingredient or drug class. Identical alerts could be
duplicated when the vendor’s own pharmacy information system provided drug-to-drug
interactions and maintained its own medication list in parallel to PharmaNet (all
pharmacy information systems are required to do this). This could lead to user alert
fatigue and alerts may be overlooked [17].
The patient’s medication profile should be up-to-date by ensuring medication
updates (changes or discontinuations) are in the system. This helps inform medication
decision-making and to generate the appropriate alerts (e.g., drug-to-drug interactions).
Limits on who can update a dispensing repository can impact accuracy of the data.
These features should fit the workflow of appropriate members of the care team. For
example, physicians often will cancel or change a medication dose without requiring a
new dispensing (e.g. “use the medication you have already, just take half a tablet
instead of a whole one”). In a dispensing repository like PharmaNet, only those who
dispensed a medication could “correct” that dispensing record. Prescribers and others
could not go in and change a previous dispense record to reflect the current activities of
the patient; this does not fit well with the task of reviewing and updating medications.
Recommendations
Systems may be designed to support primary medication activities (e.g., dispensing,

billing), but may subsequently be used to support other medication activities not
originally intended (e.g., monitoring, prescribing). As such, some of the features do not
have a strong fit for other clinical activities. These types of usability issues likely
extend beyond PharmaNet to other dispensing records that provide medication
information for clinical decision-making. At the core of these challenges is providing
useful (accurate) and easy to use information to the care provider, at a time that it
makes sense in the workflow [3]. Pharmacy information systems that do not support
the existing workflow can lead to unintended consequences, including medication
errors [14]. To improve the fit of a dispensing record to the work of reviewing
medication use, we present two recommendations: displaying (and maintaining) a
Current Medications List and Displaying a Medication Utilization Chart.
A current medications list would help a provider separate historical from current
medications in a dispensing record and focus on the most recent interventions [18].
Being able to separate current from previously taken medications is an important step
to informing treatment decisions. Reviewing a reverse-chronologic dispensing record
requires the provider to sift through dispensing details, manually calculating and
extrapolating current medications. This can be a tedious and time-consuming task that
one could assume would increase chance of error. A current medications list, however,
is a better fit for the task of reviewing medications and, with a better fit, reduces the
cognitive load of the provider [10]. This would theoretically improve usability and
reduce error, sparing the provider from mentally completing this task for each item by
displaying a list of current medications within the system (Figure 2). Having current
medications clearly highlighted is an important step towards gaining a picture of the
patient’s medication story without being overwhelmed with extraneous information.
172 N.A. Kitson et al. / Usability Inspection to Improve an Electronic Provincial Medication Repository
Figure 2: Example
E illustrattion of a potenttial current med
dication list, keyy information suuch as which
medications are current and when
w the current dispensed mediccations are expectted to end are proovided in one
list.
A meedication utiliization chart supports a review
r of meedication utillization by
providing a visual sum mmary of a specific disp pensed medicaation over tim me. When
reviewing g a dispensing record, it cann be difficult to
t draw out key
k informatioon [18] like
dosage ch hanges and addherence rates (e.g., pickup p frequency frrom the pharm macy) for a
single medication over time, especiaally when each h dispensing is
i represented separately
and mixed with all otther medicatiions. Figure 3 illustrates an a example m medication
n chart demon
utilization nstrating howw it can track medication usage
u over timme (in this
case, dailyy dispensings of methadonee). Again, thiss tool providess a better cognnitive fit to
the task of reviewing g medication utilization th han the simp ple reverse-chronologic
dispensing g record. In this
t chart, th
he provider caan see at a glance
g the freequency of
dispensing gs, dosage changes,
c d adherence. This provid
and des the userr with an
informativve snapshot too inform clinical decision-m making and reeduces cognitive load in
an alreadyy busy patient encounter. We
W observed on ne example off this in our innspections.
Figure 3: Exxample of a mediication utilization n chart using daily

y-dispensed meth
hadone as an exaample. Note it
is very easy to see that the paatient did not get any
a methadone dispensed for 4 daays in a row in Maarch.
Referencees
[1] Kohn L.
L To err is humaan. National Acaddemy Press; 2000.
[2] Bates D, Teich J, Leee J, Seger D, Ku uperman GJ, Ma’Luf N, Boyle D, Leape L. Thhe Impact of
Compuuterized Physiciann Order Entry on
n Medication Prev vention. JAMIA 1999; 6: 313-321
[3] Koppel R, Metlay J, Cohen
C A, Abauck k B, Localio R, Kimmel
K SE, Strom BL. Role of CComputerized
Physician Order Entry Systems
S n Errors. JAMIA 2005;
in Facilittating Medication 2 293: 1197--1203.
[4] Lau H, Florax C, Porsius A, DeBoer A. The completeness of medication histories in hospital medical
records of patients admitted to general internal medicine wards. Brit J Clin Pharmaco 2000; 49: 597-
603.
[5] Bell D, Cretin S, Marken R, Landman A. A Conceptual Framework for Evaluating Outpatient
Electronic Prescribing Systems Based on their Functional Capabilities. JAMIA 2004; 11: 60-70.
[6] Bassi J, Partridge C. Electronic Prescribing: Workflow Analysis Handbook. University of Victoria
eHealth Observatory: Victoria, BC; 2010.
[7] BC-Pharmacare. PharmaNet. [Internet] 2012. Available from:
http://www.health.gov.bc.ca/pharmacare/pharmanet/netindex.html
[8] Bassi J, Lau F, Bardal S. Use of Information Technology in Medication Reconciliation: A Scoping
Review. Ann Pharmacother 2010; 44: 885-897.
[9] Ash J, Berg M, Coiera E. Some Unintended Consequences of Information Technology in Health Care:
The Nature of Patient Care Information System-related Errors. JAMIA 2004; 11: 104-112.
[10] Vessey I. Cognitive Fit: A Theory-Based Analysis of the Graphs Versus Tables Literature. Decision Sci
1991; 22: 219-240.
[11] Borycki EM, Kushniruk AW, Kuwata S, Watanabe H. Simulations to Assess Medication
Administration Systems, in Nursing and Clinical Informatics: Socio-Technical Approaches, B.
Staudinger, V. Höß, and H. Ostermann, Eds. Hershey PA: Information Science Reference; 2009: 144-
159.
[12] PharmaNet: Professional and Software Compliance Standards, Volume 3 – Technical Rules, Pharmacy.
BC Ministry of Health Services: Victoria, BC; 2010.
[13] Nielsen J. 10 Heuristics for User Interface Design. [Internet]. 2005. Available from:
http://www.useit.com/papers/heuristic/heuristic_list.html
[14] Khajouei, Jaspers M. The Impact of CPOE Medication Systems’ Design Aspects on Usability,
Workflow and Medication Orders: A Systematic Review. Methods Inf Med 2010; 49: 3-19.
[15] Guidance – Patient ID Number Input and Display: Microsoft Health Common User Interface. [Internet].
2010. Available from: http://mscui.net/DesignGuide/PatientNoDisplay.aspx
[16] Kent H. BC’s PharmaNet system proving convenient. CMAJ 2000; 162: 1192.
[17] Ash JA, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some Unintended Consequences of
Clinical Decision Support Systems, In: Proceedings of the 2007 American Medical Informatics
Association Annual Symposium; 2007: 26-30.
[18] Pelayo S, Leroy N, Guerlinger S, Degoulet P, Meaux J-J, Beuscart-Zéphir MC. Cognitive Analysis of
Physicians’ Medication Ordering Activity, In: Proceedings of ENMI: Connecting Medical Informatics
and Bio-Informatics; 2005.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-174
The Long and Twisting Path: An Efficiency

Evaluation of an Electronic Whiteboard
System
Rasmus RASMUSSEN a, 1 and Andre KUSHNIRUK b
a
Department of Communication, Business and Information Technologies, Roskilde
University, Denmark, b School of Health Information Science, University of Victoria,
British Colombia, Canada
Abstract. Electronic whiteboard systems are becoming increasingly popular as

replacements for the dry-erase whiteboards previously used for communication
and workflow coordination at Emergency Departments. With this it also becomes
increasingly important that these systems do not disrupt or delay the working
practices of the departments where they are taken into use. Usability evaluations
should therefore be employed as part of developing and implementing these
systems. We report on a subset of the results from a larger usability study of a
electronic whiteboard and find that there are inefficiencies, which could be
mitigated by a relatively simple redesign and thus improve the usability of the
system.
Keywords. electronic whiteboards, usability evaluation, efficiency, GOMS-KLM
Introduction
Electronic whiteboard systems (EW) are becoming increasingly popular as

replacements for the ubiquitous dry-erase whiteboards used for communication and
workflow coordination in emergency departments (ED) [1]. However, with this
increase in popularity it becomes ever more important that these EWs do not disrupt or
delay the working practices of the EDs where they are taken into use. Usability
evaluations should therefore be conducted as part of developing and implementing
these systems to uncover any potential usability issues. In this paper we report on a
subset of the results from a larger usability study performed as part of an evaluation of
a specific EW system at two Danish EDs. The focus of this paper will be on efficiency,
which refers to the number of resources needed to complete tasks with a system and
which constitutes a key aspect of usability [2]. For example, high efficiency occurs
when minimal steps are required to complete a task using a system such as an EW. In
this paper we explore the analysis of data collected from real users working with an
EW system over time in order to determine if inefficiencies can be identified leading to
proposed redesigns developed based on the analysis.
1
Corresponding author: Rasmus Rasmussen, Roskilde University, Universitetsvej 1, 4000 Roskilde,
Denmark; E-mail: [email protected]
R. Rasmussen and A. Kushniruk / The Long and Twisting Path: An Efficiency Evaluation of an EW 175
Figure 1. Screen shot cutout of the EW system’s user interface
1. Methods
The usability study was performed as a longitudinal and naturalistic study of the ED
clinicians’ interactions with the EW system. The study involved continuous and long-
term screen recordings of the clinicians’ interactions with the EW system throughout
multiple five-hour periods during dayshifts at two EDs. The healthcare region and ED
management approved the study prior to it being conducted. Also, because the study
involved collection of live patient data the study had to be registered and approved by
the Danish Data Protection Agency. Clinicians on duty during the study were briefed
during morning meetings and throughout the study if questions or concerns arose.
1.1. The Electronic Whiteboard
The EW system is a web-based application installed on a central server and is

accessible from all web-enabled devices connected to the same network as the server,
e.g. laptops, workstations and wide screen displays. The system displays patient related
information relevant for coordinating workflow and patient care e.g. name, age,
medical problem, triage levels, attending nurse and physician, lab results, etc. Figure 1
presents the general information structure using a matrix with rows for patients and
columns for patient data.
1.2. Procedure and Materials
User interactions with the EW were captured using the HyperCam 2 screen capture
software installed on 4 Gb flash drives. The resulting video files were stored on either
an external 2 Tb hard drive or a 16 Gb flash drive. User interactions were captured over
a period of five days between 10 AM and 4 PM each day at two EDs. This period was
specifically chosen because experience proved this was often the busiest time of the
day and should therefore produce the highest number of interactions with the EW.
Name
Description
Yes Number of steps to complete
Activity Complete task?
No Number of steps before aborting
Yes Code issue

Usability issue?
No
Figure 2. The coding scheme used for logging on-screen activities

176 R. Rasmussen and A. Kushniruk / The Long and Twisting Path: An Efficiency Evaluation of an EW
The recordings rotated between different workstations throughout the departments

and the wide-screen displays located in the ED command rooms. Finally, the first-
author was present at the departments during the study to carry out concurrent
observations and in-situ interviews with users of the EW system and to administer the
recordings.
1.3. Data Analysis
Each video file was viewed and logged by the first-author using a predefined coding
scheme developed by both authors – see Figure 2. The initial viewing and logging was
carried out solely by the first-author due to restrictions imposed by the Danish Data
Protection Agency’s directives. Using the coding scheme, on-screen activity was
recorded and entered as entries in separate log files. Each entry contains a timestamp,
an activity indicator, a name for the activity and a description of the activity. In
instances of task oriented activity the entries indicate whether or not the task was
completed and the number of steps taken before completing or aborting the task. When
usability issues were discovered we marked them with an indicator and coded the
issues using one of the following categories: System bugs, efficiency problems, error
messages and work patterns. We also provided a description of the issue including
whether or not the user solved the issue.
Following the logging process both authors perused each log file and coded
activities of interest for further analysis. All codings were discussed to mitigate biases
in the analysis. The initial coding of the data was used to locate particular interactions
of interest that were further analyzed to identify efficiency issues. In this paper we
focus on the analysis of potential inefficiencies. This involved using the GOMS-KLM
method [3] to calculate how much time a theoretical expert user would spend on
completing a task with the EW system following a specific pathway determined by the
original system design.
Based on these analyses, our approach then involves proposing a redesign aimed at
improving the efficiency for that task, calculating how much time an expert user would
spend using this design before finally comparing this with the GOM-KLM calculations
for the original design.
2. Results and Discussion
We logged a total of 2863 entries and recorded 13 unique usability issues: 4 system
bugs (55 instances), 4 efficiency issues (141 instances), 3 error messages (38 instances)
and 2 inefficient working patterns (229 instances). The most common efficiency issues
found in the results concerned complicated and long pathways, which the EW system
forces the users to follow when using the system for specific tasks. This issue occurred
a total of 63 times (44.68 % of all instances of efficiency issues) in the results. The
complicated and long pathways become very apparent when new patients are added to
the EW or when certain information fields are updated. In logging the video files we
found that adding a new patient on average took 12.3 steps to complete. When users
completed this task error free e.g. without mistakes or interruptions and provide the
maximum amount of information the task required 19 steps to complete. These steps
are the following: 1) Open "Add row" dialog box 2) Open "SSN" dialog box 3) Input
SSN 4) Close "SSN" dialog box 5) Open “Note” dialog box 6) Type note 7) Close
R. Rasmussen and A. Kushniruk / The Long and Twisting Path: An Efficiency Evaluation of an EW 177
“Note” dialog box 8) Open “Problem” dialog box 9) Open problem selection dialog
box 10) Search/select problem 11) Close selection dialog box 12) Close “Problem”
dialog box 13) Open “Waiting for” dialog box 14) Select waiting for option 15) Close
“Waiting for” dialog box 16) Open “Location” dialog box” 17) Select location option
18) Close “Location” dialog box 19) Close “Add row” dialog box. As this indicates the
current design of the EW is based on individual dialog boxes for input into each
information field. In the following, we will use the add-patient task as an example to
demonstrate how the efficiency of the EW design could be improved via a simple
redesign. Using the GOMS-KLM method [3] we are able to calculate how much time a
theoretical expert user of the EW system would spend on completing the add-patient
task (see Eq. (1) where H = moving hands between mouse and keyboard, P = pointing
to a position on the display, K = tapping a key or button, M = mentally preparing for
next step – see [3] for definitions of the GOMS-KLM operators). Assuming that the
user starts the task with hands off the keyboard, that there is no system response time
and that the user inputs a 10-digit SSN and a 30-character note the calculations will be
as follows:
‫ ܪ‬൅ ‫ ܯ‬൅ ܲ ൅ ‫ ܭ‬൅ ‫ ܯ‬൅ ܲ ൅ ‫ ܭ‬൅ ‫ ܪ‬൅ ‫ ܯ‬൅ ሺ‫Ͳͳ כ ܭ‬ሻ ൅ ‫ ܪ‬൅ ‫ ܯ‬൅ ܲ ൅ ‫ ܭ‬൅ ܲ ൅ ‫ ܭ‬൅ ‫ ܪ‬൅ ‫ ܯ‬൅ ሺ‫Ͳ͵ כ ܭ‬ሻ ൅
‫ܪ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅
‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ܭ‬൅‫ܯ‬൅ܲ൅‫ ܭ‬ൌ
‫ݏ݀݊݋ܿ݁ݏ݊݅ݐ݊݁݌ݏ݁݉݅ݐ‬ (1)
When replacing the operators in Eq. (1) with the times they represent (H= 0.4
seconds, P = 1.1 seconds, K = 0.2 seconds and M = 1.35 seconds) we find that a
theoretical expert user would spend 56.4 seconds on completing the add-patient task
when following the pathway that the system currently enforces. The amount of input
information needed to complete the task is independent of the pathway followed and
therefore cannot be reduced by redesigning the interface. However, by reducing the
number of steps needed to complete the task it is possible to make the interface more
efficient than the current. We will demonstrate this by proposing a theoretical interface
design where the input information is entered directly in text fields or by selection via
drop-down menus instead of opening new dialog boxes for each individual input. Once
again we assume that the user starts the task with their hands off the keyboard and
mouse, that there is no system response time and that the user inputs a 10-digit SSN
and a 30-character note. In this case we have the following steps for the task: 1) Open
"Add row" dialog box 2) Select "SSN" input field 3) Input SSN 4) Select "Note" input
field 5) Input note 6) Select "Problem" option from drop-down menu 7) Select
"Waiting for" option from drop-down menu 8) Select "Location" option from drop-
down menu 9) Close “Add row” dialog box. When applying the GOMS-KLM
calculations to the proposed redesign we arrive at Eq. (2):
൅ ൅ ൅ ൅ ൅ ൅ ൅ ൅ ൅ ሺ ‫Ͳͳ כ‬ሻ ൅ ൅ ൅ ൅ ൅ ൅ ൅ ሺ ‫Ͳ͵ כ‬ሻ ൅ ൅ ൅ ൅

൅ ൅ ൅ ൅ ൅ ൅ ൅ ൅ ൅ ൌ ͵ͳǤʹͷ (2)
Thus, it would take an expert user of the EW system 31.25 seconds to complete the
task of adding a new patient when using the proposed redesign of the systems interface.
This simple redesign of the EW interface thereby presents a reduction of the theoretical
task completion time by 25.15 seconds (44.6 %). Taking into consideration that each
ED receives approximately 40.000 – 45.000 patients each year a redesign of this
pathway could prove to be a significant time saving improvement over the current
178 R. Rasmussen and A. Kushniruk / The Long and Twisting Path: An Efficiency Evaluation of an EW
design. In a conservative estimate the clinicians provide the maximum amount of

information for a new patient for every second patient admitted to the EDs and added
to the EW. This leads to a time saving of 157.2 hours each year for this task alone. In
cases where the clinicians do not provide the maximum amount of information this
time saving will be smaller but still noticeable. Furthermore, since adding patients to
the EW is not the only task where the system enforces longer than necessary pathways,
a redesign of the input method used throughout the EW system’s interface could even
further increase the amount of time saved when using the system.
Whether or not the proposed redesign would in fact translate to actual time saving
if taken into everyday use is of course a matter of further researcher and
experimentation. However, previous evaluations using variations of GOMS methods
have proven that these calculations are often precise [4] and therefore we feel assured
that our proposed redesign would in fact provide the calculated time saving benefits.
3. Conclusions
Through our usability evaluation of the EW system we found a wide range of usability
issues in the EW system. In this paper we chose to focus on long and complicated
pathways within the EW system. Using the GOMS-KLM we illustrated with an
example how the system could be redesigned to increase its efficiency. We found that
our proposed redesign could reduce the time needed to enter a new patient to the EW
by roughly 45 % and with the reservation that the clinicians do not always provide the
maximum amount of information we found that this could provide a time saving of
157.2 hours pr. year. Also, since the EW system can be accessed via multiple devices
the potential increase in efficiency could be far ranged and have a positive impact upon
the work practices at the ED. Furthermore, we argued that this redesign could have an
even greater impact than our results show since it could potentially affect a larger part
of the EW than we studied here. In conclusion we call for more and earlier usability
evaluations of healthcare information systems such as the EW studied here to ensure a
higher quality of the systems used by healthcare professionals.
References
[1] R. Rasmussen. Electronic whiteboards in emergency medicine: A systematic review, Proceedings of the
2nd ACM SIGHIT International Health Informatics Symposium (2012), USA, 483-492.
[2] E. Frøkjær, M. Hertzum, K. Hornbæk. Measuring usability: Are effectiveness, efficiency and,
satisfaction really correlated? Proceedings of the ACM CHI 2000 Conference on Human Factors in
Computing Systems (2000), 345-352.
[3] S.K. Card, T.P. Moran, A. Newell. The Keystroke-Level Model for User Performance Time with
Interactive Systems. Communications of the ACM 23(7) (1980), 396-410.
[4] W.D. Gray, B. John, M.E. Atwood. The precis of project ernestine – An overview of a validation of
GOMS, Proceedings of the CHI 92 Conference on Human Factors in Computing Systems (1992), 307-
312.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-179
Applying Usability Methods to Identify

Health Literacy Issues: An Example Using
a Personal Health Record
Helen MONKMANa,1 and Andre KUSHNIRUK a
a
Victoria, British Columbia Canada
Abstract. The prevalence of consumer health information systems is increasing.

However, usability and health literacy impact both the value and adoption of
these systems. Health literacy and usability are closely related in that systems
may not be used accurately if users cannot understand the information therein.
Thus, it is imperative to focus on mitigating the demands on health literacy in
consumer health information systems. This study modified two usability
evaluation methods (heuristic evaluation and usability testing) to incorporate the
identification of potential health literacy issues in a Personal Health Record
(PHR). Heuristic evaluation is an analysis of a system performed by a usability
specialist who evaluates how well the system abides by usability principles. In
contrast, a usability test involves a post hoc analysis of a representative user
interacting with the system. These two methods revealed several health literacy
issues and suggestions to ameliorate them were made. Thus, it was demonstrated
that usability methods could be successfully augmented for the purpose of
investigating health literacy issues. To improve users’ health knowledge, the
adoption of consumer health information systems, and the accuracy of the
information contained therein, it is encouraged that usability methods be applied
with an added focus on health literacy.
Keywords. usability testing, heuristic evaluation, health literacy, Personal

Health Record (PHR)
Introduction
Personal Health Records (PHRs) have been lauded for their potential to engage health
consumers and provide them with governance over their own health information [1].
PHRs are used to access, manage and share personal health information, privately and
securely [2]. Moreover, if they are interoperable, PHRs may contain data from
different sources (e.g., pharmacies, labs, EHRs) [1]. However, health consumers act
as curators of their PHRs and thus, decide what health information is and is not
included. Ideally, PHRs provide comprehensive, longitudinal depictions of both static
(e.g., allergies) and dynamic (e.g., blood pressure, cholesterol levels) health data. In
addition to serving as a data repository, some PHRs offer interactive tools to foster
self-management, education and decision-making.
1
Corresponding Author: Helen Monkman, email: [email protected]
180 H. Monkman and A. Kushniruk / Applying Usability Methods to Identify Health Literacy Issues
PHRs are unique in that they are healthcare applications designed for consumers
who have limited or no healthcare expertise [3]. For PHRs to facilitate self-
management, it is imperative that they are designed to accommodate consumers with
limited health literacy to ensure users’ comprehension of the information [4]. Ratzan
and Parker [5] define health literacy as “the degree to which individuals have the
capacity to obtain, process, and understand basic health information and services
needed to make appropriate health decisions” (pg. iv). Health literacy extends beyond
reading and writing and encompasses a variety of other skills (e.g., listening, speaking,
arithmetic, problem-solving and decision-making), and is also influenced by culture
and society [6]. Health literacy skills are associated with a broad range of benefits
from increased health knowledge, improved self-reported health status, and reduced
healthcare costs (e.g., shorter hospitalizations, less frequent use of health care
services) [7].
Consumer health information (e.g., PHRs) systems can undoubtedly benefit from
the application user-centered design methods [8]. It is argued here that there is a close
interaction between usability and health literacy, particularly when we consider the
effectiveness, efficiency and satisfaction of end users (i.e., patients and lay people)
with PHRs. Specifically, the PHR user’s capacity for obtaining, processing and
understanding basic (as well as their own) health information will influence how
usable such systems are.
Health literacy levels and usability both have the potential to impact data
accuracy, quantity and quality of the information users enter into their PHRs. For
example, medical terms and definitions not understood by PHR users could result in
data entry errors or omissions. Whereas, if PHRs are challenging to use, health
consumers are less likely to consistently monitor symptoms, vitals and other statistics
into their PHRs. Thus, to optimize the value of PHRs, it is imperative that they are
usable and designed to mitigate the demands on health literacy, as well as develop
health literacy skills.
Currently, there is a dearth of research on methods for identifying and
ameliorating health literacy issues in consumer health information systems. Given the
relationship between usability and health literacy, usability evaluation methods may
prove to be effective in detecting potential health literacy issues in consumer health
information systems.
Three studies inadvertently identified health literacy issues during investigating
of PHRs for other purposes. Kim and Johnson [9] compared the utility and
functionality of 11 web-based PHRs and revealed a barrier to use that stemmed from
a health literacy issue. These authors reported the PHRs failed to provide guidance or
explanation to facilitate user data entry. Specifically, these authors found no resources
to help users interpret prescription labels or test results to increase the likelihood that
they were entered into the PHRs accurately. Cimino, Patel, and Kushniruk [10]
reported that although patients wanted to know their lab results, some had challenges
understanding and interpreting them. Segall and colleagues’ [3] also found that many
participants could not understand the content of their medical reports. Additionally,
these authors also suggested replacing jargon with the vernacular and expanding
acronyms. For example, Segall and colleagues proposed using “as needed” rather “prn”
and “by mouth” instead of “orally” (pg. 1239) and suggested other changes to.
In the previous studies, health literacy issues were revealed without an apriori
objective of their identification. The present study will demonstrate how usability
methods can be extended with to identify both usability and health literacy issues.
H. Monkman and A. Kushniruk / Applying Usability Methods to Identify Health Literacy Issues 181
1. Methods
At the time of testing, the PHR used in this study was Canadian based and offered at
select clinics as a tethered solution. The PHR allowed the storage of a variety of
personal health information (e.g., allergies, medications, health conditions).
Additionally, several tools were provided by the PHR to manage (e.g., Appointments,
Journal, Care Plan) and monitor (i.e., body dimensions, cardio, blood glucose, peak
flow) health, with specific modules (i.e., Asthma, Cardio, Diabetes) focused on
chronic illnesses.
1.1. Heuristic Evaluation
A heuristic evaluation is a method of investigation whereby a small group of usability

experts (one to three) judges how well the system complies with usability principles
[11]; issues that arise while using the system are categorized according to the
heuristics used. A total of eleven heuristics were used for this evaluation from
Nielsen’s list of heuristics [12]. Nielsen’s heuristics were supplemented with a new
“Health Literacy” heuristic proposed by the authors of this paper. This heuristic is
used to identify and categorize when clinical information (contained in the PHR)
would be judged to be unlikely to be understood by a layperson who does not have a
healthcare background or education. Each issue identified was assigned a severity
rating to assess how detrimental these violations are to the usability of the system, and
how urgently the issue should be solved [11].
The usability specialist for this heuristic evaluation selected four common PHR
tasks. The four tasks were evaluated using PHR were: 1) Enter a Lab Result; 2) Enter
a Medication; 3) Check and Edit An Allergy; and 4) Input and Check Blood Pressure.
The investigator completed these four tasks using the PHR. Issues that arose while
attempting to do the task were assigned a heuristic and evaluated for severity. In
addition to usability issues, areas where the PHR demonstrated good design were also
identified. Screen shots were taken to illustrate system issues and recommendations
for improvements were suggested.
1.2. Usability Testing
In order to compare the results from the heuristic evaluation to the usability testing
method (and to explore the utility of this method to reveal health literacy problems)
one participant was used for pilot usability testing purposes. To do this, the PHR
account was pre-populated with health information to give the impression of an
existing record. A MacBook Pro laptop was used for this experiment. QuickTime
Player version 10.1 was used to simultaneously record the audio and the events that
occurred on-screen.
After the participant was seated in front of the laptop computer with the web
browser on the PHR login page, the audio and screen recording began. Once
QuickTime Player was running, the experimenter read the instructions to the task. The
participant was asked to “think aloud” or verbalize his thoughts as he attempted to
complete each task [8]. The participant completed the following three tasks
sequentially: 1) Enter a medication; 2) Interpret vital sign data; and 3) Determine
allergy documentation. After completing each task, the participant was asked to
evaluate and comment on various aspects of the screens they encountered during the
previous task. Finally, when the participant completed all three tasks, he was asked to
make general comments about the PHR in a brief post-task interview.
2. Results
2.1. Heuristic Evaluation
A total of 15 usability issues were revealed during this investigation of four tasks. The
heuristic that was violated most frequently was Health Literacy, followed by
Recognition rather than Recall, and Flexibility and Efficiency of Use (See Figure 1).
All of the Health Literacy violations occurred during the Medication and Allergy
tasks. As indexed by no violations for heuristic Error Prevention, the PHR was well
designed from this perspective. For example, the PHR clearly described any errors
that occurred while inputting data and the steps necessary to complete the data entry
successfully. Also, any data that was entered could be subsequently edited or deleted
very easily. The PHR was well organized; the categories and sub-sections were
logically grouped and all of the groups of items were kept within reasonable memory
limits.
One Health Literacy violation was rated as catastrophic and occurred in the
Allergies section. When adding a new allergy, the drop-down menu was confusing,
inconsistent and failed to list many common allergies (e.g., penicillin, peanut). For
example, “Penicillamine allergy”, “Penicillinase-resistant penicillins allergy” and
“Penicillinase-sensitive penicillins allergy” are allergy entries listed. The complexity
of these words and the nuances between the latter two options were unlikely to be
understood by a layperson without a healthcare background and users may be more
likely to select the incorrect option if they were trying to report a “Penicillin allergy”.
As an example of the list’s inconsistency, this menu listed pollen as a food allergy
rather than environmental. In order to remedy this issue, a detailed investigation of the
contents of the allergy options should be conducted. It is imperative that allergies can
be reported as simply as possible. If adding allergies is challenging, the probability
that errors in data entry will occur are increased and could have critical consequences
in an emergency situation.
1. Visibility of system status

2. Match between system and the real…
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, and…
10. Help and documentation
11. Health Literacy
0 2 4 6
Figure 1. Frequency of PHR Heuristic Violations

2.2. Usability Testing
The video of the participant interacting with the PHR was transcribed and annotated
in full, and then coded for usability issues. Of the three tasks completed by the
participant, the most usability problems (12) occurred when the participant was trying
to enter a medication. The PHR failed to facilitate entering a medication. For example,
the participant did not know what the strength and dosage of a medication values
were and repeatedly received error messages.
After completing all three tasks, the participant indicated that design changes
could improve the usability of the system. Specifically, the participant felt that the all
of the content font size should be adjustable, so it could be increased for easier
reading. Further, he expressed it would be helpful to have a link to more information
on medications and medication interactions. Lastly, the participant also wanted titles
on the blood pressure graphs and increased visibility of outlier values (e.g., red data
points).
A health literacy issue that was identified in both the heuristic evaluation and the
usability test was the medication entry dialogue box. This box was overly complex
and had fields for a lot of detailed information that users may not know. Further,
many of the categories of information may not be clear to users who do not have
medical expertise. The participant experienced difficulty differentiating between
strength and dosage. The PHR should facilitate understanding by adopting simpler
terms and providing examples. An approach to making prescription labels more
understandable and informative was found on the Women’s Health website
(http://www.womenshealth.gov/aging/drugs-alternative-medicine/how-to-read-drug-
labels.cfm). This website decomposed an image of a prescription to describe its
constituent parts with simple, easy to understand language. Making this information
available to users of PHRs is beneficial because the user has the opportunity to
improve their understanding of their prescription. Further, this strategy may also
increase the accuracy of the information entered into the PHR, which may be critical
in the case of an emergency. Thus, it is strongly recommended that the PHR adopts
strategies like this that increase user knowledge and simultaneously make information
easier to enter and more likely to be accurate.
3. Discussion
The heuristic evaluation and usability study proved to be complementary methods of

investigating both the usability and health literacy of the PHR. Some issues were only
revealed using one technique and thus may have gone unnoticed if only one method
was applied. Although the focus of the work described in this paper has been on
extending heuristic evaluation with health literacy considerations, we found that the
results of this study were more extensive as a result of using both heuristic evaluation
and usability testing. Whenever feasible, it is encouraged that these methods are
applied in conjunction to collect the most comprehensive data set possible. Although
one pilot participant demonstrated the utility of usability testing to identify health
literacy problems (for comparison purposes with the heuristic evaluation method),
more participants should be used in practice. Along these lines we are currently
planning on carrying out a follow-up study that will include usability testing with
users of varying levels of health literacy as subjects. Ameliorating the issues
identified in this study through design changes, will result in a more usable PHR.
Moreover, modifying the content to accommodate its layperson user population will
increase the accuracy of the information entered and the PHR could serve as a useful
resource for health consumers.
Given the exploratory nature of this study, only a single usability specialist
conducted the heuristic evaluation. However, multiple evaluators have been shown to
reveal more usability issues than a single evaluator [12]. Thus, it is recommended that
for more detailed analysis, several evaluators should be included to identify a
comprehensive list of heuristic violations.
Some health literacy issues in PHRs may arise as a consequence of designers
adopting functional components from electronic health records (EHRs), which are
used by providers. Although this is not problematic for many areas, previous research
and the current study demonstrated that some health topics (e.g., medications, lab
results, allergies) require extra attention devoted to designing with health literacy
considerations in mind.
The utility of applying and extending usability methods to detect health literacy
issues in consumer health information systems was demonstrated in this study. Health
consumers might have limited medical knowledge, and design strategies should be
developed to assist users in understanding health information and inputting data
accurately. Some examples of techniques that could be adopted by consumer health
information systems to help improve the health literacy of their applications are: using
multi-modal (e.g., illustrations, movies) descriptions; linking to other resources; and
integrating definitions of term. To improve users’ health knowledge, the accuracy of
the information, and the adoption of consumer health information systems, it is
encouraged that usability methods be applied with an added focus on health literacy.
References
[1] P. C. Tang, J. S. Ash, D. W. Bates, J. M. Overhage, D. Z. Sands. Personal health records: definitions,
benefits, and strategies for overcoming barriers to adoption. Journal of the American Medical
Informatics Association : JAMIA 13(2) (2006), 121-126. doi: 10.1197/jamia.M2025
[2] D. A. Jones, J. P. Shipman, D. A. Plaut, C.R. Selden. Characteristics of personal health records:
findings of the medical library association/national library of medicine joint electronic personal health
record task force. Journal of the Medical Library Association: JMLA 98(3) (2010), 243-249. doi:
10.3163/1536-5050.98.3.013
[3] N. Segall, J. G. Saville, P. L'engle, B. Carlson, M. C. Wright, K. Schulman, J. E. Tcheng. Usability
evaluation of a personal health record. AMIA Annu Symp Proc (2011), 1233-42.
[4] L. Wuerdeman, L. Volk, L. Pizziferri, R. Tsurikova, et al. How accurate is information that patients
contribute to their electronic health record? AMIA Annu Symp Proc (2011), 834–838.
[5] S. C. Ratzan, R. M. Parker. Introduction. In C. R. Selden, M. Zorn, S. C. Ratzan, R. M. Parker, (Eds.),
National library of medicine current bibliographies in medicine: Health literacy. NLM Pub. No. CBM
2000-1. National Institutes of Health, U.S. Department of Health and Human Services, Bethesda,
2000.
[6] C. C. Cutilli. Health literacy: what you need to know. Orthopaedic nursing / National Association of
Orthopaedic Nurses 24(3) (2005) 227-233. doi: 10.1097/00006416-200505000-00014
[7] C. Speros. Health literacy: Concept analysis. Journal of Advanced Nursing 50(6) (2005), 633-640.
doi: 10.1111/j.1365-2648.2005.03448.x
[8] A. W. Kushniruk, V.L. Patel. Cognitive and usability engineering methods for the evaluation of clinical
information systems. Journal of Biomedical Informatics 37(1) (2004), 56-76. doi:
10.1016/j.jbi.2004.01.003
[9] M. I. Kim, K. B. Johnson. Personal health records: Evaluation of functionality and utility. Journal of
the American Medical Informatics Association: JAMIA 9(2) (2002), 171-180. doi:
10.1197/jamia.M0978
[10] J. J. Cimino JJ, V. L. Patel, A. W. Kushniruk. The patient clinical information system (PatCIS):
Technical solutions for and experience with giving patients access to their electronic medical records.
International Journal of Medical Informatics 68(1-3) (2002), 113-127. doi: 10.1016/S1386-
5056(02)00070-9
[11] Usability.gov: Your guide for developing usable & useful Web sites. U.S. Department of Health &
Human Services. Accessed October 18, 2012 from
http://www.usability.gov/methods/test_refine/heuristic.html
[12] Nielsen, J. Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods,
pp. 25-62. John Wiley & Sons, New York, 1994.
Patient Safety, Medical Errors &
Quality Management
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-189
Clinician Variations in Data Trust and Use

1
Karen L. COURTNEY,a, Yun JIANG,b Scott R. BEACH, b Judith T. MATTHEWS,b
Richard SCHULZ b
a
School of Health Information Science, University of Victoria, Victoria, BC, Canada
b
University of Pittsburgh, Pittsburgh, PA, USA
Abstract. This study examined how variations in the source and type of patient
health data affected health care providers’ perceptions of the trustworthiness and
usefulness of the data. Overall, respondents (n=107) reported moderate to high
levels of trust and usefulness of health status data from all sources. Technology
sources were rated as more trustworthy than traditional, non-technology sources
(paired t = -2.84, p <0.006). However, there was no significant difference between
technology sources and non-technology sources (paired t = -1.63, p <0.108) in
perceived usefulness for clinical decision making.
Keywords. Technology Assessment, Biomedical; Decision Making; Trust
Introduction
Little is known about how health care providers value and subsequently use external or
remotely acquired data in their clinical decision making. Existing literature has focused
on the process of exchanging information across care settings, with little attention to
the perceived value of sharing data or the perceived value of data acquired from remote
sources [1]. Recent models in data provenance suggest that the lineage or tracing of the
source of the data reflects on the quality and trustworthiness of the data [2]. Indeed,
Simborg [3] notes that health care providers need to consider the source of the data in
order to evaluate its validity. While data quality can vary in any given instance [4, 5],
clinician values or generalized attitudes toward remotely acquired data could be just as
important as actual quality variance. Negative attitudes may lead clinicians to disregard
or discount non-traditional sources or unique types of health data.
With the dramatic expansion of technology use in health care, clinicians are
bombarded with clinical data, some of which may be perceived to be of limited or
unknown value and therefore contributing to the “noise” of clinical decision making [6].
Despite this overabundance, clinicians may decline to utilize a system that fails to
provide the “right” data [7]. New technological applications outside of traditional
health care settings are delivering novel types of data. Though these data are intriguing
and promising, neither researchers nor clinicians yet know their clinical value. For
example, motion sensor data gathered at TigerPlace, a smart home-enabled assistive
living facility, are being used to create activity density maps for residents [8]. While
the comparison of activity density maps with known clinical changes are intriguing,
the clinical utility of such data is still unknown.[8]. Uncertainty in intended use and
lack of experience with these new sources and types of data may heavily influence the
1
Corresponding author: Karen Courtney, email: [email protected]
190 K.L. Courtney et al. / Clinician Variations in Data Trust and Use
perceived value and subsequent adoption of novel data.[9]. Without understanding the
attitudes and behaviors of clinicians, we may design and implement clinical data
collection systems that health care providers neither trust nor use. In this study we
examined how variations in the source and type of data and prior experience with
health information technology affected health care providers’ perceptions of the
trustworthiness and usefulness of patient data.
1. Methods
1.1. Data Collection
Following Institutional Review Board approval (PRO07080326), the Quality of Life

Technology Engineering Research Center conducted the Health Care Provider Data
Trustworthiness Survey. The self-administered survey was distributed to health care
providers attending a geriatric care conference. Approximately 469 clinicians attended
the conference. Among the 107 respondents (22.8% response rate), 39 were physicians,
32 were registered nurses, 22 were nurse practitioners or physician assistants, 12 were
other care providers and 2 did not indicate their professional background. This survey
required five to ten minutes to complete.
The survey consisted of two main sections of potential data sources. The first
section explored traditional sources of data, including primary care providers,
specialists, home health agencies, nursing home or assisted living facilities, community
health screenings, patients, and family members of patients. The second section of the
survey focused on information technology-mediated data sources, including telehealth
monitors, wearable monitors, environmental monitors and other monitors. Other
monitor data sources were defined as “technologies that may not have traditionally
been thought of as health sensors but could generate and share information that may be
relevant for health care providers. An example might be longitudinal performance on
the Wii™ balance board.”
For each traditional data source there were three to six specified types of data such
as vital signs, physical assessment, psychological assessment, images or pathology
reports, diagnoses, and treatment plans. For each health technology monitor, the
specified data types varied depending on their monitoring characteristics. For example,
the data captured by wearable monitors included emergency signals, vital signs or
electrocardiograms, and activity or exercise. Additionally, respondents were asked if
they had ever received data from each of these technology sources.
For each data source /data type combination respondents rated the trustworthiness
and usefulness of the data and indicated whether they had experience with such data.
Respondents were asked “In general, how trustworthy are data from these
sources?” The trustworthiness scores ranged from 1 to 10, with “1” for “completely
untrustworthy” and “10” for “completely trustworthy.” To capture perceived
usefulness, clinicians were asked of each data source “How useful are these data in
your clinical decision making?” The four-point likert scale of response options
included Not at all (coded as 1); A little (coded as 2); Moderately (coded as 3); and
Very Much (coded as 4). The survey instrument is available upon request.
K.L. Courtney et al. / Clinician Variations in Data Trust and Use 191
1.2. Data Analysis

Survey data were analyzed using SPSS Statistics 17.0. There was high internal
consistency across specific types of data (e.g. vital signs, psychological assessment,
treatment plans, etc….) within each source, and thus these were combined for analysis.
Internal data consistency within each data source and between data sources was
measured by Cronbach’s alpha and ranged from 0.80 to 0.96. We examined
trustworthiness and usefulness ratings for both individual data sources and for broader
combined categories of sources (e.g., technology vs. non-technology sources).
Descriptive statistics were used to compute mean trustworthiness and usefulness scores.
Paired t-tests were used to compare trustworthiness and usefulness between data
sources, specifically between non-technology sources and technology sources in
general and within each specific technology source. All significance tests were
evaluated at »< 0.05 with two-sided tests. Non-technology sources were aggregated
into the following pairings based on high correlations among perceived trustworthiness
(r between 0.649 and 0.857, p < 0.000): primary care providers and specialists; home
health and nursing home or assisted living facilities; and patients and family members.
2. Results
2.1. Technology Exposure vs. Non-Exposure
Exposure to individual types of data across data sources varied widely, ranging from
9.5% for exercise or social contact data from other monitor sources to 40.4% for
physiologic data captured by telehealth monitor sources. If a respondent indicated
having received any type of data within a technology data source in the past, we
considered the respondent to have prior exposure to that particular data source. In our
sample, 57.9% had been exposed to at least one type of data from any technology.
Prior exposure varied by data source. Overall, more respondents reported exposure to
data from telehealth monitor sources (38.3%) and wearable monitor sources (37.4%)
than other monitors (22.4%) or environmental monitors (18.7%). Prior exposure to a
particular technology source did not significantly influence respondents’ perceptions of
trustworthiness or clinical decision making usefulness of data from that source.
2.2. Trustworthiness
Respondents indicated a moderate to high level of trust in data from varying sources
(range M = 6.22-8.39). Among the traditional, non-technology data sources, the
specialist data source (M = 8.34 ± 1.41 SD) had the highest trustworthiness rating and
the wearable monitor data source (M = 8.39 ± 1.34 SD) had the highest trustworthiness
rating among the technology sources and overall. The community health screening
data source (M = 6.22 ± 1.93 SD) ranked lowest for trustworthiness overall and among
the non-technology sources. Among technology sources, other monitor data sources
(M = 7.08 ± 1.81 SD) held the lowest rating for trustworthiness. When data sources
were aggregated into two broad categories, technology sources (M = 7.75 ± 1.38 SD)
were rated as significantly more trustworthy than traditional, non-technology sources
(M =7.37 ± 1.24 SD) (paired t = -2.84, p <0.006), although the absolute difference in
ratings was relatively small. To explore whether this finding was still true without the
influence of community health screening data sources (the lowest rated source) on the
non-technology data sources, we compared technology sources against each of the
three broad pairings (primary care provider-specialist, home health- nursing home, and
patient-family member). Technology sources were rated as more trustworthy than
the data sources in the home health-nursing home comparison (M =7.08 ± 1.54 SD)
(paired t = -4.25, p <0.000), the patient-family member comparison (M =7.41 ± 1.54
SD) (paired t = -2.09, p <0.039) but not the primary care provider- specialist
comparison (M =8.22 ± 1.30 SD) (paired t = 2.99, p <0.004).
2.3. Usefulness
With few exceptions, respondents considered data collected by various sources to be at

least “moderately” useful in clinical decision making (range M = 2.45-3.49). For non-
technology sources, only nursing homes (M = 2.91± 0.75 SD) and community health
screenings were rated (M = 2.45 ± 0.87 SD) between “a little” to “moderately” useful.”
For technology sources, only other monitors received a mean rating (M = 2.81 ±
0.81SD) that suggested they were less than moderately useful in clinical decision
making. When aggregated into broad categories, there was no significant difference
between technology sources (M = 3.13 ± 0.62 SD) and traditional, non-technology
sources (M =3.02 ± 0.50 SD) (paired t = -1.63, p <0.108) in perceived usefulness in
clinical decision making. As with the trustworthiness data, we examined technology
sources against each of the three broad comparison groups (primary care provider-
specialist, home health-nursing home, and patient-family member). Technology
sources were rated significantly more useful in clinical decision making than the data
obtained from home health-nursing home (M = 2.92 ± 0.65 SD) (paired t = -2.71,
p<0.008), but significantly less useful than data from primary care provider- specialist
sources (M = 3.42 ± 0.59 SD) (paired t = 5.25, p <0.000). No significant difference
was found between technology sources and the patient-family member pairing (M =
3.22 ± 0.59 SD) (paired t = -1.45, p <0.152).
2.4. Relationship between Perceived Trustworthiness and Perceived Usefulness

inClinical Decision Making
With one exception, there was a significant, moderate positive relationship between
perceived trustworthiness of the data and perceived usefulness of the data in clinical
decision making across data sources (r = 0.363 – 0.646, p < 0.001). The weakest
association between perceived trustworthiness and perceived clinical decision
usefulness was for specialist-sourced data; whereas the strongest was for community
health screening-sourced data. A positive, moderate association was found between
perceived trustworthiness and usefulness when traditional, non-technology data sources
were aggregated (r = 0.494, p < 0.000). This association was considerably higher when
technology sources were aggregated (r = 0.702, p < 0.000).
3. Discussion
Of concern to engineers, informaticists and clinicians alike are factors that influence
the adoption and continued use of optimal health information technologies by health
care providers. Although anecdotal evidence has suggested that the provenance (data
source and process used to create the data) of health data might affect the clinician’s
perception of it, no studies to date have directly assessed clinicians’ perceptions of the
K.L. Courtney et al. / Clinician Variations in Data Trust and Use 193
trustworthiness and usefulness of data by source and type. As expected, we found that
perceived trustworthiness and usefulness are moderately correlated. Since our study
was conducted, other work has been published that also supports the relationship
between perceived usefulness and perceived information quality. [10]. More
importantly, our findings show that technology data sources were perceived at least as
favorably as data from traditional, non-technology, sources. This finding supports the
movement in health care towards more technology based sources of data. Ironically
this technology trend also includes data from outside traditional health care facilities
and may rely heavily on patients or family members as primary data providers.
This trend will ultimately raise other challenges. Some challenges may be cultural
such as how to integrate technology data into practitioner workflow. Other challenges
may be more technological in nature such as how to integrate new data sources into
existing health information systems. Inclusion of data provenance metadata alone
within electronic health records will require careful consideration of affected
terminologies and standards.
When developing the study, our team was concerned that clinicians may have had
little prior exposure to data from the technology sources. As a result, we added an
exposure question for each technology data source. Our results showed that prior
exposure to any of the technology data sources did not affect perceptions of
trustworthiness or clinical utility. Further research should attempt a more fine grained
assessment of prior experience. This study was limited by its small, non-representative
sample of health care providers. Some of the respondents may have also been pre-
disposed with positive attitudes towards technology in general as they were attending a
geriatric medicine update conference with an optional concurrent technology workshop.
This investigation is a critical first step in identifying differences that exist in
clinicians’ perceptions of data based on source. Understanding how information
technology as the delivery mechanism may change the perceived trustworthiness or
clinical decision making utility merits further inquiry.
References
[1] Crooks VA, Agarwal G. What are the roles involved in establishing and maintaining informational
continuity of care within family practice? A systematic review. BMC Fam Pract. 2008;9:65.
[2] de Lusignan S, Liaw ST, Krause P, Curcin V, Vicente MT, Michalakidis G, et al. Key Concepts to
Assess the Readiness of Data for International Research: Data Quality, Lineage and Provenance,
Extraction and Processing Errors, Traceability, and Curation. Contribution of the IMIA Primary Health
Care Informatics Working Group. Yearb Med Inform. 2011;6:112-20.
[3] Simborg DW. Consumer empowerment versus consumer populism in healthcare IT. J Am Med
Inform Assoc. 2010;17:370-2.
[4] Porter SC, Forbes P, Manzi S, Kalish LA. Patients providing the answers: narrowing the gap in
data quality for emergency care. Qual Saf Health Care. 2010;19:e34.
[5] Shaw A, Somerset M, Dunn L, Peters T, Faulkner A, Sharp D. Can we trust the quality of routine
hospital outpatient information in the UK? Validating outpatient data from the patient
administration system (PAS). J Health Serv Res Policy. 1998;3:203-6.
[6] Markman M. Information overload in oncology practice and its potential negative impact on the
delivery of optimal patient care. Curr Oncol Rep. 2011;13:249-51.
[7] Hincapie AL, Warholak TL, Murcko AC, Slack M, Malone DC. Physicians' opinions of a health
information exchange. J Am Med Inform Assoc. 2011;18:60-5.
[8] Rantz MJ, Skubic M, Miller SJ. Using sensor technology to augment traditional healthcare.
Annual International Conference of the IEEE Engineering in Medicine and Biology Society
20092009. p. 6159-62.
[9] Shabati I, Leshno M, Blondheim O, Kornbluth J. The value of information for decision-making in the
healthcare environment. Stud Technol Inform. 2007;127:91-7.
[10] Hsiao JL, Chang HC, Chen RF. A study of factors affecting acceptance of hospital information
systems: A nursing perspective. J Nurs Res. 2011;19:150-60.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-195
Design and Implementation of Synoptic

Operative Report Template Using
Interoperable Standards
Wilfred BONNEYa,1, Sean CHRISTIEa, Grace PATERSONb, Ginette THIBAULT-
HALMANa, Areej ALSULAIMAN c
a
Division of Neurosurgery, Capital Health, Halifax, Nova Scotia, Canada, b Medical
Informatics, Dalhousie University, Halifax, Nova Scotia, Canada, c Dalhousie
University, Halifax, Nova Scotia, Canada
Abstract. The increasing use of synoptic operative reports in clinical settings

represents a major milestone in the advancement of health information technology.
Synoptic operative report templates enable clinicians to capture and display
succinct clinical information in a standardized and logical manner. Synoptic
operative report templates also provide the optimum goal of enriching
personalized health information of a given patient at the point of care so as to
support the exchange of clinical information across the continuum of multiple
healthcare providers. However, most of the available synoptic operative report
templates in many clinical settings do not incorporate interoperable standards in
their design and implementation. This paper proposes a novice template (i.e.,
eSOR-SCI) that uses interoperable standards for its design and implementation.
Keywords. Synoptic operative report, interoperable standards, HL7 V3 CDA, HL7

V3 CCD, SNOMED-CT, LOINC
Introduction
Synoptic operative reports are simply structured or discrete medical reports that
provide a snapshot of patient encounter with a clinician at the point of care.
Specifically, synoptic operative reports often have a direct and critical role in patient
care by providing the necessary information needed to understand, interpret, and
communicate a patient’s surgical history [1]. Synoptic operative reports have emerged
as the gold standard for capturing and displaying structured surgical data in clinical
settings [1-5]. Citing the work of Temple et al. [6], Paterson, Swain, Christie, Thibault-
Halman, and Behzadi [7] noted that the replacement of traditional narrative with a
synoptic operative report has proven to be an effective strategy in improving surgical
data for the knowledge transfer process. This approach enables the capture of discrete
data items and transforms a narrative operative report that is qualitative in nature to a
quantitative one that can be aggregated to generate information and knowledge [2].
Electronic synoptic operative reports have replaced dictated reports at many
healthcare institutions because users could demonstrate improvements in reliability,
1
Corresponding Author: Wilfred Bonney, Division of Neurosurgery, Capital District Health Authority,
Halifax, NS, Canada B3H 3A7; E-mail: [email protected]
196 W. Bonney et al. / Design and Implementation of Synoptic Operative Report Template
completeness, and timeliness with this alternative. Electronic synoptic operative reports
use “computer-based forms to describe the findings and important steps of an operation,
based on predefined templates for individual procedures” [1, p. 308]. In other words,
electronic synoptic operative reports are dependent upon predefined templates for
individual procedures. Such templates are based on synoptic methodology, which on its
own refers to the creation of a library of keywords and phrases to show a preferred way
to communicate a clinical concept in an operative report.
Electronic synoptic operative templates support the creation of clinical documents
that are of higher quality than those produced by the same individual using dictated
narrative reporting method [8]. The template serves as a prompt for pertinent
information, which could be reused to support the information needs of researchers and
administrators. In accordance with the clinical documentation standards set by the
Joint Commission Accreditation of Healthcare Organizations (JCAHO) and
Accreditation Association for Ambulatory Health Care (AAAHC), synoptic operative
report templates contain sufficient information to: identify the patient; support the
diagnosis; justify the treatment; document the postoperative course and results; and
promote continuity of care [1,9]. Thus, the continuing use of electronic templates in
medical practice promotes health policy informatics and secondary data use.
1. Related Studies
Several synoptic operative templates have been proposed and used by many vendors
and researchers in the healthcare domain. Noticeable amongst them is the WebSMR
template proposed and developed by the Alberta Cancer Surgery Working Group. The
Working Group developed and implemented a web-based synoptic surgical medical
record, known as the Alberta WebSMR, for rectal cancer resection [1]. Edhemovic,
Temple, de Gara, and Stuart [3] reported that following the implementation of a rectal
cancer template, the synoptic report captured 99% of the required data elements versus
45.9% captured via a dictated narrative report. This showed that the science of surgical
technique could be better measured using a synoptic reporting method. The synoptic
checklist approach has consistently shown superior results to narrative reports for
completeness [4]. The WebSMR did not, however, incorporate major interoperable
standards in the system design.
In a related study, Paterson and Soroka [10] developed and implemented Health
Level Seven (HL7) Version 3 Clinical Document Architecture (CDA) Release Two
(R2) template for capturing discharge summaries for chronic kidney disease patients.
The template was designed using Microsoft InfoPath and HL7 V3 CDA R2
specifications. Even though the study by Paterson and Soroka [10] utilized
interoperable standards in the system design and implementation, they did not focus on
synoptic operative reports. The proposed electronic Synoptic Operative Report-Spinal
Cord Injury (eSOR-SCI) template not only focuses on synoptic operative report but
also uses interoperable standards in its design and implementation.
2. Methods
The methods were primarily based on the principles of health interoperability using
HL7 V3 CDA R2, Systematized NOmenclature of MEDicine - Clinical Terms
W. Bonney et al. / Design and Implementation of Synoptic Operative Report Template 197
(SNOMED-CT), and Logical Observation Identifiers Names and Codes (LOINC) [12].
Drawing upon the work of Rector, Qamar, and Marley [11], Benson [12] asserted that
electronic information systems operate at two different levels: Model of Use and Model
of Meaning. Whereas Model of Use is defined as the human interface that describes
how the system is actually used, Model of Meaning is defined as the representation for
reporting and analysis that supports data processing and reasoning [12]. These two
distinct levels of system operation are necessary because each “model of use needs to
be convertible into a model of meaning to make it computable” [12, p. 221]. This study
extended the work of Rector et al. [11] by adding another level (i.e., Model of Design)
to the methodology to describe how electronic information systems are designed.
The methodology, therefore, involved the design and use of HL7 V3 CDA R2 and
SNOMED-CT together in the context of synoptic operative report for patients with
spinal cord injury. The eSOR-SCI template is the model of use in that it enables data to
be captured and displayed by clinicians after the completion of each surgical procedure.
The eSOR-SCI template was implemented using a model of meaning methodology to
ensure that data can be aggregated and analyzed for research purposes. The binding of
the SNOMED-CT with the HL7 V3 CDA R2 template structure depicts how the model
of meaning was achieved. In the case of the eSOR-SCI template, the clinical document
is the synoptic operative report generated by the system.
3. Results
The implementation of the eSOR-SCI template followed three processes: Model of

Design, Model of Use, and Model of Meaning.
3.1. Model of Design
The model of design represents the design architecture of the eSOR-SCI template. The
design architecture components of the eSOR-SCI template consisted of a template
based on the Rick Hansen Spinal Cord Injury Registry (RHSCIR) Forms, Interoperable
Standards; and Microsoft Infopath.
3.1.1. RHSCIR Forms
The RHSCIR forms for diagnoses and procedures provided the minimum core data
elements for the eSOR-SCI template. Both the diagnoses and procedures forms were
the source of the template sections. Based on secondary data use requirements, the data
elements provided the content to populate the diagnoses and the procedures as per
RHSCIR diagnosis and procedure forms. These data elements were represented using
HL7 V3 CDA R2 specifications to bind to the template hosted on the InfoPath platform.
3.1.2. Interoperable Standards
The eSOR-SCI template used a congruence of interoperable standards including HL7

V3 CDA R2, HL7 V3 Continuity of Care Document (CCD), SNOMED-CT, and
LOINC in its design and implementation [12]. These standards support the exchange of
clinical information across the continuum of multiple providers. The implementation of
these standards was facilitated by the use of published implementation guides. In the
context of eSOR-SCI template, the interoperable standards were used to capture,
represent, and store the minimum core datasets for the template and made available in
XML format that could be uploaded to proprietary document management systems.
The use of the interoperable standards was necessary to improve the data quality of the
template thereby facilitating secondary data use [13].
3.1.3. Microsoft InfoPath
Microsoft InfoPath was used to build the eSOR-SCI template. The InfoPath platform
enables users of the system to enter patient’s surgical information and then view and
print the resulting report from the interface. The InfoPath platform also supported the
binding of the template with the modified XML schemas from the HL7 V3 CDA R2
specifications. InfoPath is ubiquitous and is already installed in many computers used
by the clinicians at Capital District Health Authority (CDHA) in Halifax, Nova Scotia.
3.2. Model of Use
The model of use comprises of the way that data are captured and displayed in the
eSOR-SCI template. The human interface of the template is made up of two views:
data entry view and report view. The template also consists of several sections
including Patient Information, Surgeon and Anaesthesiologist Information, Procedure
Date and Time, Pre-operative Diagnosis, Indications, Pre-operative Clinical Status (i.e.,
ASIA Scale, ASA Grade, and Glasgow Coma Scale), Co-morbidities, Post-operative
Diagnosis, Anaesthesia Information, Surgical Procedures, Implants, Surgery
Descriptions/Findings, Surgical Drains, Bone Graft, Fluid Intake/Output, Discharge
Disposition, and Author of Operative Report. A user manual has been developed to
assist clinicians and residents in understanding the appropriate use of the template.
3.3. Model of Meaning
In facilitating the model of meaning of the eSOR-SCI template, several concepts and
terminologies, utilized by the template, were encoded to SNOMED-CT expressions
with the uttermost goal of making the stored data reusable to support information needs
of researchers and administrators. Consequently, 391 out of the 402 clinical data
elements presented in the electronic template were encoded to SNOMED CT
expressions, representing 97.26% of all clinical data elements presented in the eSOR-
SCI template. Approximately 2.74% of the clinical data elements were considered to be
difficult to encode using the SNOMED-CT expressions. Table 1 shows the SNOMED-
CT compositional grammar for sample three procedures (i.e., Decompression
laminoplasty, Suboccipital craniectomy, and Facet Rhizotomy) adapted from the
RHSCIR form for procedures.
W. Bonney et al. / Design and Implementation of Synoptic Operative Report Template 199
Table 1. SNOMED-CT compositional grammar for three procedures adapted from the RHSCIR form for
procedures.
Procedures SNOMED-CT Compositional Grammar
Decompression laminoplasty 438374004|laminoplasty|+410756002|surgical decompression|
Suboccipital craniectomy 36910002|craniectomy|+260623005|suboccipital approach|
Facet Rhizotomy 19214009|incision of spinal nerve root|:363704007|procedure site|

= 89836005 |entire zygapophyseal joint|
4. Conclusion
This research project aims to improve the quality of data for primary and secondary use
through implementing an electronic template for synoptic operative reports for spinal
cord injury patients. Success in this project would improve the quality of data, which is
already collected for other research projects, such as the RHSCIR, as well as for the
Canadian Institute for Health Information Discharge Abstract Database (CIHI DAD),
which collects health information from institutions across the country. The eSOR-SCI
template has been implemented and is being piloted by the residents at the Division of
Neurosurgery, CDHA in Halifax, Nova Scotia. The pilot study will gather data on
usefulness, usability, and use of the template for Neurosurgery operative reports.
It is expected that the synoptic reporting will enhance the transfer of information,
which will lead to improvements in the clinical data captured on spinal cord injury
operations for secondary data use (i.e., RHSCIR and CIHI DAD). Moreover, it is
anticipated that the eSOR-SCI template would assist surgeons in quickly completing
clinical documentation; improving data quality to support secondary data use; and
generating standardized reports that are easier to read and interpret. Success is achieved
if the implemented eSOR-SCI template is usable, useful, and used [14].
References
[1] Park J, Pillarisetty VG, Brennan MF, Jarnagin WR, D’Angelica MI, DeMatteo RP, Coit DG, Janakos M,
Allen PJ. Electronic synoptic operative reporting: Assessing the reliability and completeness of
synoptic reports for pancreatic resection. J Am Coll Surg. 2010; 211: 308-315.
[2] Urquhart R, Grunfeld E, Porter GA. Synoptic reporting and the quality of cancer care: A review of
evidence and Canadian initiatives. Oncol Exch. 2009; 8(1): 28-31.
[3] Edhemovic I, Temple WJ, de Gara CJ, Stuart GCE. The computer synoptic operative report: A leap
forward in the science of surgery. Ann Surg Oncol. 2004 Oct; 11(10): 941-947.
[4] Mack LA, Dabbs K, Temple WJ. Synoptic operative record for point of care outcomes: A leap forward
in knowledge translation. Eur J Surg Oncol. 2010 Sep; 36 (Suppl 1): S44-S49.
[5] Vergis A, Gillman L, Minor S, Taylor M, Park J. Structured assessment format for evaluating operative
reports in general surgery. Am J Surg. 2008 Jan; 195(1): 24-29.
[6] Temple WJ, Francis WP, Tamano E, Dabbs K, Mack LA, Fields A. Synoptic surgical reporting for
breast cancer surgery: An innovation in knowledge translation. Am J Surg. 2010; 199(6): 770-775.
[7] Paterson G, Swain D, Christie S, Thibault-Halman G, Behzadi F. Evaluation of SNOMED CT for the
spinal cord injury domain. In: proceedings of AHIC 2012: Advances in Health Informatics Conference.
Toronto, Ontario, Canada. 2012.
[8] Paterson G. Boundary infostructures for chronic disease: Constructing infostructures to bridge
communities of practice. Saarbrücken, Germany: VDM Verlag; 2008.
[9] Abdelhak M, Grostick S, Hanken MA, Jacobs EB. Health information: Management of a strategic
resource. 4th ed. St. Louis, Missouri: Saunders; 2007.
[10] Paterson GI, Soroka SD. Formative evaluation of the clinical pragmatic attributes of components
chosen for a boundary infostructure. In: Abidi R, Bath P, Keselj V, editors. Proceedings of iSHIMR
2006: The 11th International Symposium on Health Information Management Research; Halifax, NS,
Canada: Faculty of Computer Science, Dalhousie University, 2006, p. 375-385.
[11] Rector AL, Qamar R, Marley T. Binding ontologies and coding systems to electronic health records
and messages. Appl. Ontol. 2009 Jan; 4(1): 51-69.
[12] Benson T. Principles of health interoperability HL7 and SNOMED. London, United Kingdom:
Springer-Verlag; 2010.
[13] Bonney W. Improving the quality of primary care data with interoperable standards. In: Traver V, Fred
A, Filipe J, Gamboa H, editors. Proceedings of HEALTHINF 2011: International Conference on Health
Informatics; Rome, Italy: SciTePress; 2011. p. 446-450.
[14] Grant AM, Moshyk AM, Kushniruk A, Moehr JR. Reflections on an arranged marriage between
Bioinformatics and Health Informatics. Methods Inf Med. 2003; 42: 116–120.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-201
Processing Medical Reports to

Automatically Populate Ontologies
Lu´s BORREGO a and Paulo QUARESMA b,1,
a
University of Évora, Évora, Portugal,
b
University of Évora, Évora, Portugal
Abstract Medical reports are, quite often, written and stored in computer systems in
a non-structured free text form. As a consequence, the information contained in
these reports is not easily available and it is not possible to take it into account by
medical decision support systems. We propose a methodology to automatically
process and analyze medical reports, identifying concepts and their instances, and
populating a new ontology. This methodology is based in natural language
processing techniques using linguistic and statistical information. The proposed
system was applied successfully to a set of medical reports from the Veterinary
Hospital of the University of Évora.
Keywords. Ontologies, Information Extraction, Natural Language Processing,
Machine Learning, Semantic Web
Introduction
In the last years complex computational systems have been developed in the health
domain. The integration of information is one of the major challenges in health
informatics. Medical reports are often made at several institutions, which use
proprietary vocabularies and, if this information is not properly prepared and presented,
can cause important failures of interpretation. To solve this problem one can resort to
the use of ontologies, which allow to design and model concepts and relations correctly
[1]. Through the languages of the Semantic Web ontologies can be represented by
adding semantic annotations. To facilitate the population of the ontology, automated
processes or semi-automatic ones can be applied. Text mining techniques are applied,
using several techniques, such as natural language processing and machine learning [2].
Thus, through the creation of ontologies and its population, it is possible to represent
medical information, allowing institutions to improve patient health care systems.
The main objective of this work is to create an ontology that allows us to represent
information expressed in medical reports and to create automatic mechanisms that ex-
tract information and carry out a population of this ontology. Therefore, we will obtain
a medical knowledge base, making possible to access it through the use of inference
engines and to develop more powerful health information systems.

1
Paulo Quaresma, [email protected]
202 L. Borrego and P. Quaresma / Processing Medical Reports to Automatically Populate Ontologies
1. Methodology
The proposed work has four sequential phases.
1.1. Analysis of Documents
In the context of this work, we used a set of 590 histopathological reports, which were
supplied by the Veterinary Hospital of the University of Évora. The reports were
analyzed and all the relevant fields identified.
1.2. Creation of Ontology
From the analysis of documents it was possible to create an ontology covering the do-
main of histopathological reports. Among the various existing languages to represent
ontologies, the OWL language was chosen because it is more efficient on the Web than
XML, RDF and RDFS, it is a W3C recommendation and, also, because it has a formal
semantics definition.
As methodology to create the ontology we followed the proposal of Noy and
McGuiness [3], from the Stanford University, which defines the sequence of steps
needed to create the ontology and to model the knowledge.
The ontology was built from scratch due to the shortage of work developed in this
area, most likely due to the great complexity of the domain and the fact that Semantic
Web languages are a relatively new area. First, we analyzed all medical reports and
extracted all terms. Then, with the help of the health professionals classes, their
properties and restrictions were created. We normalized all concepts and, when
possible, we used the UMLS terminology (http://www.nlm.nih.gov/research/umls/),
aiming to facilitate the interoperability of the system. The ontology was developed
using the Protege tool (http://protege.stanford.edu/). The overall scheme of the domain
is composed of twelve classes related to each other, as shown in Figure 1.
1.3. Automatic Labeling of the Reports
In this phase of work we used Minorthird (http://minorthird.sourceforge.net),

software that was developed in Java and includes some machine learning and
information extraction algorithms, and allows labeling text automatically [4]. To
automatically label the reports it is necessary to have first a set of reports manually
labeled. These will serve as the training set for the machine learning algorithms. From
the analysis of the ontology we identified 29 major concepts, varying from simple
entities, such as “date of the exam”, to complex ones, such as “description of the
exam”.

L. Borrego and P. Quaresma / Processing Medical Reports to Automatically Populate Ontologies 203
Figure 1. Modeling of the Global Schema Domain

Seventy-four reports were labeled with the help of health professionals
(approximately 12.5% of the size of the corpus), value that we found enough to get
good results. After analyzing the final results, we verified that the algorithm that
produces better results is the SVM- CMMLearner, a probabilistic support vector
machine to learn conditional Markov models. The results obtained by this algorithm are
quite satisfactory: for 21 of 29 labels the algorithm identified and extracted above 90%
of the information. For some text labels the results were not so good, presenting,
however, percentages above 75%. From these results we created a classifier
SVMCMMLearner for each label of the training set, allowing all reports to be
automatically labeled by Minorthird in XML format.
1.4. Ontology Population
The last phase of the work concerns the extraction of information from the XML
labeled medical reports, creating class instances in the ontology. For such, we
developed a Java application that extracts information from reports using the DOM
parser xml (http://www.w3schools.com/dom/dom_parser.asp), processing the
information contained in labels and creating the instances in the ontology using the
Jena framework (http://jena.sourceforge.net). All the information extracted from
“simple” fields, such as, “Report”, “Place”, “Technician”, “Responsible”, “Doctor”,
“Proprietary” and “Animal”, could be directly instantiated in the ontology. Regarding
the content of examinations and of the conclusion, the information needs to be
processed because it corresponds to a text with a set of anatomical entities, lesions and
observations related to each other. Thus, the text of each label is extracted and created a
vector.
For example, for the text “Foi enviado para ana´ lise um fragmento cutaˆ neo com
um no´ dulo cutaˆ neo ulcerado. Ao corte apresentava coloraça˜ o acinzentada./It
was sent to be analysed a cutaneous fragment with an ulcerated cutaneous nodule.
When cutting it showed a grayish colour”, we obtain the vector represented by Figure
2.
We developed a 4-step algorithm that processes this initial vector and transforms
it: 1) remove tags within tags; 2) create anatomical entity(ies) that will eliminate some
labels of the vector simplifying it; 3) create lesion(s) that will eliminate other labels of
the vector (“lesionName”, “type”, “size”, “color”, “degreeMalignancy”) simplifying it;
and 4) create relations between anatomical entities, lesions and observations.
The first phase of the transformation of the vector is fundamental because it is
important that instances with labels don’t appear in the ontology. For the previous

204 L. Borrego and P. Quaresma / Processing Medical Reports to Automatically Populate Ontologies
vector, the label ”type” within the label “lesionName” is removed adding a new
position of the vector, as shown in Figure 3.
Figure 2. Vector with the labels of the text of the paragraph
Figure 3. Vector after Removing Labels within Labels
The second and third phase of the proposed transformation of the vector is mainly to
facilitate, in first place the relationship between the labels occurring in the fourth phase,
and later the creation of instances in the ontology. Figure 4 corresponds to the
application of these two phases to the previous vector.
Figure 4. Vector after creating Anatomical Entity and Lesion Labels

Finally, the last phase of transformation of the vector is to create relations between tags.
Figure 5 represents the final processing stage for the previous vector.
Figure 5. Vector after creating Relations between Labels

With this approach it is possible to create the instances of the ontology corresponding to the

L. Borrego and P. Quaresma / Processing Medical Reports to Automatically Populate Ontologies 205
information extracted from the medical reports.
2. Results and Application of Ontology
After the creation of instances in the ontology, we used once again the Protege´
framework to evaluate the results and perform some tests. In total, for all reports of the
corpus, 7824 instances were created in the ontology. These are correctly structured
according with the texts. We also observed that all information labeled by Minorthird is
instantiated in the ontology and errors were not detected. The results are very positive
and achieve the objectives originally proposed.
Using the Minorthird plugin DL Query, it is possible to make queries and obtain results
directly from the created medical knowledge base. For instance, applying the query
“Which are the medical reports that have exams that have the lesion with the name
‘Furunculosis’?”, the system quickly provides the resulting instances, saving the time
from a manual search and allowing the results to be used in new searches.
3. Conclusion
This work aims to create and to populate ontologies as a way to acquire knowledge
from medical reports. We have used histopathological reports, written in a semi-
structured form and having textual fields. An ontology was created for this domain of
histopathological reports and heuristic and machine learning techniques were applied,
allowing to initially automatically label the reports and, at a later stage, to relate entities
and automatically populate the ontology. The results show that this is a promising
approach, allowing the automatic extraction of relevant information.
The use of information systems in the health domain is nowadays indispensable,
aiming to help professionals in health or patient care. Our work helps to support the
claim that the use of ontologies in medical reports can represent an important step in
this area, providing new and more powerful tools and supporting either investigators or
health professionals in their work.
References
[1] Quaresma P, Bratsas CH, Maglaveras N. A framework to describe problems and algorithms in medical
informatics via ontologies. In 4th European Symposium on Biomedical Engineering, Patras, Greece.
University of Patras, 2004.
[2] Feldman R. Sanger J. The Text Mining Handbook Advanced Approaches in Analyzing Unstructured
Data. Cambridge: Cambridge University Press, 2007.
[3] Noy N, McGuinness D. Ontology Development 101: A Guide to Creating Your First Ontology.
Stanford Knowledge Systems Laboratory Technical Report KSL-01-05 and Stanford Medical
Informatics Technical Report SMI-2001-0880, March 2001.
[4] Cohen W. Minorthird: Methods for Identifying Names and Ontological Relations in Text using
Heuristics for Inducing Regularities from Data. 2004. Available from: http://minorthird.sourceforge.net

Health Modeling
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-209
A Complex Adaptive Systems Perspective

of Health Information Technology
Implementation
Karim KESHAVJEEa,c,1, Craig KUZIEMSKY b, Karim VASSANJIc, Ahmad GHANYd
a
University of Victoria, Victoria, Canada, b University of Ottawa, c InfoClin Inc,
d
McMaster University
Abstract. Implementing health information technology (HIT) is a challenge

because of the complexity and multiple interactions that define HIT
implementation. Much of the research on HIT implementation is descriptive in
nature and has focused on distinct processes such as order entry or decision
support. These studies fail to take into account the underlying complexity of the
processes, people and settings that are typical of HIT implementations. Complex
adaptive systems (CAS) is a promising field that could elucidate the complexity
and non-linear interacting issues that are typical in HIT implementation. Initially
we sought new models that would enable us to better understand the complex
nature of HIT implementation, to proactively identify problem issues that could be
a precursor to unintended consequences and to develop new models and new
approaches to successful HIT implementations. Our investigation demonstrates
that CAS does not provide prediction, but forces us to rethink our HIT
implementation paradigms and question what we think we know. CAS provides
new ways to conceptualize HIT implementation and suggests new approaches to
increasing HIT implementation successes.
Keywords. Complex adaptive systems, interoperability, HIT implementation,

process redesign, simulation modeling.
Introduction
There is increasing investment in information technology for health care to enhance

healthcare delivery in North America and elsewhere [1]. To date the introduction of
health information technology (HIT) into clinical settings has been problematic
because of changes to care delivery workflows and unintended consequences. Several
studies have examined the role and nature of these consequences from the perspective
of HIT implementation [2-4]. Much of the current research on these consequences has
focused on the negative aspects of HIT implementation including technology-induced
errors [5], workarounds [4], power issues, workflow disruptions [2-5] and patient
safety concerns [4,6].
The existence of these so-called unintended consequences emphasizes that we
cannot look at HIT implementation as process centric (i.e. order entry or decision
support) in isolation from other processes. Rather we need to look at the overall system
that HIT is automating. Safe use of HIT requires a continuous monitoring of
1
Corresponding Author.
210 K. Keshavjee et al. / A CAS Perspective of Health Information Technology Implementation
implementation over the continuum of providers who provide care, the patients who
receive care, the organizational context in which care is provided, the tasks that are
associated with a process, the environment within which the organization exists and
other relevant processes and constraints [7].
Complex adaptive systems (CAS) are a means of looking at the complexity and
interacting issues that define HIT implementation [8]. CAS can help reframe the
thinking about HIT implementation and acknowledge the non-linearity of processes
and how behaviors and tasks emerge as a result of non-linearity [9].
Surprisingly, a search for complexity theory and CAS within the health informatics
literature yields a paucity of articles [8,9]. While these studies have suggested that HIT
implementation issues are characteristic of CAS, they have not specifically looked at
HIT implementation through the CAS lens.
The medication prescribing, dispensing and administration process is one of the
most common sources of medical errors due to the complexity of the process and the
significant potential for harm from potent medical interventions [11, 12]. Therefore it is
an excellent process to study from the perspective of CAS. In this paper we analyze
and link the prescribing process to CAS theory. This allows us to leverage the prolific
research being done in other disciplines for gaining insight into developing a
framework for HIT implementations.
1. Methods
We developed and used a use case of prescribing and dispensing in the community to
identify CAS elements. Figure 1 displays the use case.
We analyzed the prescribing and dispensing process using the lens of CAS. We
listed the features of CAS and identified instances of prescribing and dispensing which
fit those features. Our goal was to identify a mechanism or method to predict
unintended consequences and to predict areas of higher risk in HIT implementation.
Figure 1. Simplified medication prescribing, dispensing and administration process in the outpatient setting
K. Keshavjee et al. / A CAS Perspective of Health Information Technology Implementation 211
2. Results
Table 1 maps features of the medication prescribing process to features of CAS.
Table 1. Features of CAS and Medication Prescribing
Features of Complex Corresponding Features of Medication Prescribing Process

Adaptive Systems
Non-linear processes Changes in one area affect things in another area, but in non-linear and non-
intuitive ways. E.g., decreasing drug-drug interactions leads to increased
work for physicians. Some errors in prescribing are easily solved by
pharmacists; others require a phone call.
Emergent behavior Unintended consequences of HIT [2,3]. Workload shifts from pharmacist to
physician when e-prescribing is implemented.
Feedback loops Pharmacists phone physicians to confirm information. Physicians monitor

drug effects and side effects. Patients revert to their physician or pharmacist
if they face any issue with their medication.
Co-evolution When physicians start printing prescriptions, patients experience a decrease

in dispensing errors due to illegibility and pharmacists have to make fewer
calls back to physicians to clarify the prescription. Overall, this leads to
fewer phone-calls between providers.
Requisite Variety Each network of prescribers, dispensers and patients has its own unique
characteristics. There are no uniform methods for prescribing and dispensing.
Changes that work in one network do not work in other networks.
Connectivity Relationships and communication between physicians, patients and

pharmacists is very important to ensure patient safety. The prescription is an
important tool for communicating between prescribers and dispensers.
Simple Rules Prescriptions have a very small number of elements, yet they can have far-
reaching implications. The act of prescribing is made of several simple rules
(the contents and process of a prescription) but the end-result is an
information-rich document with far more complex interpretations and
implications for patient health.
Self-organization Pharmacists can monitor prescription problems and work with patients and
physicians to correct them. E.g., during the recent large scale switching from
one opioid to another in Ontario.
Non-discrete The prescribing process is regulated by licensing authorities and by

Boundaries legislation. These can have profound effects on prescribing and dispensing.
212 K. Keshavjee et al. / A CAS Perspective of Health Information Technology Implementation
A review of Table 1 quickly demonstrates the futility of trying to predict the path
and outcomes of an HIT implementation when there are large numbers of interacting
variables with non-linear relationships between them. Even a downturn in the
economy can cause a perturbation in an HIT implementation if strategic plans change
because of financial issues.
The key insight to emerge from our analysis is that HIT implementations are
inherently unpredictable and that unintended consequences are to be expected. Rather
than trying to predict and control an implementation, CAS theory points to an approach
that is more organic and iterative in nature: implement a small, minimally viable part of
the HIT and observe the results (non-linear processes, feedback loops). Measure the
impact of the implementation on key goals (i.e., develop new feedback mechanisms for
the new system, watch out for emergent behaviours). Listen to nay-sayers and
objectors, they may have useful advice to give and may be your best predictors of what
could go wrong (self-organization, feedback loops). Make course corrections and
adjustments (feedback, self-organization). Watch out for boundary issues such as
regulations impinging on workflow (non-discrete boundaries). Don’t expect that what
worked in one setting will necessarily work in another (requisite variety). Develop
new mechanisms for communication, especially for meta-communication
(connectivity). Implement some more. This approach is reminiscent of quality
improvement approaches such as the Plan-Do-Study-Act cycle which has been a
successful model for generating change in complex clinical situations and is also
reminiscent of the lean start-ups movement which promotes the idea of continuous
quality improvement with small, incremental changes.
3. Discussion
In this paper we advocate for using complex adaptive systems theory to study HIT
implementation issues and to make recommendations on how we can improve the HIT
implementation process. The real strength in complex adaptive systems is in the word
system. Healthcare processes are not disparate entities that exist in isolation. Rather
they are complex systems with multiple actors and parts that are not linear but are
defined by variation and dynamic evolution. Complex adaptive systems acknowledge
that complexity exists in a system because of the range of actors and the multiple ways
that they integrate with one another while engaging in system processes. Modeling
healthcare processes from the perspective of this complexity allows us to understand
the impact that implementing HIT will have a on a process such as how the burden of a
task may shift from one provider to another or how the process may evolve over time.
The end point of our analysis is that CAS does not provide prediction, but forces us
to rethink our HIT implementation paradigms and question what we think we know.
CAS provides us new ways to conceptualize HIT implementation and suggests new
approaches to increasing HIT implementation successes.
References
[1] Institute of Medicine. Crossing the quality chasm: a new health system for the twenty-first century.
2001. Washington.
K. Keshavjee et al. / A CAS Perspective of Health Information Technology Implementation 213
[2] Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care:
the nature of patient care information system-related error. Journal of the American Medical
Informatics Association 2004 (11); 104–112.
[3] Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of
unintended consequences related to computerized provider order entry. Journal of the America Medical
Informatics Association 2007 (14); 415– 423.
[4] Koppel R, Wetterneck T, Telles JL, Karsh BT. Workarounds to barcode medication administration
systems: their occurrences, causes, and threats to patient safety. Journal of the American Medical
Informatics Association 2008 15 (4); 408-423.
[5] Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability:
the relationship between usability problems and prescription errors when using a handheld application.
International Journal of Medical Informatics 2005 (74); 519-526.
[6] Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health
care – an interactive sociotechnical analysis. Journal of the American Medical Informatics Association
2007 14 (5); 542-549.
[7] Sittig DF, Classen D. Safe electronic health record use requires a comprehensive monitoring and
evaluation framework. Journal of the American Medical Aassociation 2010 303(5); 450-451.
[8] Westbrook JI, Braithwaite J, Georgiou A, Ampt A, Creswick N, Coiera E, Iedema R. Multimethod
evaluation of information and communication technologies in health in the context of wicked problems
and sociotechnical theory. Journal of the American Medical Informatics Association 2007 14; 746–755.
[9] Ellis B, Hebert SI. Complex adaptive systems (CAS): an overview of key elements, characteristics and
application to management theory. Informatics in Primary Care 2011 19; 33–37.
[10] Koppel R, Leonard C, Localio AR, Cohen A, Auten R, Strom BL. Identifying and quantifying
medication errors: evaluation of rapidly discontinued medication orders submitted to a computerized
physician order entry system. Journal of the American Medical Informatics Association 2008 15(4);
461-465.
[11] Phillips DP, Bredder CC. Morbidity and mortality from medical errors: an increasingly serious public
health problem. Annual Reviews of Public Health 2002 23; 135–150.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-214
A Systems Theory Classification of EMR

Hazards: Preliminary Results
Fieran MASON-BLAKLEYa1 and Jens WEBERa
a
Abstract. Jurisdictions in Canada, the US, the EU and Australia are struggling
with regulation of ever evolving software in medicine. Recently this discussion has
had a focus on electronic medical records (EMRs). There is a mountain of evi-
dence that EMRs have actualized potential to lead to the injury of patients through
the information they offer to facilitate care. We are undertaking a systematic re-
view of relevant literature in the field to uncover some of the latent hazards. We
hypothesize that this exploration, using a variation on Leveson's system theoretic
accidents models and processes (STAMP) model as a classification tool, will pro-
vide two benefits. First, the model will be sufficient to capture the complexity of
the domain and its hazards, thus providing a holistic perspective on the problem.
Second, the classification process will provide insight as to what steps might be
taken to mitigate the risk that medical errors associated with these software tools
will arise in health care systems which employ them. In this continuation of our
study we still have not been able to produce evidence which contradicts either hy-
pothesis.
Keywords. electronic medical record, safety, systems safety
Introduction
We recently published a classification of electronic medical record (EMR) hazards

based on a grounded theory approach [1] as part of a body of work in the area of EMR
system's safety which also includes [2, 3]. This classification effort was undertaken
with a systems theory perspective and a focus on human factors. The classification pre-
sented there is in many ways more complex than necessary. We present here prelimi-
nary results of our second iteration on this work. This study, at its completion, will ex-
tend the period of literature covered in the preliminary results and will also rely on a
variation of the EMR STAMP model presented in [2, 3] rather than the grounded theo-
ry approach in [1]. In other words, in this study we begin with the hypothesis that all
hazards described in the literature can be classified according to the EMR STAMP
model. We then set about searching for counterexamples to disprove the hypothesis.
The rest of this paper is laid out as follows. In section one, we will discuss relevant
background including an overview of our previous work and the work of others, an
overview of Leveson's system theoretic accidents models and processes [4], and finally
a number of definitions. In section two and three we will discuss the body or work which
has been inspected so far, the process which was used to identify it, and the methodology
1
F. Mason-Blakley and J. Weber / A Systems Theory Classification of EMR Hazards 215
and tools we used to extract the results we present in section four. In section five will
provide our conclusions and a discussion of the work that remains.
1. Background
We have motivated this topic a number of times in our EMR hazards research [1, 2,
3].We will provide a cursory overview of Leveson's STAMP model, an overview of
previous work, and we will review and revise a number of definitions.
1.1. System Theoretic Accidents Models and Processes
Leveson [4] describes a theory she calls the System Theoretic Accidents Models and
Processes (STAMP). A fundamental underlying concept of this theory is that when
accidents occur, it is not because of human error or even any single particular weakness
in the system of participants and interactions from which that accident arose. Much like
the "Swiss cheese" model for which Reason [5] advocates, Leveson proposes that in-
stead, accidents occur as a consequence of the alignment of a number of weaknesses in
the control structure that is intended to prevent them. That is to say that any system,
whether it be the aeronautics industry, the nuclear power industry or the health care
industry can be modeled as a web of control loops, and each loop and control node in
that loop has its own weaknesses. Leveson describes this control web as being com-
prised of interactions and controllers that rely on process models to affect their control
of processes in the system. Weaknesses in control structures can come in two primary
forms, weak or missing controllers, or weak or missing control interactions.
1.2. Related Work
There are a number of notable authors who have provided critical perspectives on the
benefits of EMRs. These tools are popularly touted as the cure to the ills of the health
care system. The media has at times implied that with the use of EMRs medical error
and economic waste in health care will be a thing of the past. Authors like Bates [7],
Ash [8], Koppel [9], van der Sijs [10], Saleem [11] and others however have put this
notion in doubt.
Saleem [11] identified a sense of resentment in professional practice against EMR
tools. This resentment is a reaction to the failure of the studied tools to satisfy the
workflow of their users. He describes how the elements of the EMR system operate in
unison to provide care. The studied care providers used the EMR tools, but also used a
variety of paper workarounds when the EMR tools did not support their workflow. In
order to understand the safety of the EMR system which Saleem studied, one would
need to understand all of its components and could not inspect the software in isolation.
Safety is an emergent property.
In [1] we described the first iteration of the study we now present. In [3] we pro-
vide a STAMP-centric hazard analysis of an e-iatrogenesis case study by Horsky [6]. In
[2] we provide a formal methods focused evaluation of that study.
216 F. Mason-Blakley and J. Weber / A Systems Theory Classification of EMR Hazards
1.3. Definitions
We provide here our definitions which will constrain our problem. We will define the
term EMR and we will also contrast it to two other terms: EHR, and PHR. These are
provided to clarify the inclusion and exclusion criteria which were used for the corpus
selection. We will refine and refactor the definitions based on out work in [1]. Finally,
we will refine the definition of e-iatrogenisis [6], to clarify our intended sense of the
term safety.
Electronic Medical Record: An EMR is a unit/suite of software that is used
for collecting, storing, manipulating or making available clinical information
about a patient's health care delivery process to facilitate the progression of
that patient's care.
Electronic Health Record: An EHR is a unit or suite of software that is de-
signed or used for extracting data about a patient's process of care from an
EMR for the purpose of maintaining an epidemiological understanding of a
population.
Personal Health Record: A PHR is much like an electronic medical record
except for one fundamental difference. A PHR is used by the patient to record
their health care process rather than by their physician.
E-Iatrogenesis: E-Iatrogenesis can be defined as medical error which is
caused (in part) by IT.
2. Corpus Identification
The corpus of work which has been inspected to date in this iteration of our literature
review was extracted strictly from the Engineering Village indexing service. This ser-
vice indexes a breadth of technical literature including IEEE and ACM, but does not
cover medical literature. Upon completion of this study we intend on including Pub-
Med as a source and seeking additional expertise in identifying other medical sources.
In order to capture the initial corpus of papers we executed the same query as in
[1]. This initial query yielded 116 results of which 21 were finally included in the re-
view. Inclusion and exclusion criteria were applied to determine acceptance in the
study; however, as the degree of agreement between the authors as to how the criteria
are to be applied remains low at this time, the criteria will only be summarized here. At
the completion of this study it is the aim of the authors to include an analysis of the
agreement on the application of the criteria including the use of a metric such as
Cronbach's alpha [13]. Here, we will only state that the papers were filtered through
three phases that inspected titles, abstracts and finally full papers. This filtering process
will by the end of this study be supplemented by including papers that are highly cited
in the primary corpus in a secondary corpus, and similarly a tertiary corpus will be in-
cluded based on references in the primary and secondary corpora.
3. Data Extraction Methodology and Tooling
The Atlas.TI version 6.2.27[13] application was used to record observations about the
papers in the corpus. This tool allows annotation and quotation of the text of pdfs. Dur-
ing the study, only when the information about the reported e-iatrogenesis hazards and
or instances were sufficient to populate an error report template based on the nomencla-
ture of Phillips and Gong [15] was information extracted. The information extraction
template consists of two primary elements: an error state and a precipitating event.
The error state is comprised of three subcomponents: the error element, the error
condition, and the error context. The error element describes the precise element which
was in error. The error condition describes how the error element was in error. Finally,
the error context describes how the error element fits into the broader context of the
software system.
The precipitating event is comprised of three elements as well: the event agent, the
event task, and the event context. Event agents are users who participated in the initiat-
ing or propagating the error state. The event task is the activity in which the event agent
was engaged when they initiated or propagated the error state. A challenge of the
methodology is that though much could be reported about context of e-iatrogenesis
instances and hazard, typically little is reported.
4. Results
The initial query returned 116 results. Of those 116 results, 30 were duplicate listings, 5
referred to non-articles/books, and 1 had been retracted. 29 papers of the initial 116
were excluded based on title, 26 were excluded based abstract, and 5 papers were ex-
cluded based on full content. 21 papers remained for inclusion in the primary corpus.
We uncovered 200 reported e-iatrogenesis instances or hazards. In order to classify
these discoveries, we employed an EMR specific instance of the STAMP model. We
present the annotated model in Figure 1. The annotations in this figure represent the
number of reported e-iatrogenesis instances or hazard and. the number of parent con-
cepts respectively. The arrows represent the control channels between controllers and
processes.
In analyzing the distribution of hazards across the model we present in Figure 1, it
is clear that two primary themes emerge: tool interaction issues and weaknesses in pro-
cess control. The tool interaction weaknesses are expressed across the nodes and edges
to which the EMR and physician process models are attached. Upon inspection of the
hazard descriptions it becomes clear that these hazards can be broken down into two,
non-exclusive subcategories: tool/work flow and display/control issues.
Figure 1. Representation of the STAMP model used to classify hazards and e-iatrogenesis instances.
218 F. Mason-Blakley and J. Weber / A Systems Theory Classification of EMR Hazards
Though these concepts, tool/workflow issues and display/control interaction issues

are commonly understood to be of significance in the field, the systems focus taken in
the evaluation of the corpus also strongly, and somewhat unexpectedly, highlighted the
prevalence of the recognition of the role of systemic controls in the failures of EMRs.
These systemic control failures extend from the highest levels where legislators and
regulators act, to much lower levels where care providers, hospital administrators and
software engineers hammer out product requirements for the EMRs which are to be
created. Out of the 21 papers in the primary corpus, 10 papers reported a total of 103
interaction problems of either the tool/workflow mismatches type or the user inter-
face/control type, and 12 papers reported on a total of 65 systemic control issues in-
cluding difficulties in requirements gathering/specification or legislative/regulatory
issues. Amongst the papers that discuss legislative and regulatory issues we cluster not
only those that talk about jurisdictional legislation or regulation as might be imposed
by a state, but also those issues of policy which arise within an organization.
5. Conclusions and Future Work
The most immediate conclusion which can be drawn from the results is that our hy-
potheses were not disproved. It is no surprise that the issues with tool/work flow mis-
match and user interface issues were highlighted as major hazard and e-iatrogenesis
sources. Experts in the field have been writing about these topics since the inception of
the EMR. The third theme which arose from this study, the theme of problems with
system controls, is more interesting in that it has been less explored in this domain.
If there were a single lesson to be learned from the preliminary results of this
study, it would be that additional effort in quality assurance, both in its enforcement
and execution may be key to reducing the frequency and severity of e-iatrogenesis. By
employing quality assurance controls which are used in developing EMRs including
user acceptance testing and usability assessment we may be able to simultaneously
mitigate tool/workflow mismatch issues as well as user interface problems.
References
[1] F. Mason-Blakley, J. Weber, CIS system hazards derived from literature using systems and human
factors perspectives. IHI 2012 (2012), 419-428.
[2] J. Weber, and F. Mason-Blakley, On the Safety of Electronic Medical Records, FHIES 2011 (2011),
177-194.
[3] J. Weber, F. Mason-Blakley, The Safety of Electronic Medical Record (EMR) Systems: What Does
EMR Safety Mean and How Can We Engineer Safer Systems? ACM SIGHIT Record 1(2) (2011), 13-22.
[4] N. Leveson, Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, 2011.
[5] J. Reason, Human Error: Models and Management, BMJ 320(7237) (2000), 768-770.
[6] J. Horsky, G. Kuperman, V. Patel, Comprehensive Analysis of a Medication Dosing Error Related to
CPOE, JAMIA 12(4) (2005), 377-382.
[7] D. Bates, et al. Improving EHR Accuracy and Increasing Compliance with Health Maintenance Clinical
Guidelines Through Patient Access and Input, IJMI, 75(10-11) (2006), 693- 700.
[8] J. Ash, M. Berg, E. Coiera, Some Unintended Consequences of Information Technology in Health
Care: The Nature of Patient Care Information System-Related Errors, JAMIA 11 (2004), 104-112.
[9] R. Koppel, J. Metlay, A. Cohen, B. Abaluck, A. Localio, S. Kimmel, B. Strom, Role of Computerized
Physician Order Entry Systems in Facilitating Medication Errors, JAMA 293 (3/2005), 1197-1203.
[10] H. van der Sijs, T. van Gelder, A. Vulto, M. Berg, J. Aarts, Understanding Handling of Drug Safety
Alerts: A Simulation Study, IJMI 79(5) (2010), 361-369.
[11] J. Saleem, A. Russ, C. Justice, H. Hagg, P. Ebright, P. Woodbridge, B. Doebbeling, Exploring the Per-
sistence of Paper with the Electronic Health Record, IJMI 78(9) (2009), 618-628.
[12] R. Holden, Cognitive Performance-Altering Effects of Electronic Medical Records: An Application of
the Human Factors Paradigm for Patient Safety, Cognition, Technology and Work 13(1) (2011), 11-29.
[13] M. Allen, W. Yen, Statistics notes: Cronbach's Alpha, BMJ (1997) 314-572.
[14] Scientific Software. Atlas.ti. [cited Sep. 2012], Available from: http://www.atlasti.com.
[15] W. Phillips, Y. Gong, Developing a Nomenclature for EMR Errors. HCI: Interacting in Various Appli-
cation Domains 13th International Conference. HCI International 2009 5613 (2009), 587-596.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-220
Benefits of a Clinical Planning and

Coordination Module: A Simulation Study
a,1 a a
Sanne JENSEN , Søren VINGTOFT Christian NØHR
a
Department of Development and Planning, Aalborg University, Denmark
Abstract. Digital Clinical Practice Guidelines are commonly used in Danish

health care. Planning and decision support are particularly important to patients
with chronic diseases, who often are in contact with General Practitioners, Com-
munity Nurses and hospitals. In the Capital Region of Denmark the potential bene-
fits of a planning and coordination module has been assessed in a full-scale simu-
lation test including 18 health care professionals. The results showed that health
care professionals can benefit from such a module. Furthermore unexpected new
possible benefits concerning communication and quality management emerged
during the test and potential new groups of users were identified.
Keywords. Clinical simulation, eHealth, clinical practice guidelines, usefulness
Introduction
Clinical practice guidelines (CPG) have been used more frequently during the last
years [1]. Continuity of care programs containing CPG aimed at planning and decision
support for healthcare professionals are therefore being developed [2]. The Capital
Region of Denmark is exploring the potential benefits of an information system
supporting the planning and coordination of chronic patient across sectors [3]. Patients
with Chronic Obstructive Pulmonary Disease (COPD) and Diabetes Mellitus Type 2
(DM2) are selected to establish a proof of concept project. Currently there are no
information system supporting the coordination and planning across community
nursing, general practitioners and hospitals in Denmark. The consequence is limited
planning and reduced coordination across the three sectors followed by decreased
quality and compliance of CPG. International experiences indicate that IT-systems can
enhance compliance as well as quality of care [4;5].
The Capital Region in Denmark has launched a project: “Chronic 5” that aims to
demonstrate the potential benefits of a Planning and Coordination Module (PCM). The
project analyzes and specifies requirements for such a system and builds and tests a
PCM prototype. Clinician end-users, clinical managers, quality managers, IT-architects
and health informaticians performed the analysis and the specification.
The PCM is basically designed to establish and maintain a cross organizational
overview and virtualized management of all health services in individual patient cases
among all relevant health actors – including the patient. All health services in an
individualized patient plan are mapped to relevant CPGs
1
Sanne Jensen: [email protected]
S. Jensen et al. / Benefits of a Clinical Planning and Coordination Module: A Simulation Study 221
The purpose of PCM is to support the coordination across sectors, concerning the
status and planning for patients with COPD and DM2 according to the CPG, and
handling of derived activities and services. This digital support will be groundbreaking
in Denmark, and will offer new opportunities for coherence and continuity in the care
activities. Moreover it will possibly ensure a higher compliance to the existing
continuity programs and CPG.
To realize the intended benefits of a PCM usability of the system is pivotal [6].
Usability may be defined as “extent to which a system, product or service can be used
by specified users to achieve specified goals with effectiveness, efficiency and
satisfaction in a specified context of use” [7]. When using simulation it is possible to
assess the effect of an information system in different contexts as well as evaluating
efficiency, satisfaction and effectiveness [8]. The objective of the simulation study was
to assess the potential benefits of a PCM for health care professionals involved in
planning and coordination of patients with COPD and DM2, primarily focusing on the
efficiency of the PCM, and secondary on satisfaction. Efficacy and effectiveness has
not been assessed.
This paper presents the test of the PCM-prototype and the results from the test, and
discusses the potential benefits and concerns of a PCM. Furthermore the use of
simulation as a method for testing potential use of clinical information systems is
discussed.
1. Method
The test was conducted as a controlled full-scale simulation study. The concept of
simulation has been used for training medical skills during the last 40 years [9] and has
during the last decade been used to assess health information systems [10]. A
simulation study makes it possible to assess the use of a prototype in a realistic
environment [11], and is well suited for assessing potential impact [12] as well as
cognitive processes and usability [13].
The test encompassed 18 simulation runs including six general practitioners (GP),
six community nurses, six hospital physicians and two simulation patients. The
simulation runs were bundled into six tests. In each test healthcare professionals from
each of the three end-user groups were participating. 10 scenarios were composed; five
about a patient with COPD and five about a patient with DM2. The scenarios covered
planning of therapy and further diagnosing concerning a recently diagnosed patient at
the GP, visitation at the community nurse, rehabilitation at the community nurse,
treatment of a patient at an outpatient clinic due to exacerbation of the condition, and
assignment of responsibility from the hospital physicians to the GP. The scenarios
resembled different points of impact focusing on core functionalities and the
assignments from one healthcare professional to another. Interface issues such as colors,
buttons and minor functionalities were not part of the assessment. The scenarios were
composed to assess nine hypotheses i.e. the first nine questions in Figure 1.
Before the simulation took place the testers were introduced to the concept and the
functionalities of the PCM and they could get hands-on in order to get acquainted with
the information system. During the test the same general tasks were performed. In
cooperation with the “patient” and on the basis of the existing findings and plans, the
healthcare professionals were asked to revise and modify the plans for the patient. The
prototype had simulated integrations to other information system in order to replicate
222 S. Jensen et al. / Benefits of a Clinical Planning and Coordination Module: A Simulation Study
the intended integrations to legacy information systems. A test-coordinator was sitting

next to the tester during the test to assist the tester in case of problems using the system.
The tester was asked to “think-aloud” [14] during the test, and the test-coordinator did
obser-view [15] asking more exhaustive questions if necessary. By asking questions
about the system, the “patient” was able to force the tester to describe the system and
the functionalities in a close to natural setting. Health informatics experts experienced
in simulation test conducted the role of the patient. In the control room a test instructor
and several observers followed the test through a one-way mirror. The instructor was in
radio contact with both the “patient” and the test-coordinator during the test. Hereby
the instructor was able to direct the test to ensure the objectives. The observers
monitored the test and their observations were used in the subsequent debriefing-
interview. During each test testers from the three sectors were present, but only one
was testing, while the others observed from the control room.
Data for the evaluation was acquired by questionnaire and debriefing-interviews
with testers and observers. The questionnaire had nine questions concerning the
hypothesis, two about quality, four about overview, two about the division of
responsibilities, four about work practice and efficiency, and three questions about the
simulation and the realism in the scenarios. The interview guide started with open-
ended questions concerning positive and negative features of the system, followed by
specific questions to clarify and elaborate on issues from the questionnaires and other
issues that came to their mind. At the end of each day the data from the interviews were
analyzed using Instant Data Analysis (IDA) [16]. As supplement to IDA the
observations from the simulations, the notes from the interviews and the IDA notes
were analyzed using Nvivo (QSR International).
2. Results
The results from the questionnaire are shown in Figure 1. The horizontal scale depicts
the median of the respondents answer on a five point likert agree/disagree scale to the
24 questions on the vertical axis. The hypotheses tested in the first nine questions were
verified. Among the remaining questions only one obtained a lower score than 3 i.e. the
question concerning whether the PCM would release more time to be spend with the
patients. From the interviews, however, the general opinion was that the PCM would
reduce the time spend on the planning and coordination, but it remain unresolved,
whether the time would be spend with the patients. This result was the only one with
discrepancy between interview and questionnaire.
The core concept of the PCM was assessed as being very useful and creating many
benefits. New ideas were brought up during the interviews – eg. the PCM could be a
coaching tool for senior doctors and an instrument for communication among
colleagues or between other groups of healthcare professionals. Primary care nurses
were not part of the original scope, but were spotted as new potential users by a GP
who saw the PCM as a very valuable tool for them. Also quality management was
perceived to be enhanced, and the content of referrals and discharge letters could
possibly be reduced, since information concerning the patient would be known by all
parts.
Most of the healthcare professionals had difficulties understanding the concept of a
PCM in the beginning. The concept was innovative and forced them to see planning
and coordination in a new way. The simulation and observation of the others using the
S. Jensen et al. / Benefits of a Clinical Planning and Coordination Module: A Simulation Study 223
Figure 1. Potential benefits: Result from Questionnaires (n=14)

system helped them to understand the concept. Several issues of concern were also
brought up. 1) The healthcare professionals found that the PCM module gave a good
overview of the patients, but at the same time they wanted the possibility of looking
into more details about the patient. They recommended to specify this in the
requirements. 2) The test showed that the terminology used in the three sectors,
differed on several central terms such as “referred to” and “deselected”. 3) Sharing of
responsibility as all will have the same access to data, but should it be possible for a
physician at the hospital to overrule a prescription from the PG - or vice versa? 4)
Several users stressed that realization of integrations were of vital significance.
3. Discussion
The healthcare professionals found potential clinical benefits in using the PCM, which
would improve quality and patient safety. Furthermore new future users were
discovered and new potential ways of using the PCM were revealed. Only simulated
patients were used during the test, but several potential benefits for the patients were
detected. A supplementing simulation with genuine patients would therefore be
recommended to test the sustainability of these observations. The healthcare
professionals were quite satisfied with the realism in the simulation, and it helped them
to gain insight in the possibilities of PCM.
The scenarios did not cover all possible applications of the PCM but were
composed to enable assessment of the nine hypotheses. A simulation test does not fully
224 S. Jensen et al. / Benefits of a Clinical Planning and Coordination Module: A Simulation Study
resemble the use of an information system in the clinic, but offers a high degree of
realism, depending on the degree of fidelity. A simulation test should therefore not be a
substitution for a pilot implementation, but regarded as a complementary test without
risk of injuring real patients.
Several issues that were brought up e.g. terminology and responsibility had not
been visible before the simulation test, but are very relevant and needed to be addressed
prior to implementation. Furthermore it was discovered how the simulation was a
powerful learning tool for the new users in spe.
The results from this simulation study conclude that GPs, community nurses and
the hospital physicians and patients will benefit from a PCM. The benefits include
improvements in communication, planning and coordination, work practice, and
quality management. Several organizational issues have to be addressed including use
of terminology and delegation of responsibilities before an information system as PCM
can be implemented. Furthermore the results show that full-scale simulation studies are
a useful method for testing the feasibility of information systems.
References
[1] Hanson D, Hoss BL, et al., Evaluating the evidence: guidelines, Aorn Journal 16 (2008), 388-393.
[2] National Board of Health in Denmark. Clinical Practice Guidelines concerning patients with chronicle
deceases; recommandations from the Danish National Board of Health in Denmark.
http://www.regionh.dk/menu/sundhedOghospitaler/Til+fagfolk/Kronisksygdom/Forloebsprogrammer/I
ndhold_forloebsprogrammer/Forloebsprogrammer+for+kronisk+sygdom/Kronisk+sygdom+programme
r.htm (in Danish) (accesses March the 22nd, 2012).
[3] Program for projects concerning chronicle patients in the Capital Region of Denmark 2011
http://www.regionh.dk/NR/rdonlyres/F783D082-5E74-469D-87A9-88755A39A9A3/0/ Kronikerpro-
grammets_projektkatalog_sep_2011.pdf (accesses June the 15th, 2012).
[4] Hogan TP, Wakefield B, et al., Promoting access through complementary eHealth technologies:
rcommandations for VA's Home Telehealth and personal health record programs, Journal of General
International Medicin 26 (2011), 628-635.
[5] Jackson GL, Krein SL, et al., Core Issues in Utilizing Information Technology to Improve Access:
Evaluation and Reseach Agenda, Journal of General Internal Medicine 26 (2011), 623-627.
[6] Peute LW, Spithoven R, et al., Usability studies on interactive health information systems; where do we
stand? Stud Health Technol Inform 136 (2008), 327-332.
[7] ISO 9241-11, Ergonomic requirements for office work with visual display terminals (VDT's). Part 11:
Guidance on usability, 2012.
[8] Dahl Y, Alsos OA, Svanæs D. Fidelity Considerations for Simulation-Based Usability Assessments of
Mobile ICT for Hospitals. Journal of Human-Computer Interaction, 26(5) (2010), 445-476.
[9] Lippert A, Dieckmann PG, et al., Simulation in medicine, Notfall Rettungsmed. 12 (2009), 49-52.
[10] Borycki E, Kushniruk A, et al.. Designing and Integrating Clinical and Computer-based Simulations in
Health Informatics: From Real-World to Virtual Reality. Modeling, Simulation and Optimization –
Focus on Applications (2010), 31-52.
[11] Rossnagel H, Zibuschaka J. Using Mobile Media for emergency Management - A Design Approach
Proceedings of the 8th International ISCRAM Conference (2011).
[12] Ammenwerth E, Hackl WO, et al. Simulation studies for the evaluation of health information
technologies: experiences and results. Health Information Management Journal, 41(2) (2012), 14-21.
[13] Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical
information systems. J Biomed Inform 37(1) (2004), 56-76.
[14] Jaspers MW, Steen T, van den Bos C, Greenen M. The think aloud method: a guide to user interface
design. International Journal of Medical Informatics 73 (2004), 781-795.
[15] Kragelund L. Obser-view seen as a data generating method and a learning space. 2009.
http://static.sdu.dk/mediafiles/Files/Om_SDU/Institutter/Ifpr/Konferencer/Adult_Learning/Abstracts/3
%20LindaKragelund.pdf ( accesses June the 15th 2012).
[16] Kjeldskov J, Skov MB, Stage J. Instant data analysis: conducting usability evaluations in a day.
NordiCHI '04 Proceedings of the third Nordic conference on Human-computer interaction (2004).
Mobile Technologies and Telehealth
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-227
Issues and Considerations for Healthcare

Consumers Using Mobile Applications
Elizabeth CUMMINGSa,1, Elizabeth Marie BORYCKI b and Erin ROEHRER a
a
eHSRG, University of Tasmania, Hobart, Tasmania, Australia
b
Abstract. There is a large and ever increasing number of mobile phone health,
wellness, and medical applications on the market. However, there is little guidance
or quality assurance available for consumers. This paper provides a health
consumer focused framework for considering a range of issues associated with
selecting and using mobile phone applications downloaded from the Internet.
Keywords. Mobile phone apps, M-health, data storage issues, software issues
Introduction
Internationally we have seen a significant rise in mobile phone use. According to

International Telecommunications Union statistics there were over 5.3 billion mobile
phone subscribers globally [1]. In 2011 491.4 million smart phones were shipped
globally [2]. With the rise in mobile phone (including smart phones) usage rates, there
has also been an increase in the development and sale of mobile phone applications
across the globe. Many of these applications are healthcare related and consumers are
identifying and using mobile phone applications to: (1) support their health decision-
making and (2) assist in self-management of their disease or maintain wellness. This
paper provides a discussion of some of the considerations surrounding mobile phone
software and the data these applications collect to help support consumers decision-
making when using mobile phones and their associated software applications. In the
literature there are many discussions on the use of health applications by health care
professionals or those on formal clinical trials [3], these applications are not
specifically under consideration in this paper. This paper provides a consumer health
focus on some of the issues for consideration when selecting and using mobile phone
health and wellness applications downloaded from the Internet.
1. Background
Mobile phone applications are small software programs that provide a specific
functionality and can be downloaded onto a phone via the Internet. Most frequently
they are associated with smart phones, which use a proprietary operating system (OS)
1
Elizabeth Cummings, School of Computing & Information Systems, University of Tasmania, Private
Bag 87, 7001, Hobart, Tasmania. E-mail: [email protected]
228 E. Cummings et al. / Issues and Considerations for Healthcare Consumers
and allow the user to download specifically designed applications that run only on that
OS, although cross platform mobile apps are now beginning to be developed. Many
mobile phone applications can only be run using the Internet to provide real-time
information and data exchange whilst others are independent from the Internet and
developed as standalone applications, which do not require Internet access. Other apps
can be operated in either mode, with restricted functionality in the offline mode.
It has been postulated that smart phone applications can transform mobile phones
into assistive devices for people with disabilities [4]. As smart phones become more
affordable and the applications remain relatively low cost there is a rapidly increasing
acceptance and use of assistive applications by consumers. Consumer use of mobile
applications is varied. Some consumers use mobile phones to obtain relevant health
information over the World Wide Web (WWW). Other consumers have taken
advantage of the growing availability of mobile applications that are specifically
targeted towards assisting individuals with the promotion of their own health and
wellness as well as the self-management of chronic illnesses such as chronic
obstructive pulmonary disease, diabetes and cardiovascular disease. Other mobile
applications are provide tools that improve communication with health care providers
or facilitate the delivery of patient care, for example: animated videos to educate
patients about the human body, wellness, prevention activities, disorders and diseases.
There are currently more than 13,500 medical, healthcare and fitness applications
in the apple app store alone. The majority of these applications are aimed at consumers
rather than healthcare professionals. Although many are potentially good applications,
researchers and health professionals are increasingly advocating that consumers
consider the types of applications they are downloading or purchasing with some
skepticism, particularly when selecting and using health focused applications [5]. Some
of these considerations form the focus of this paper.
2. Method
The authors of this paper had each independently examined and used a range of mobile
applications for health and wellbeing. This examination had led to considerable
concern regarding the security and quality of mobile phone applications and the data
they collect. In this context we developed a “Consumer Perspective Framework” (see
Table 1) within which we consider the use of mobile applications from a consumer
perspective. The framework emphasizes some key aspects of mobile phones and their
associated applications that need to be assessed by consumers when considering their
use. Key elements of the framework will be discussed in the following section.
Table 1: Consumer Perspective Framework

Data Issues Software Issues
Storage and Privacy Accessibility
Ownership Clinical Effectiveness
Corporate Use Credibility
Location Information Quality
Completeness Consumer Usage
E. Cummings et al. / Issues and Considerations for Healthcare Consumers 229
3. Discussion
Two major areas of concern relating to mobile phone software applications and data
storage were identified: (1) data issues and (2) software issues.
3.1. Data Issues
There are several data issues that consumers should concern themselves with,
including: (1) storage and privacy, (2) ownership, (3) corporate use and (4) location.
Storage and Privacy: One challenge is keeping personally identifiable health
information secure. Essentially, mobile health requires the use of strong encryption and
authentication processes to ensure the security of these data. However, as patients are
increasingly recognising the potential benefits of digitally recording their health
information this issue is becoming more discussed in the patient population.
Researchers such as Walker et al [6] have identified that patients are less concerned
about privacy than health care professionals. However, Gilbert et al [7] emphasise that
ensuring privacy is not violated where sensitive information is concerned is a difficult
problem to overcome. There is minimal evidence of the use of secure networks or
encryption (as in eHealth applications).
Ownership: There is minimal information available about these mobile phone
applications at the point of purchase regarding data storage, use (primary and
secondary), durability and ownership. These issues need to be considered when
entering personal data, including health data, into a mobile phone application. This
information may not be available to the consumer via the app store prior to purchase.
There is a requirement that all applications cover these issues in their licensing
agreement, which the consumer accepts at the time of initial use. However, some users
do not understand the importance of these licensing agreements (which by requirement
are very long). One application’s licensing agreement, containing standard information,
was 26 screen pages long and difficult to read on a mobile phone due to the small
screen size. Thus the important elements of the agreement may be difficult to read.
Corporate Use: Some low cost mobile phone applications are sponsored by private
enterprises, which can have implications for data privacy and security. One of the key
issues is the secondary use of data as application sponsors often provide support to
developers in exchange for access to end user personal data. Although, through
licensing agreements, this is agreed to by end users it is unlikely that the implications
of such agreements have been understood by consumers. Third parties do not have to
disclose their actual use of the data and application developers have little or no control
over secondary data use. There is limited information about the duration of data storage,
or its subsequent use. In the event that the consumer is aware of secondary data
transmission, there is the element of consent to the frequency of data transmission.
There is concern about the timing of data transmission and to what extent consumers
expect their information to be transmitted automatically, and what information is
transmitted?
Location: The applications available through the Apple iTunes store are required
to meet certification requirements regarding the content and development of the
application [8]. This certification does not extend to when and where personal
information may be sent once the application’s disclaimer has been accepted by the
user. No certification process is required for apps available through the android market
[8] and so there is little evidence of where the data is processed or stored. The location
230 E. Cummings et al. / Issues and Considerations for Healthcare Consumers
of data storage or transfer can be important, particularly if users are anticipating

personalised responses to data inputs. If data are only stored locally, on the mobile
phone, then it is necessary that users be aware of this so they do not anticipate others
will be ‘monitoring’ them. The reverse is also important where users may consider that
they are undertaking a personal venture and then discover their information is being
reviewed by others. Consumers need to understand the importance of these issues in a
computerised world.
3.2. Software Issues
There are several software issues that also need to be considered: (1) accessibility, (2)
clinical effectiveness, (3) credibility, (4) information quality, and (5) consumer usage.
Accessibility: One of the first issues confronting consumers of health and
wellbeing applications is the issue of how to locate them. Each of the different phone
operating systems (e.g. android) has its own propriatary application stores (app stores)
where applications can be purchased and downloaded. The sheer number of
applications currently avaliable has the potential to make the search and selection of a
useful health or wellness app overwhleming. The majority of app stores have relatively
unsophisticated search mechanisms and so for individuals who are uncertain of what
they want may find identifying and selecting an app time consuming and cumbersome.
Clinical Effectivness: There are a large number of medical, health and wellbeing
applications. Most of these applications have a limited description of the application’s
purpose and few reviews documenting the application’s clinical quality (thereby
making selection of mobile applications on the basis of clinical effectiveness more
difficult for consumers). Thus, mobile applications may or may not even function as
expected, let alone be of use to the individual in supporting a wellness behaviour or
helping with the self-management of a consumers’ disease. Where applications are
calculating health or wellbeing measures from data inputs it is important to know and
understand the means, or algorithms, by which these calculations are being made.
Inaccurate calculations, or non-standard ones, may be in conflict with current medical
trends and advice and may lead to medical error [10]. It is also possible that results may
be ambiguous or misinterpreted by consumers with low health literacy.
Credibility: Anyone can create an application so it is difficult to differentiate
credible, safe applications from those which are not. There is little information
available in an app store to indicate credibility and if there are reviews these can also be
biased or based on unhelpful metrics. As a result of this, there is a need for
undestanding of the possible issues for both consumers and health care practitioners.
There is potentially a role for policy development which will assist people in
classifying these applications. For example inn the USA the Food and Drug
Administration recently released draft guidelines for Mobile Medical Applications [9].
Information Quality: As with health related websites, there is concern about the
quality and accuracy of information and advice provided through mobile phone apps.
For the scope of this paper the concept of information quality is made up of several
elements: (1) quality of the information in the application program incorporated by
developers, (2) quality of the information entered into the application by consumers,
and (3) quality of information received by the user through their interaction with the
application.
Consumer Usage: The motivations behind the use of mobile apps have the
potential to impact on the quality of the information recorded. Is the health consumer
E. Cummings et al. / Issues and Considerations for Healthcare Consumers 231
after real behavioural change or are they after information to show a healthcare
provider to illustrate compliance with a care program? Information quality has the
potential to impact on healthcare consumers overall program of care, depending on the
credibility that the application holds with the healthcare provider and the methods by
which it is used by the consumer.
Having devised and described this framework the authors are intending to move to
the next stage and conduct research with consumers to test the reliability of the
framework. This work is intended to provide a consumer focused understanding of the
most appropriate method by which the framework can be implemented for use by
individual consumers to assist them in selecting the most reliable applications. This
work will include a comprehensive exploration of literature relating to the framework.
4. Conclusion
There are many facets to consider when selecting a mobile app for use with health and
wellbeing in mind. Consumers are encouraged to consider the security and secondary
use of the data prior to engaging with these applications. App stores and developers
should be encouraged to provide more reputable information about the privacy and
security of the data as well as the efficacy and source of the programs. Healthcare
professionals as well as consumers, providers and developers have a role to play in
ensuring that only high quality efficacious apps are developed into the future. In this
paper we describe a framework that can be used by consumers and health professionals
when considering mobile application for healthcare use.
References
[1] International Telecommunications Union (ITU), ICT data and statistics. 2012. [cited 2012 Jan 20]
Available from: http://www.itu.int/ITU-D/ict/statistics/.
[2] Mobithinking. Global mobile statistics 2012 Part A: Mobile subscribers; handset market share; mobile
operators. 2011[cited 2012 Aug 21] Available from: http://mobithinking.com/mobile-marketing-
tools/latest-mobile-stats/a#subscribers.
[3] Jones J, Hook S, Park S, Scott L. Privacy, security and interoperability of mobile health applications.
Proceddings of the 6th international conference on universal access in human-computer interaction.
Orlando, FL. Springer-Verlag; 2011.
[4] Turisco F, Garzone M. mHealth for your organization, CSC, 2011. [cited 2011 Oct 10]. Available from:
http://assets1.csc.com/health_services/downloads/CSC_Harnessing_the_Value_of_mHealth.pdf,.
[5] Doughty K. SPAs (smart phone applications) – a new form of assistive technology. Journal of Assistive
Technologies. 2011; 5(2): 88-94
[6] Walker J, Ahern DK, Le LX, Delbanco T. Insights to Internists: I Want the Computer to Know Who I
Am. J Gen Intern Med. 2009; 24(6): 727-732
[7] Gilbert P, Chun BG, Cox L, Jung J. Automating privacy testing of smartphone applications. Technical
Report CS-2011-02, 2011. Duke University
[8] Rose C. Smart Phone, Dumb Security. Review of Business Information Systems, 2012; 16(1): 21-
262012
[9] FDA, Draft Guidance for Industry and Food and Drug Administration Staff: Mobile Medical
Applications, July 21, 2011.
[10] Baylis TB, Kushniruk AW, Borycki EM.. Low-cost rapid usability testing for health information
systems: Is it work the effort? Stud Health Technol Inform. 2012; 180: 363-7.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-232
The Introduction and Evaluation of Mobile

Devices to Improve Access to Patient
Records: A Catalyst for Innovation and
Collaboration at BCCA
Jonn WUa,1, John WALDRON b, Shaina HOOD b, Adam KAHNAMELLI a, Mohamed
KHAN a, Jeff BARNETT a, John FRENCH a, Stacey SLAGER c, Shadi MELHEM c and
Omid SHABESTARI c
a
BC Cancer Agency,
b
Provincial Health Services Authority (PHSA), IMITS,
c
Abstract. Prompt and efficient access to patient records is vital in providing

optimal patient care. The Cancer Agency Information System (CAIS) is the
primary patient record repository for the British Columbia Cancer Agency
(BCCA) but is only accessible on traditional computer workstations. The BCCA
clinics have significant space limitations resulting in multiple health care
professionals sharing each workstation. Furthermore, workstations are not
available in examination rooms. A novel and cost efficient solution is necessary to
improve clinician access to CAIS. This prompted the BCCA and IMITS to embark
on an innovative provincial collaboration to introduce and evaluate the impact of a
mobile device to improve access to CAIS. The project consisted of 2 phases with
over 50 participants from multiple clinical disciplines across BCCA sites. Phase I
evaluated the adoptability, effectiveness and costs associated with providing
access to CAIS using a generic viewer (Citrix). Phase II incorporated the feedback
and findings from Phase I to make available a customized mobile deviceǦspecific
application. Phase II also addressed privacy and security requirements.
Keywords. Mobile technologies, Technologies as agents of change, Electronic

Health Records, Human Computer Interaction
Introduction
Searching about the clinical use of mobile devices produces impressive results about
their increasingly ubiquitous use by care providers. Anecdotes about Apple iPads
having improved teaching and medical practice are common. However, is this
technology hyperbole or are there tangible benefits?
1
Corresponding Author. Jonn Wu, BCCA, 600 West 10th Avenue, Vancouver, BC V5Z 4E6, Canada;
E-mail: [email protected].
J. Wu et al. / The Introduction and Evaluation of Mobile Devices to Improve Access 233
1. Statement of Problem
The BC Cancer Agency (BCCA) uses an electronic health record named CAIS;
however, it lacks adequate physical space for clinicians to access records via desktop
computers. Also, clinicians do not have access to computers in the patient rooms,
during clinical conferences or grand rounds.
2. Purpose
Determine if providing a mobile device with access to CAIS, clinical reference

applications and administrative tools improved care delivery and clinicians’ experience.
3. Methodology
The project followed a two-phased approach. Phase I gave users the ability to
download clinical reference applications and access the Cancer Agency Information
System (CAIS) using a generic viewer (Citrix). Phase II incorporated usability
feedback from Phase I to deploy a highly configurable health record viewer delivered
as a native iPad application.
3.1. Phase I – Methods and Procedure
3.1.1. Sampling and Recruiting

Thirty-four (34) Radiation Oncologists (RO) at Vancouver Centre BCCA were
supplied with iPads. Each RO active in clinic was invited to participate in the study and
evaluate the effectiveness and utility of the iPad.
3.1.2. Instrumentation
Participants were emailed with a link to complete two confidential surveys. This
involved a survey before the iPads were distributed and a survey after twelve weeks of
use. Participants rated items on a Likert scale.
3.1.3. Data Collection

The survey tool collated data collected from participants. As well, informal interviews
were completed before Phase I and at the end of Phase II.
3.1.4. Data Analysis

The responses to the second survey assessed if and how iPads were effective in a
clinical setting. Information collected was analyzed and interpreted using descriptive
statistic analysis. For qualitative answers, the research team interpreted the written
responses.
234 J. Wu et al. / The Introduction and Evaluation of Mobile Devices to Improve Access
3.2. Phase II – Methods and Procedures
3.2.1. Sampling and Recruiting

There were 50 participants in the study. Twenty-five (25) radiation oncologists from
multiple BCCA sites, some of whom participated in Phase I, were included in Phase II.
Additionally, 25 medical oncologists from Systemic Therapy groups across all 5
BCCA Centers were invited to participate.
3.2.2. Instrumentation
Communication was addressed with an intranet website. Participants were emailed with
a link to the intranet site and listed the four steps required to enroll in the pilot initiative.
Step 1 provided information about the purpose of pilot; Step 2 incorporated privacy and
security requirements by having participants sign a project privacy agreement; Step 3
provided a baseline survey that aligned with Phase I; and Step 4 provided instructions
on how to launch the patient record viewer application on the iPad.
3.2.3. Mobile Patient Record Viewer Application Functionality

The mobile application presented information from CAIS in a user interface designed
for an iPad. The application provided users with read-only access to all of the patient
information available on CAIS (outside of imaging). Also, the application provided
access to clinicians’ clinic schedule and patient appointments. The application has a
user-defined configuration for the visual layout of the patient record.
3.2.4. Data Collection

At the end of Phase II, participants were asked to complete a post-implementation
survey to measure the effect of the iPad and patient record application. A subset of
participants was observed while using the current tools such as desktop computers and
paper charts. This was compared to the post-pilot observations when the iPad and
patient record viewer application was available. The observations captured data about
time required to access patient information, turnaround time between patients, and
challenges with using the tools. Twelve semi-structured interviews were also conducted
during the project period.
3.2.5. Data Analysis

A multi-method research study was used for the evaluation that involved surveys,
observational studies, and interviews. The responses to the post-implementation survey
and interviews determined if and how iPads and the patient record viewer application
are useful and effective in a clinical setting. The baseline survey results were
compared to the post-implementation survey results. Similarly, the results of the
baseline observations were compared to the post-implementation observation results.
The interview results were analyzed using grounded theory methodology to identify the
major themes.
4. Results
4.1. Phase I - Preliminary Findings
Pre Usage
In the pre-usage survey, the user group rated themselves as very comfortable with
mobile devices, with a large majority (86%) of the group using mobile devices daily
and 78% already finding mobile devices useful in their professional practice. A
majority (67%) of the users envisioned that having a mobile tablet would enhance their
clinical workflow, with 43% of users already having an idea of what functions they
could employ, and close to all users (91%) looking forward to adding a mobile tablet to
their clinical practice.
Post Usage
Upon completion of the three-month pilot project, a large majority of the users (83%)
reported having an overall positive experience using the mobile tablet, with only 3% of
users summarizing the pilot project as a negative experience.
The mobile tablet was reported as being easy to carry around (70%), had
satisfactory screen resolution (63%) and an appropriate screen size (50%). Users found
using the mobile tablet an efficient way to check and write email (62%) and review
electronic records (55%), with a majority (63%) of users noting a positive impact on
their workflow through the use of a mobile tablet; highlighted by 76% of users
explaining that CAIS through their mobile tablet was more up to date than the patient’s
paper chart. In addition, once users became comfortable with the mobile tablet, over
60 percent of users began searching out other medical application to incorporate into
their clinical workflows.
Initially, a number of users had minor complaints regarding Internet connectivity
(38%), and login issues to PHSA Network (16%). In most cases, the help of on-site
support addressed these issues. Another suggestion was to increase the font size of the
CAIS viewer (55%).
In summary, users found that the use of a mobile tablet was a useful tool in their
clinical practice (73%), with the tablet primarily being used for email (96% of users),
review of imaging and pathology (90% of users), and review of labs and appointment
notes (87% of users). For 75% of users, the use of a mobile tablet met or exceeded
their expectation in terms of functionality in their practice.
4.2. Phase II - Preliminary Findings
Pre Usage
In general, the results show that clinicians at BCCA are among the early adopters of
mobile computing technology according Rogers’ Diffusion Theory (with more than
90% of the respondents are using mobile computing technology in their clinical
practice). The majority of the respondents (> 80%) are either very comfortable or
comfortable with using mobile computing technology.
BCCA clinicians highly appreciate (98%) the in-time availability of information, a
privilege peculiar to mobile technology that subsequently offer clinicians the ability to
respond instantly to patients questions and enquiries. Other highly ranked and expected
functionalities include: using the iPad for patient education, communication purposes,
and for clinical decision-making. Moreover, almost all respondents (98%) expect that
236 J. Wu et al. / The Introduction and Evaluation of Mobile Devices to Improve Access
the iPad and other mobile computing technology to be become larger part of the cancer
care experience for both clinicians and patients.
Post Usage
The majority of the post-implementation survey respondents (92%) are either highly or
moderately satisfied with the iPad and VH Chart application. BCCA clinicians rated
the iPad project between 4 and 5 with 5 being extremely positive. The iPad application
visual interface, multiple functionalities, and ease of use were the main drivers for
clinicians’ positive feedback. Figures 1, 2 and 3 illustrate early utilization.
Based on the results of the post-pilot survey results, most of the post-
implementation survey respondents (92%) found that the response time, security and
reliability of the iPad and the mobile application to be either moderately or highly
acceptable. In terms of the information quality, most of respondents (92%) found the
information within mobile application to be either highly or moderately acceptable.
Similarly, most participants (90%) agree that there were sufficient technical support
and training resources for the iPad Mobility project. in general, most of the respondents
(95%) rated the project implementation process either moderately or high acceptable.
In terms of productivity and efficiency, clinicians reported reduction in the number
of interruptions during patient visits and a reduction in the need for printing (reports,
conference documents). Most of the survey respondents (> 80%) believe that the iPad
improved their workflow as it facilitated the retrieval of information from CAIS. This
allowed the clinicians to involve the patients in care planning, increase collaboration
with other healthcare providers and enhance support in decision.
5. Conclusion
The majority of clinicians supported the use of a mobile device to improve access to
patient health records, but recognized limitations of a generic viewer to simulate a
desktop view of CAIS. Clinicians preferred the iPad mobile application because of its
ergonomic features and ease of use. Also, clinicians provided feedback to enhance the
mobile application features. Notable is that most clinicians viewed the iPad application
as complementary to their desktop computer and improved their workflow.
Figure 1. Phase II – Usage Statistics for First Month - # Logins per day.
Figure 2. Phase II – Usage Statistics for First Month - # Unique Logins per day.
Figure 3. Phase II – Usage Statistics for First Month – Total Logons by top 20 users.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-238
Moving Mobile: Using an Open-Sourced

Framework to Enable a Web-Based Health
Application on Touch Devices
Joseph LINDSAYa,1, J. Allen McLEANa, Amrita BAINSa, Tom YINGa
M.H. KUOa
a
School of Health Information Science, University of Victoria, Victoria, BC
Abstract. Computer devices using touch-enabled technology are becoming more

prevalent today. The application of a touch screen high definition surgical monitor
could allow not only high definition video from an endoscopic camera to be
displayed, but also the display and interaction with relevant patient and health
related data. However, this technology has not been quickly embraced by all health
care organizations. Although traditional keyboard or mouse-based software
programs may function flawlessly on a touch-based device, many are not practical
due to the usage of small buttons, fonts and very complex menu systems. This
paper describes an approach taken to overcome these problems. A real case study
was used to demonstrate the novelty and efficiency of the proposed method.
Keywords. APEX, jQuery Mobile, tablet, touch-based, database, web-based,

health, informatics, open-source, iPad
Introduction
Over the past thirty years, personal computer use has traditionally tethered users
to a fixed point due to both the physical size of devices, the requirement of an
electrical outlet as a power source, and the use of hardwired network cabling.
Recently, personal computer technology has seen an immense change through the
decrease in component size, increase in microprocessor performance, advancements
in battery technology and the rise of cellular communications and mobile Internet.
Since the introduction of the Apple iPad in the spring of 2010, an immediate shift
in the form of touch-based tablet computer devices has been observed. Touch
enabled technology is becoming more and more prevalent today and its influence
can be observed in the interfaces of the latest computer operating systems such as
Microsoft Windows 8’s ‘metro’ interface and the Launchpad in Apple’s Mac OS
10.8 Mountain Lion.
Computer devices have become extremely portable with fingers and voice
replacing the mouse and keyboard as the primary means of control. Although the
introduction of such technology has been exciting, especially to many in the health
care industry, it has not been quickly embraced by all health care organizations [1].
1
Corresponding Author: Joseph Lindsay, Room A2102, Parkwood Hospital, 801 Commissioners Rd E,
London CA; Email: [email protected]
J. Lindsay et al. / Moving Mobile: Using an Open-Sourced Framework 239
A major issue with touch-based technology is the fact that previous software,
which was designed with a traditional mouse and keyboard interface in mind, was
not intended for use with a touch-based interface. Regardless of how accurate the
touch screen at recognizing the user’s touch, the accuracy is restricted by the size
and position of a user’s finger [2, 3]. Although these software programs may
function flawlessly on a touch-based device, many are not practical due to the
usage of small buttons, fonts and very complex menu systems. This paper
describes the approach taken to overcome this problem when encountered by a
team of students at the School of Health Information, University of Victoria
(UVic).
1. Background
The project began as a team assignment in a graduate level database design course
offered at the School of Health Information Science, UVic. This course is one
requirement of the MSc degree in Health Informatics, a unique program in Canada.
The four student team members came from a variety of backgrounds and
experiences, all with strong clinical backgrounds but only one with technical
expertise. None of the team members had experience programming in SQL though
skills were developed throughout the course with the guidance of the instructor,
Dr. M.H. Alex Kuo. This course ran for just over a three-month period, from
January until the beginning of April
2012.
In the class, teams were instructed to design and build an electronic medical
record (EMR) application using Oracle 10g/XE and the application builder in Oracle
Application Express (APEX). For this project, it was decided to design the
application to be used in the Aging Brain and Memory Clinic (ABMC) at Parkwood
Hospital, in London Ontario Canada. This clinic’s primary function is to assess
referred geriatric patients who have complaints of memory impairment.
Through a needs-assessment conducted in the ABMC, tablet computers were
identified as the preferred primary clinical data-capturing device, with a desktop
version also being required. Currently, the data recorded in ABMC clinical
assessments is completed on paper as no electronic tool has been found which
meets the needs of such a specialized geriatric clinic. The preference for staff to
use a tablet device was not surprising, as mobile computing is fast becoming one
of the main methods in which electronic information is utilized today. Some of the
newest and most popular computer tools on the market are not provided or
supported by hospitals, but yet are still being carried by the physicians, residents,
fellows, and nurses [4].
The omnipresence of powerful handheld communication devices, such as
smartphones and tablets, are fundamentally changing both the quantity and quality of
information available to clinicians at the point of care [5]. These mobile devices
are far more than storage devices; they are sophisticated tools with beautiful full-
color screens, high-speed internet access, and offer users the ability to
communicate, via voice or text, to anyone in the world. Tablet computers have
become noticeably popular as physician adoption for professional purposes nearly
doubled in the United States between 2011 and 2012, reaching 62% according to
the Taking the Pulse survey [5]. In addition, the study stated that one-half of
240 J. Lindsay et al. / Moving Mobile: Using an Open-Sourced Framework
tablet-owning physicians, who participated in the national survey, have used their
device at the point-of-care.
Due to the increasing demand for mobile information, the team began to
research options for enabling an Oracle APEX application to be used conveniently
on a tablet device with a touch-based interface. Team members first came across
the Internet blog of Marc Sewtz, a senior software development manager at Oracle
in APEX [6]. The blog post was entitled Getting started with mobile in apex and
familiarized team members with the jQuery Mobile open-source user interface
framework. As summarized in the jQuery Mobile documentation, jQuery Mobile is
“a unified user interface system that works seamlessly across all popular mobile
device platforms”[7]. Through careful examination, the jQuery Mobile framework
was selected as the user interface for the mobile version of the EMR application.
2. Methods
Dr. Michael Borrie, D irector of the ABMC, provided the team with the paper based
assessment tools and clinic forms which were used to create the dataset for the EMR
system. The dataset was normalized and then each table was programmed into
Oracle 10g/XE in the SQL Workshop in APEX. The latest version of APEX
(version 4.1) for this project, however version 3 and above would have met the basic
requirements to complete the project while allowing for the use of the jQuery
Mobile framework.
The core application, entitled mEMRy, was created for use as a desktop
version of the EMR and was able to perform the features that were required by
the clinic staff. These features included a modular design, being able to capture
required data fields, the ability to database the images of completed cognitive tests,
and the generation of summary reports based on the data collected in clinic visits.
Using the desktop version of mEMRy as a framework, the team began to create
new APEX templates, which allowed the standard APEX user interface to be replaced
by the jQuery Mobile interface.
In APEX, the way in which data is displayed on the screen is determined by the
selected template. For jQuery Mobile to be enabled and used as the user interface in
the application, mobile templates needed to be created for every data element chosen
for display on a page. Figure 1 shows a storyboard illustration of the initial
search page and is labeled to indicate the type of APEX element along with the name
of the template used to enable the element for use with the jQuery Mobile interface.
Templates were created with the awareness of the ergonomics of touch screen
use and then tested using a trial and error process [2, 3, 8]. Team members tested
and took observations of the mobile and desktop versions of the mEMRy
application on the iPad 3, Samsung Galaxy Tab, Windows XP desktop computer
(1024x768 resolution display), Windows 8 Tablet (Acer Iconia T ab W500), and an
Apple Mac Mini (1920x1080 resolution display). As the development of these
applications was completed for academic purposes, no formal evaluation or usability
assessment was conducted on either version of the application.
Figure 1. Creating templates to enable jQuery Mobile in Oracle APEX
3. Results
Once the final prototype version of the mEMRy mobile EMR had been
completed, 41 new mobile APEX templates had been created. Of those, 22
templates were used in the application with 2 being button templates, 5 page
templates, 8 region templates and 7 report templates. The desktop and mobile
versions of mEMRy are featured in figure 2 respectively. When viewing both the
mobile and desktop versions of mEMRy on a tablet device, a number of interesting
points were observed. In the desktop version on both the iPad 3 and Samsung
Galaxy Tab, text and visual elements overlapped one another consistently. Screen
orientation needed to be locked in order to prevent the page from being reoriented
when the tablet was rotated. Many buttons were difficult to activate by touch and
were found to be much smaller than the 20mm size recommended by ergonomists [8].
Text based links were also found to be difficult to select and activate.
Figure 2. The desktop (right) and mobile (left) versions of the mEMRy application.
242 J. Lindsay et al. / Moving Mobile: Using an Open-Sourced Framework
In using the mobile application, it became quickly apparent that there was a
conflict between the jQuery Mobile framework and APEX. If page transitions were
enabled, page links seemed to act erratically with some links functioning correctly,
others becoming inactive, and others pulling information from linked pages and
displaying it on top of the previous page content. Through troubleshooting and
consulting the jQuery Mobile source documents, it was found that page transitions
operated using Asynchronous JavaScript and XML (Ajax) could be disabled with a
simple command in the header of the page template. In the upcoming release of
Oracle APEX 4.2, jQuery Mobile is planned on being fully supported, with a base
mobile theme included in the package, and full support for Ajax [9]. Once Ajax was
disabled, the jQuerly Mobile framework was successfully incorporated into APEX as
the user interface. All pages displayed properly on every device and no limitations in
performance were observed.
4. Discussion
As technology advancement continues at a staggering pace, it is essential to ensure

that older technology can continue to be used on new device platforms. One of the
major benefits to using web-based technologies is the fact that despite of
advancements and development in web-based programming languages, web
browsers continue to provide legacy support for older versions of web-based
languages. Another key point is the fact that web content, excluding select content
such as Adobe Flash, is universally accessible across most computer platforms.
Computer users can securely access web sites from a desktop computer at home or
work, a kiosk, gaming console, smartphone, tablet computer or even a web enabled
television. With computer hardware technology changing so quickly, it only makes
sense to continue to embrace web-based computing and consider it as the future
universal standard platform of computer information systems used in healthcare. For
instance, one could easily imagine the application of a touch screen high
definition surgical monitor that is web enabled, allowing not only high definition
video from an endoscopic camera to be displayed, but also the display and
interaction with relevant patient and health related data.
Another important point that has arisen through this project is the use of an open
sourced framework such as jQuery Mobile to ensure that tools developed using
previous technologies can function and remain supported on newer devices. The
jQuery Mobile framework was incorporated into an Oracle Apex database
application by a team with little experience, with no associated financial cost, and
in a very short period of time. This illustrates the ease with which a user interface
can be programmed in a web-based environment. By including platforms such as
jQuery Mobile, accessibility is opened to virtually all commonly used devices and
operating systems. It is expected that this shift, partnered with innovation, will
continue as a trend for developers and leaders in both the software and healthcare
industries.
References
[1] Gold J. For Hospitals, There’s No App For That, Kaiser Health News. [Internet]. 26 Dec 2011. [cited 8
Sep 2012]. Available from http://www.kaiserhealthnews.org/Stories/2011/December/26/Hospitals-

iPads.aspx
[2] Hoffmann ER, Sheikh IH, Finger width corrections in Fitts’ law: implications for speed-accuracy
research, J. Mot. Behav. 1991;23 (4):259–262
[3] Lee YL. Comparison of the conventional point-based and a proposed finger probe-based touch screen
interaction techniques in a target selection task, Int. J. Ind. Ergon. 2010;40(6): 655-662
[4] Savel RH, Munro CL. Scalpel, Stethoscope, iPad: The Future Is Now in the Intensive Care Unit, Am. J.
Crit. Care.2011;20 (4):275-277
[5] Vecchione A. Doctors' Tablet Use Almost Doubles In 2012, InformationWeek. [Internet]. 16 May
2012.[cited 10 Sep 2012]. Available from http://www.informationweek.com/healthcare/mobile-
wireless/doctors-tablet-use-almost-doubles-in-201/240000469
[6] Sewtz M. Getting Started With Mobile in APEX, Mark Sewtz. [Internet] 12 Nov 2011. [cited 8
Sep2012]. Available from http://marcsewtz.blogspot.ca/2011/11/getting-started-with-mobile-in-
apex_17.html
[7] jQuery Mobile: Demos and Documentation, jQuery Foundation. [Internet] 2012. [cited 8 Sep 2012].
Available from http://jquerymobile.com/demos/1.1.1/docs/about/intro.html
[8] Colle HA, Hiszem KJ. Standing at a kiosk: Effects of key size and spacing on touch screen numeric
keypad performance and user preference. Ergonomics 2004;47 :1406–1423.
[9] Oracle Application Express 4.2 Early Adopter Feature Descriptions, Oracle. [Internet]. 2012. [cited 9
Sep 2012]. Available from http://apex.oracle.com/pls/apex/f?p=38997:1
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-244
The Use of Social Media in Healthcare:

Organizational, Clinical, and Patient
Perspectives
Mowafa HOUSEH a,1
a
Abstract. The purpose of this review paper is to explore the impacts of social
media on healthcare organizations, clinicians, and patients. This study found that
healthcare organizations, clinicians and patients can benefit from the use of social
media. For healthcare organizations, social media can be used primarily for
community engagement activities such as fundraising, customer service and
support, the provision of news and information, patient education, and advertising
new services. The study also found that the most widely used social media venues
for physicians were online communities where physicians can read news articles,
listen to experts, research new medical developments, network, and communicate
with colleagues regarding patient issues. Patients can benefit from the use of social
media through education, obtaining information, networking, performing research,
receiving support, goal setting, and tracking personal progress. Future research
should further examine other financial, technological, informational, ethical, legal,
and privacy issues surrounding the use of social media in healthcare.
Keywords. consumer informatics, Saudi Arabia, health websites, patient

empowerment.
Introduction
The use of social media by healthcare institutions, clinicians, and the public has
increased over the past few years. This use of social media is part of a growing trend
and is due to a realization that healthcare institutions and clinicians need to be more
engaged with their patients. Social media provides an online platform for interactions
to occur around various health topics relating to patient education, health promotion,
community outreach, public relations, and crisis communication [1]. Social media
includes various approaches such as blogs, microblogging (e.g., Twitter), social
networking (e.g., Facebook and Patients Like Me), video and file sharing (e.g.,
YouTube), e-games, and wikis. Despite the use of social media related to healthcare,
the effects of social media on patient care have not been well documented, and with
1
Corresponding Author: Dr. Mowafa Househ, King Saud Bin Abdul Aziz University for Health
Sciences, College of Public Health and Health Informatics, Riyadh, Kingdom of Saudi Arabia; Email:
[email protected]
M. Househ / The Use of Social Media in Healthcare 245
over 80% of Americans seeking health information online [1], the impacts of social
media on health cannot be ignored.
With the widespread use of the internet and the relatively low cost of bandwidth,
social media, especially social networking, is being adopted by healthcare professionals
and patients. There has been growth in the use of social media, especially online social
networking, related to healthcare. Perhaps one of the earliest studies carried out to
assess the use of social media in healthcare focused the use of the Bulletin Board
System for the education of nurses [2]. That study found that the major impediments to
implementing such a system were cost, maintenance, and the lack of training of nurses.
In a commentary published by the Journal of the American Medical Association,
Shachak and Jadad argue that the use of social networks will eventually lead to a more
patient-centered healthcare system that will improve communication and information
flow between patients, providers, and administrators [3]. Other studies have shown that
Facebook users seek health information [4], and social networking websites, such as
Patients Like Me, are beginning to cater to the need for social networks devoted to
providing health information [5, 6]. With the growing use of social media in healthcare,
the purpose of this paper is to review the impacts of social media on healthcare
organizations, clinicians and patients.
1. Organizational perspective
It is estimated that approximately 70% of U.S. healthcare organizations use social

media as part of various community engagement activities, such as fundraising,
customer service and support, the provision of news and information, patient education,
and advertising new services, with Facebook and YouTube being the most widely used
[1, 7]. Social media has allowed healthcare institutions to increase visibility and
improve their overall image. As a result, 12.5% of surveyed American healthcare
organizations have successfully attracted new patients through the use of social media
[8]. In addition, there seems to be an increasing rate of social media adoption by non-
profit healthcare organizations [9]. In 2010, the Mayo Clinic established the Social
Media Health Network with a stated vision of being “the authentic voice for patients
and healthcare professionals, building relationships through the revolutionary power of
social media.” There are currently over 100 members affiliated with the Mayo Clinic
initiative, and some of their broad initiatives include blogs, Twitter posts, conferences,
and webinars to engage various community stakeholders.
One of the main challenges for healthcare organizations is related to understanding
the meaningful use of social media sites by patients. Merely visiting a Facebook page
or viewing a YouTube video does not signify meaningful use. Higher levels of
interaction are needed between the patient and the healthcare institution, and studies to
evaluate such interactions should be conducted. Other challenges include dedicating
resources to design, maintain, advertise, and update social media sites. Healthcare
organizations have not made sufficient investments in this area, and some are
beginning to abandon or neglect their social media sites [7]. Yet, the potential for
community engagement through social media remains an opportunity for healthcare
organizations to engage their communities and market their services.
246 M. Househ / The Use of Social Media in Healthcare
2. Physician perspective
Physicians have also been users of social media for both professional and personal
interests. It is estimated that 65% percent of physicians use social media for
professional purposes [10]. The most widely used social media venues for physicians
are online communities where physicians can read news articles, listen to experts,
research new medical developments, network, and communicate with colleagues
regarding patient issues [10]. Physicians rarely use social media sites to communicate
with patients. In fact, the American Medical Association has advised physicians to
maintain “appropriate boundaries of the patient-physician relationship” and has
recommended that physicians should “consider separating personal and professional
content online.” A study documenting patient privacy violations on the Facebook
profiles of medical students and residents found that medical students were more likely
than residents to violate privacy, especially when working in medical missions in
developing countries [11]. The study concluded that medical professionals have a
responsibility to use social networking sites in an ethical manner that does not violate
patients’ privacy rights. However, physicians are beginning to develop an interest in
interacting with patients online. Approximately 60% of physicians favor interactions
with patients through social media, showing strong support for patient education,
followed by monitoring patients’ health, behavior and drug adherence and giving care
advice to patient groups that would lead to “better education, increased compliance,
and better outcomes” [10]. The challenge in the future will be for physicians to develop
guidelines regarding how to interact with patients through social media websites in a
manner that is ethical and that does not violate patients’ privacy rights.
3. Patient perspective
Patient groups have benefited the most from the use of social media for health
purposes. It is estimated that in the United States, 74% of all internet users utilize social
media [12], and searching for health information online has become one of the most
popular online activities [1]. Patients can use social media for a variety of reasons that
include education, obtaining information, networking, research, support, goal setting,
and tracking personal progress. Patients have the ability to express themselves, share
their stories, learn from others and spread health knowledge [10]. Social media creates
a forum for patient participation that extends beyond the reach of the hospital or the
local clinic. These forums can help empower and uplift patients when they read the
experiences of other patients [10]. Some physicians believe that social media can be
beneficial for patients with chronic diseases, cancer, rare diseases, and depression and
for patients with questions or goals related to maternal and infant care, wellness and
prevention, and weight management [10].
The introduction of social media sites such as Facebook, Twitter, and YouTube
has revolutionized the way individuals seek, share and use information, specifically
health information. A recent study on the sharing of sensitive health information
through Facebook found that Facebook users openly sought and shared behavioral,
mental, and genetic information [6]. An interesting finding from this study was that
most Facebook users included in the study publicly identified themselves by divulging
their name, photo, and location when seeking sensitive health information through
online postings. The author of the study suggested that the growing use of social media
M. Househ / The Use of Social Media in Healthcare 247
and the sharing of personal information may be changing society’s perceptions on

privacy from the strict implementation of privacy to a policy that is more lenient.
Research is starting to show that there are positive impacts of the use of internet-
based interventions, including interventions based on social media, on weight loss,
tobacco cessation, and physical activity. Studies have also found that different age
groups, income groups, and gender groups use social media in different and similar
ways. For example, Korda et al. note that women are more likely than men to look for
information relating to symptoms, treatments, diseases and conditions, and medication,
whereas men are more likely to search for information about vitamins and supplements,
health insurance providers, and physicians [1]. The benefits to the patient in connecting
with other groups and communities are understandable, but many challenges with
moving forward remain.
4. Discussion
Various challenges arise when social media is used by healthcare institutions,

physicians, and patients. For healthcare institutions, reaching out to the community is
an activity that requires more resources, and more research is needed to determine the
impacts of the use of social media on healthcare outcomes, the quality of healthcare
services, and revenue or costs. Any conclusions based on the limited and anecdotal
evidence available are premature. More robust evidence is needed to justify the case for
healthcare organizations making social media an integral part of their healthcare
service delivery models. For physicians, there needs to be a very strict separation of
their personal identities and their professional identities when dealing with patients and
colleagues. Many issues arise, especially when interacting with patients, regarding
compensation, privacy (both patient and physician), and the regulation of such
interactions. Although physicians realize that using social media to communicate with
patients has potential benefits, the evidence regarding the effects on health outcomes
remains premature and is mired by ethical and professional considerations.
Regarding patients, the primary threat when seeking health information through
social media is the intentional or unintentional spread of false information that could
harm a patient. Another threat is the adverse impacts of sharing sensitive personal
health information online, such as genetic, sexual, and psychological information, as
this information can be traced back to the individual if he/she does not take the proper
precautions [11].
5. Conclusions
Moving forward, healthcare organizations, policy makers, physicians, and patient

advocates should collaborate and communicate regarding the development of social
media platforms as part of a healthcare model that addresses the needs of all
stakeholder groups. Further studies should focus on the utility of social media in
healthcare. These activities are needed to improve and ensure the safe use of social
media related to healthcare information [7]. In one study, most patients discussed the
health information found online with their physician and believed that sharing such
information improved the patient-physician relationship [2].
248 M. Househ / The Use of Social Media in Healthcare
Future research should further investigate other financial, technological,

informational, ethical, legal, and privacy issues surrounding the use of social media in
healthcare. There are many privacy issues, for example, when sharing information
online or using social networking data for research. There are also technological issues,
such as usability and network security, when using social media. There are legal and
ethical issues surrounding clinical communication with patients. There are also
financial issues regarding the compensation of physicians and revenue generated from
social media websites focused on health. Regarding information issues, misinformation
or disinformation may harm patients who seek health information through social media
sites.
More research needs to be performed, and policies and guidelines that can help
patients understand the impacts of using social media to obtain health information
should be developed. Additionally, healthcare organizations, as well as clinicians,
should take the lead in providing healthcare information that is reliable, credible, and
trustworthy.
References
[1] H. Korda, Z. Itani, Harnessing Social Media for Health Promotion and Behavior Change, Health
Promotion Practice (2011): 1-9.
[2] M.M. Russin, J.H. Davis, Continuing education electronic bulletin board system: provider readiness and
interest, J Contin Educ Nurs 21(1) (1990),23-7.
[3] A. Shachack, A. Jadad, Electronic Health Records in the Age of Social Networks and Global
Telecommunications, JAMA 303 (5) (2010), 452-453.
[4] M. Househ, Sharing Sensitive Personal Health Information through Facebook: the Unintended
Consequences, Stud Health Technol Inform 169 (2011), 616-20.
[5] M. Househ, Re-examining Perceptions on Healthcare Privacy - Moving from a Punitive Model to an
Awareness Model, HEALTHINF 2012 - Proceedings of the International Conference on Health
Informatics, Vilamoura, Algarve, Portugal, 1 - 4 February, 2012. SciTePress, 2012, 287-291.
[6] M. Househ, Mobile Social Networking Health (MSNet-Health): Beyond the mHealth Frontier, Stud
Health Technol Inform 180 (2012), 808-12.
[7] Social media use in healthcare for quality improvement, [cited Jun 21, 2011], [updated Jan 15, 2012],
Available from: http://tcf.innovationcell.com/book/archive/2010/10/general-use-social-media-
healthcare-organizations.
[8] Greystone Net, [cited Jan 25, 2012], [updated Jan 18, 2012], Available from:
http://www.greystone.net/wtn/Page.asp?PageID=WTN000018.
[9] N. Hum, et al. A picture is worth a thousand words: A content analysis of Facebook profile photographs,
Computers in Human Behavior (2011) 1828-1833.
[10] M. Modahl, L. Tompsett, T. Moorhead, Doctors, Patients and Social Media [cited Sep, 2011], Available
from: http://www.quantiamd.com/q-qcp/doctorspatientsocialmedia.pdf.
[11] L. Thompson, et al, Protected Health Information on Social Networking Sites: Ethical and Legal
Considerations, Journal of Medical Internet Research (2011): e8.
[12] P. Eckler, G. Worsowicz, W. Rayburn, Social Media and Health Care: An Overview, Practice
Management (2010), 1046-1049.
Evaluation
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-251
POND4Kids: A Global Web-based

Database for Pediatric Hematology and
Oncology Outcome Evaluation and
Collaboration
Yuri QUINTANAa, 1, Aman N. PATEL a , Magada ARREOLAc , Federico G.
ANTILLON c, Raul C. RIBEIRO a,b, Scott C. HOWARD a,b
a
International Outreach Program, St. Jude Children’s Research Hospital
b
Department of Oncology, St. Jude Children’s Research Hospital, Memphis, TN
c
Unidad Nacional de Oncología Pediátrica, Guatemala City, Guatemala
Abstract. The Pediatric Oncology Network Database, (www.pond4kids.org,

POND), is a secure, web-based, multilingual pediatric hematology/oncology
database created for use in countries with limited resources to meet various clinical
data management needs including cancer registration, delivery of protocol-based
care, outcome evaluation, and assessment of psychosocial support programs.
Established as a part of the International Outreach Program at St. Jude Children’s
Research Hospital in Memphis, Tennessee, POND serves as a tool for oncology
units to store patient data for easy retrieval and analysis and to achieve uniform
data collection to facilitate meaningful comparison of information among centers.
Launched in 2003, POND now has 233 sites registered with over 1,000 users in 66
countries. However, adoption and usage of POND varies widely among sites. This
paper reviews some of the challenges to developing a global collaborative clinical
platform based on the experiences of developing POND. The paper also presents a
case study of POND use in Guatemala, where the Guatemalan National Oncology
Unit (UNOP) has developed extensive internal and external global collaborations
using POND.
Keywords. cancer registry, pediatric oncology, international, collaboration,

clinical improvement, clinical informatics, e-health, data management programs
Introduction
Developed by International Outreach Program at St. Jude Children’s Research Hospital,

the St. Jude Pediatric Oncology Networked Database (POND, www.POND4Kids.org)
is a secure, online, multilingual database for pediatric hematology/oncology patients.
Its purpose is to improve the care of pediatric oncology patients in countries with
limited resources by the exchange of information and experience between oncologists
in diverse geographic regions who practice in a similar medical environment [1-4].
The major objectives of POND are to:
1. Allow oncologists to store, share, and control access to patient data.
2. Provide uniform data collection to facilitate meaningful comparisons
1
Corresponding author
252 Y. Quintana et al. / POND4Kids: A Global Web-Based Database
Total Number of logins to POND4Kids

Number of Patients Records in the
5000 POND Guatemala Site
5000
4000 4500
4000
3000 3500
3000
2000 2500
2000
1000 1500
1000
0 500
0
2003-07
2004-04
2005-01
2005-10
2006-07
2007-04
2008-01
2008-10
2009-07
2010-04
2011-01
2011-10
2012-07
2004-05
2005-01
2005-07
2006-01
2006-07
2007-02
2007-08
2008-02
2008-08
2009-02
2009-08
2010-02
2010-08
2011-02
2011-08
2012-02
2012-08
Figure 1.Total Number of Logins (all sites) Figure 2. Records in POND Guatemala
3. Provide educational support for data collection, quality control, and analysis.
4. Provide a multi-lingual system that supports data sharing and comparison in a
universal format while allowing data entry and reporting in the user’s native
language.
This paper presents a technical overview of the system, reviews global usage of
POND, identifies some of the barriers and facilitators for adoption and usage of POND,
and provides an example of successful POND use at the National Pediatric Oncology
Unit of Guatemala, and highlight on-going challenges.
1. POND4Kids Systems Design
POND4Kids allows uniform data collection so that clinical data can be shared and
meaningfully compared between centers. POND4Kids uses a centralized web server
(Linux, MySQL, and PHP) that can host multiple virtual sites. Each site has an
administrator that can add users and control the level of access of each user. Data can
be shared with individuals at another POND site, and sharing can be deactivated at any
time. By design, personal information such as names cannot be shared between sites,
such that shared data always complies with HIPAA privacy rules in the US, which are
similar to privacy rules in many other countries. The intent is to allow data about a
particular disease to be shared with a disease expert to assist with analysis and
development of strategies to improve outcomes. Users can create Quick Data Entry
forms that can be used within a POND site or shared with other sites via a common
library. Protocols templates can also be created and used locally, or shared with other
sites via a common library. Using the online meeting capabilities of the Cure4Kids
website (www.Cure4Kids.org), training is provided on best practices for clinical data
collection and clinical quality improvement programs. Additional details of the POND
system can be found in [4-6]
Y. Quintana et al. / POND4Kids: A Global Web-Based Database 253
2. Outcomes
Since its launch in 2003, POND use has increased steadily, with use spreading only by
word of mouth in the pediatric hematology/oncology community. As of September 12,
2012, POND has 1,222 users from 233 sites in 66 countries. There are over 50,000
records and 800 collaborative data shares and 4,000 logins each month (Figure 1).
POND Reports Generated (all sites) POND Reports Generated (Guatemala)
2000 900
800
1500 700
600
500
1000
400
300
500 200
100
0 0
2005-12
2006-06
2006-12
2007-06
2007-12
2008-06
2008-12
2009-06
2009-12
2010-06
2010-12
2011-06
2011-12
2012-06
2005-12
2006-06
2006-12
2007-06
2007-12
2008-06
2008-12
2009-06
2009-12
2010-06
2010-12
2011-06
2011-12
2012-06
Figure 3. Total Number of POND reports. Figure 4. Number of Guatemala Reports.

In Guatemala, a pediatric oncology clinic called Unidad Nacional de Oncología
Pediátrica (UNOP) was established with a focus on pediatric cancer. Given the limited
government resources, a non-profit foundation called Fundación Ayúdame a Vivir was
founded in May 1997 to provide the necessary resources for the proper maintenance,
operation, and growth of UNOP as a center of excellence for treating pediatric cancer
in Guatemala. Each year, about 400 children newly diagnosed with cancer are treated at
UNOP. Survival rates for some types of cancer have increased from 28 percent in 1996
to 70 percent in 2012. Fundación Ayúdame a Vivir subsidizes the treatment for patients
who cannot pay. The rate of abandonment of treatment has decreased mostly due to
psychosocial intervention, and the provision of housing and transportation. The use of
POND by UNOP was initiated in 2004 and has grown consistently with the support of
five dedicated data managers. The UNOP clinic has entered data for both current and
past patients. As of September 12, 2012 UNOP POND Site had 4748 records of
patients (Figure 2). The UNOP site is currently the most active of all POND sites.
UNOP accounts for almost half the logins to POND each month. Over the years the
number of active data shares has increased as UNOP expands its collaborations with
experts in specific diseases at other pediatric cancer centers. UNOP accounts for more
than half of the 800 data shares among all POND sites, and UNOP participates in
collaborative clinical protocols with many other Latin American countries. Over 1000
reports are generated each month (Figure 3) by all POND sites and UNOP accounts for
over 700 of them (Figure 4). These reports are used widely by a multi-disciplinary team
of doctors, nurses and allied health professionals at the clinic for patient care,
administration, and outcome evaluation. The increased usage and analysis of clinical
data is a contributing factor to the increase in survival rates at the UNOP clinic.
3. Discussion
POND provides clinical tools that fill an unmet need at many pediatric oncology
hospitals, particularly those that have common protocols and wish to share information.
There are several reasons for the success of POND at some sites, and lack of use at
other sites. Challenges include training, adequate data manager staffing, and quality
management. Availability of online training and support is one of the factors that has
helped the deployment of POND but at times it can be difficult due to limited
bandwidth and lack of Internet reliability. POND is sufficiently easy to use with
minimal training for most users. Online help and training are available in English and
Spanish. However, data managers need to have computer experience, and some may
need additional supervision and guidance. A regional network of POND data managers
was formed in Central America that allows for continued training and mutual support
and encouragement. The UNOP site in Guatemala has provided regional training
support for other Latin American centers both online and by hosting data managers
from other Spanish-speaking sites when they are starting their data management
programs.
The availability of appropriately trained data mangers and funding for salaries is a
major challenge for many sites. Lack of consistent funding for data managers at some
sites has led to inconsistent data entry. Since 2004 a grant from the Pediatric Oncology
Group of Ontario (Canada) and financial support from St. Jude Children’s Research
Hospital (Memphis, TN) have funded full-time data managers in several Central
America countries, and these tend to be the sites with most active use. Other sites do
not have sufficient funding to hire data managers, and the clinical staff does not have
the time to enter records. The UNOP site now has 5 data managers that facilitated the
widespread use of POND by doctors, nurses, nutritionists, and psychosocial support
staff.
High-quality data is the cornerstone on which quality improvement rests, so POND
has specific quality reports that identify the most common data entry errors, including
unlikely and impossible values [4]. However, these reports need to be regularly run and
acted upon to improve areas of missing or incomplete data. There may also be a lack of
standard operating procedures, inconsistent data collection methods, or missing paper
records. More training and on-site audits could facilitate additional improvement in
data quality and efficiency. Recent additions to POND include error-checking tools that
show missing or possible out of range data. UNOP also has implemented a daily review
all data entered in POND in the previous day that has greatly improved data quality.
Security is an important part of any medical records system. The POND software
resides in a dedicated server with a security firewall. Users with administrator
credentials manage access to data by users of the site and control sharing with external
collaborators. Although the data are encrypted and password protected, some
administrators are reluctant to store data outside their institution. The location of the
POND server in the United States has provided some concern to some sites. Future
development of a local POND system would facilitate use by those sites that do not
wish their data to reside on foreign servers.
To obtain benefit from the data-sharing capabilities of POND, access to clinical
experts is essential. Common treatment protocols used by pediatric oncology units in
Central America were created in POND. Principal investigators of shared protocols
have regular virtual meetings to review individual patients with complicated medical
problems, protocol data, and administrative issues, and to develop strategies to improve
Y. Quintana et al. / POND4Kids: A Global Web-Based Database 255
clinical outcomes. These clinical meetings are held online on www.Cure4Kids.org, a

web-based education and conferencing platform [5]. UNOP regularly participated in
multiple online meetings per week to discuss clinical cases that are in POND.
POND is upgraded regularly in response to feature-specific requests from users.
Developing features that meet the priorities of each site remains a challenge not only
because of the volume of requests, but also because countries have different data
standards, different systems of units for common laboratory values, and other
national/regional particularities that must be stored both in the original value and
converted to other values when comparing data. Some sites also suffer from slow or
unreliable Internet connections, and POND currently lacks offline functionality.
4. Conclusions
POND4Kids provides pediatric hematology/oncology units with platform for data

collection and outcome evaluation that facilitates local evaluation and multicenter
collaboration. Despite many of the challenges that developing countries [1,2,3,7] face
in health care, many sites in developing countries have been able to use POND to
collect data for clinical improvement. The UNOP site is one of the most active POND
sites, and its staff has been able to overcome many of the challenges related to funding,
training and data management. Further efforts are still needed in the areas of standard
operating procedures for data collection, data entry, data quality monitoring, and data
analysis. The increasing usage of data shares shows that global collaborations can
occur between low income countries as well as between low and high income countries.
Protocol template forms have allowed consistent data collection and enabled
comparison of data across multiple countries. Future development plans include
improvements in usability, more flexible reports with additional customization options,
support for additional languages, enhanced statistical analysis, the ability to enter and
access data while offline, and synchronization across a network of online and offline
POND4Kids servers. Future challenges include the prioritization of requests for new
features and the need for common data standards for comparisons among centers.
Acknowledgments
We thank support from the American Lebanese Syrian Associated Charities (ALSAC)
Cancer Center Support (CORE) grant CA-21765 from the National Cancer Institute,
the Pediatric Oncology Group of Ontario, and a grant from Johnson and Johnson Inc.
(Centocor and Janssen divisions).
References
[1] S. Howard, M. Metzger, Y. Quintana, CH Pui, L. Robison, R. Ribeiro, Childhood cancer epidemiology
in low-income countries, Cancer 112 (2008), 461-472.
[2] S.C. Howard, M. Marinoni, L. Castillo, M. Bonilla, G. Tognoni, F. Antillon, M.G. Valsecchi, C.H. Pui,
R.C. Ribeiro, A. Sala, R.D. Barr, G. Masera, MISPHO Consortium. Improving outcomes for children
with cancer in low income countries in Latin America, Pediatric Blood Cancer 48 (2007), 364-369.
[3] M. Valsecchi, E. Steliarova-Foucher, Cancer Registration in developing countries: luxury or necessity?
The Lancet Oncology 9 (2008),159-167.
[4] L. Ayoub, L. Fu, A. Pena, J.M. Sierra, P.C. Dominguez, C.H. Pui, Y. Quintana, A, Rodriguez, R.D. Barr,
R.C. Ribeiro, M.L. Metzger, J.A. Wilimas, S.C. Howard. Implementation of a data management
program in a pediatric cancer unit in a low income country, Pediatric Blood Cancer 49 (2007), 23-27.
[5] Y. Quintana, A.N. Patel, P.E. Naidu, S.C. Howard, F.A. Antillon, R.C. Ribeiro, POND4Kids: A Web-
based Pediatric Cancer Database for Hospital-based Cancer Registration and Clinical Collaboration,
Studies in Health Technology and Informatics 164 (2011), 227-31.
[6] Y. Quintana, R. O’Brien, A. Patel, J. Becksfort, A. Shuler, A. Nambayan, D. Ogdon, G. Chantada, S.C.
Howard, R.C. Ribeiro, Cure4Kids: Research challenges in the design of a website for global education
and collaboration, Information Design Journal 16 (2008), 243-249.
[7] E. Supriyadi, P.H. Widjajanto, I.Purwanto, J. Cloos, A.J. Veerman, S. Sutaryo, Incidence of childhood
leukemia in Yogyakarta, Indonesia, 1998-2009. Pediatr Blood Cancer. 57(4) (2011), 588-93.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-257
Knowledge Translation in eHealth:

Building a Virtual Community
Jesdeep BASSI a,1, Francis LAU a, Simon HAGENS b, Chad LEAVER b and Morgan
PRICE c
a
School of Health Information Science, University of Victoria
b
Canada Health Infoway
c
Department of Family Practice, University of British Columbia
Abstract. Knowledge can be powerful in eliciting positive change when it is put

into action. This is the belief that drives knowledge translation. The University of
Victoria (UVic) eHealth Observatory is focused on deriving knowledge from
health information system (HIS) evaluation, which needs to be shared with HIS
practitioners. Through an application of the Knowledge-to-Action Framework and
the concept of a virtual community, we have established the virtual eHealth
Benefits Evaluation Knowledge Translation (KT) Community. This paper
describes the foundational elements of the KT Community and our overall KT
strategy.
Keywords. knowledge translation, virtual community, health information system
Introduction
A common saying is that ‘knowledge is power’. However, knowledge without

application can be powerless to invoke change. The gap between knowledge and
practice needs to be closed to turn knowledge into action [1, 2] to make it powerful.
Use of information technology in healthcare is on the rise. In 2007, the National
Physician Survey found that 12.3% of Canadian family practitioners and general
practitioners used electronic medical records (EMR) exclusively and 19.4% used a
combination of EMRs and paper-based charts [3]. These figures rose to 21.5% and
27.5% respectively in 2010 [4]. For eHealth adoption in Canadian hospitals, the
Ontario Hospital Association found the percentages of hospitals at Stages 1 through 6
of the Healthcare Information Management Systems Society’s (HIMSS) Electronic
Medical Record Adoption Model (EMRAM)SM rose from 2010 to 2012 [5]. The
potential benefits of such systems are driving adoption. However, adoption does not
ensure benefits are realized [6]. Thus, there is a need to ensure benefits are being
achieved and that demonstrated achievements are shared and applied broadly. This is
the idea behind eHealth benefits evaluation. Organizations need to measure and
demonstrate the impacts systems are having which requires knowledge of the
methodologies and tools available for evaluation. However, this knowledge is often
held in large organizations or academic settings and is not readily available to those
who need it. To address this gap we initiated a knowledge translation (KT) activity to
1
258 J. Bassi et al. / Knowledge Translation in eHealth: Building a Virtual Community
increase capacity in conducting eHealth benefits evaluation. This paper describes the
theory behind KT and describes how it has been applied in the context of eHealth to
create a KT strategy.
1. Knowledge Translation Background
1.1. Defining Knowledge Translation
KT is “a dynamic and interactive process that includes synthesis, dissemination,

exchange and ethically sound application of knowledge to improve the health of
Canadians, provide more effective health services and products and strengthen the
health care system” [7]. Knowledge needs to be put into practice to be useful.
Additional terms have been used in the literature such as “knowledge transfer” and
“knowledge exchange”. In Canada, all three are used [2], however “knowledge
translation” is gaining prominence [8]. The Canadian Institutes of Health Research
(CIHR) definition encompasses knowledge transfer and exchange as well. According to
Reardon et al. [9], knowledge exchange is one model of knowledge transfer in which
relationships are built between knowledge producers and users to exchange information,
ideas and experience. This describes the intent of our KT activity.
1.2. Theoretical Basis: Knowledge-to-Action
In terms of KT theories CIHR has adopted the Knowledge-to-Action Framework

developed by Graham and colleagues [2]. This framework divides KT into two
concepts: knowledge creation and action [8]. Knowledge creation is shown as a cycle
containing a knowledge funnel whereby knowledge becomes refined through the
phases of knowledge inquiry, synthesis, and tool creation. Although it is primarily
research based, knowledge also includes experiential knowledge. The action cycle
represents activities for knowledge application which are to: identify problem; adapt
knowledge to local context; assess barriers to knowledge use; select, tailor, implement
interventions; monitor knowledge use; evaluate outcomes; and sustain knowledge use.
Collaboration between knowledge producers and users is inherent in this framework.
Our role as researchers is typically focused on the knowledge creation cycle but as the
framework shows, the boundaries between the two cycles are fluid. Information and
Communications Technology (ICT) can be applied throughout this framework and is
referred to as Technology-Enabled Knowledge Translation (TEKT) [1].
1.3 Knowledge Translation Components
To effectively translate knowledge in any context, a planned approach is required.

Numerous guides exist for preparing a KT plan. Barwick et al. [10] developed a
comprehensive KT planning template which guides planners through a series of steps
from identifying research partners to describing how the KT strategy will be
implemented. In a KT training workshop they identified five key pieces as goals,
audience, main message, strategies, and assessing impact. CIHR’s KT casebook for
end-of-grant KT [11] outlines the key components for a KT plan as: goals, audience,
J. Bassi et al. / Knowledge Translation in eHealth: Building a Virtual Community 259
strategies, expertise, and resources. Based on these guides, we describe our KT activity
below in terms of:
Goal: the purpose of sharing knowledge and its expected action
Message: the knowledge being shared
Audience: individuals and/or groups that should receive the knowledge
Strategy: approaches used to translate the knowledge
Expected Impact: effect of the KT activity
2. The KT Plan
2.1. Goal
In discussing healthcare innovation, Berwick stated “in healthcare, invention is hard,

but dissemination is even harder” (2003, p. 1970) [12]. He went on to say that the most
powerful driver of adopting innovation is the perceived benefit of change. A parallel
can be drawn for health information system (HIS) adoption. Systems have been
developed but adoption relies on potential users believing they will receive benefit.
Heathfield and Pitty [13] examined the roles of evaluation for creating accountability,
strengthening development, and creating knowledge. This is why benefits evaluation is
important. There is a need to demonstrate benefits and impacts to drive adoption and to
do this, one needs to know how to conduct an evaluation and assess the evidence to
apply it. Unlike previous work around knowledge translation in healthcare which has
been about translating research evidence on medical interventions and innovations, our
KT activity is not about translating evidence on the benefits of eHealth, but rather
translating the knowledge on how to determine the benefits of eHealth. As explained in
the introduction, the push behind this KT activity stems from a need to empower
individuals in conducting eHealth benefits evaluation. To do this we need to put the
knowledge and tools into the hands of evaluators who can apply them.
2.2. Message
In terms of knowledge creation, the UVic eHealth Observatory was created in 2009 to
monitor the effects of HIS deployment in Canada [14]. Researchers at the eHealth
Observatory have been collaborating with Canada Health Infoway to explore and
synthesize evidence around HIS evaluation. Five types of knowledge products have
been created: (1) conceptual eHealth evaluation models; (2) a rapid evaluation toolkit;
(3) systematic literature reviews; (4) educational materials, and (5) the results of
eHealth evaluation studies. For example, the Benefits Evaluation Framework specifies
a set of dimensions and categories for factors which influence HIS adoption [15]. A
knowledge product stemming from the Benefits Evaluation Framework is the updated
Infoway Benefits Evaluation Indicators Technical Report [16]. We are also aware of
other sources of knowledge that would be valuable to those working in this area. For
instance, the Agency for Healthcare Research and Quality in the United States has
developed the Health IT Evaluation Tookit [17]. This particular KT activity focuses on
the HIS domains of EMRs in physician offices, prescribing, and consumer health
systems such patient health portals, with data quality as an overarching area. Emphasis
is placed on the exchange portion of KT as we are stressing the need for individuals to
contribute their own knowledge and experiences as well. In this way, knowledge
creation and sharing is the message at the core of this KT activity.
2.3. Audience
Since this KT activity is to foster learning to increase capacity in and application of

benefits evaluation, the audience for our message is quite broad. Four target groups will
benefit most from this knowledge. The first includes individuals from healthcare
organizations who are directly involved with benefits evaluation or wish to learn about
them. This can include managers and staff from ministries of health, health authorities,
and so on. The second group includes HIS researchers who work to create evidence-
based knowledge in this area. They bring expertise in knowledge creation. The third
group consists of clinicians such as physician, nurses, and pharmacists who are the
adopters of HIS. They work on the front lines and directly experience the impacts. The
last group includes individuals from the private sector who wish to learn how their
products are evaluated in practice.
2.4. Strategy
In terms of methods to facilitate KT, there is a plethora of approaches available and

they do not need to be used in isolation. Barwick et al. [10] reviewed evidence for KT
strategies and found that combined interventions are mostly effective. In designing our
KT strategy we considered the goal, message and audience and determined the best
approach is the creation of a virtual eHealth benefits evaluation KT community to
facilitate the use of several KT strategies. Our concept of a web-based virtual
community borrows elements that have been described for virtual communities of
practice and virtual learning communities. They enable people to share their experience
and knowledge to improve their abilities and skills and foster learning [18].
McCartney et al. [19] performed a review of virtual communities of practice in
healthcare, identifying some characteristics for success. These include: being problem
driven; having organic involvement; diverse champions, facilitators or leaders; having
administrative support; and regular incentives to participate. While there is no specific
“problem” our virtual community is working towards solving, there is a clear goal of
sharing knowledge about benefits evaluation. We outline the goals and expectations for
participants in the terms of reference. People are not obligated to join yet do so because
of a shared interest in learning. From this group, we have identified mentors (including
our own team members) to act as leaders within the community. Since this is a web-
based virtual community, a technical infrastructure and support is needed. Specific
roles have been established within the eHealth Observatory team for administration.
The incentives to participate consist of the activities within the community.
Our KT Community offers five KT features. The first is a series of live online
sessions on benefits evaluation topics which are presented by mentors. The sessions are
scheduled and conducted using web conferencing software so that participants can join
in regardless of where they are. Didactic lectures have had mixed results in terms of
effectiveness for KT but interactive education sessions with participation have been
generally effective [9, 10]. Therefore each session also includes three key discussion
questions to be discussed in the session or afterwards. This links to the next feature of
the KT community which is an online discussion forum. Here, participants are able to
communicate with each other asynchronously. Currently there is limited evidence on
J. Bassi et al. / Knowledge Translation in eHealth: Building a Virtual Community 261
effectiveness of electronic communication for KT [9,10] so this project presents an

opportunity to contribute findings in this regard. Since many of our knowledge
products are in the form of reports and tools, we have made them available through a
repository on the KT Community website. Since the sole distribution of educational
materials has had mixed results for effectiveness in KT [10] the materials provided will
also be referenced in the sessions where possible so that context of use is provided.
Participants are also encouraged to contribute their own materials. The links page is
another feature that points to knowledge in other places. Finally, a section of the KT
Community is dedicated to case studies. This represents another collaborative KT
strategy where participants contribute their experiences in benefits evaluation methods
as examples for others to learn. It is also a goal of the KT Community to produce case
studies that can be compiled into a new knowledge product.
2.5. Expected Impact
KT activities should expect to elicit change or have a positive impact. For example, the
impact can be a direct change in practice or indirect such as informed decision making
[9]. According to McCartney et al. [19] there is no consensus on measures of success
for electronic communities but some potential areas are effectiveness in meeting
objectives and health of the community in terms of member satisfaction and level of
activity. Barwick et al. [10] list examples of reach, quality, and use indicators and
potential output and process measures. For this KT Community, our impact measures
include statistics on participation in live sessions and discussion forums, contributions
to the repository and case studies. The majority of our impact evidence will be
collected through user feedback surveys completed upon joining the community, after
each live session, and at the conclusion of the project. The KT Community currently
has attracted over 70 health care professionals, researchers and developers,
demonstrating a great interest and need. Although we may not be able to measure it
directly, the larger impact will be in terms of the frequency and quality of benefits
evaluations conducted in the field using the knowledge shared through the KT
Community as well as the new knowledge and ideas generated from interactions
between community members. Evaluation is often not included within the scope of HIS
projects, and is sometimes considered the exclusive domain of HIS researchers. By
bringing together a diverse audience, the community hopes to challenge perceptions
around roles of evaluators and to encourage broader participation in evaluation.
3. Conclusion
Knowledge translation is an important mechanism to ensure that the best available

evidence and research is put into practice. In the context of eHealth benefits evaluation,
individuals involved with HIS adoption need to be aware of the methodologies and
tools already available to ensure that systems are achieving the desired benefits.
Through the creation of the KT Community we have established a combined
intervention of KT strategies that will enable knowledge translation for this context.
Acknowledgement
Funding support for this work was provided by a Canadian Institutes of Health
Research Knowledge Translation Grant.
References
[1] R. Scott, C. Saunders, M. Palacios, D. Nguyen, S. Ali. Technology-Enabled Knowledge Translation and
Our Environment, In: K Ho, S Jarvis-Selinger, HN Lauscher, J Cordeiro, R Scott (Eds). Technology
Enabled Knowledge Translation for eHealth Principles and Practice. New York, Springer, 2012: 325-
337.
[2] S.E. Straus, J. Tetroe, I. Graham. Defining knowledge translation. CMAJ. 181(3-4) (2009), 165-168.
[3] National Physician Survey. 2007 results. 2007. [cited Aug 20, 2012]. Available from:
http://www.nationalphysiciansurvey.ca/nps/2007_Survey/2007nps-e.asp.
[4] National Physician Survey. 2010 results. 2010. [cited Aug 20, 2012]. Available from:
http://www.nationalphysiciansurvey.ca/nps/2010_Survey/2010nps-e.asp.
[5] Ontario Hospital Association (OHA), eHealth Adoption Findings and Comparisons: Levels of eHealth
Adoption across Ontario, Canada and US Hospitals, 2011. [cited Aug 23, 2012]. Available from:
http://www.oha.com/CurrentIssues/Issues/eHealth/Pages/eHealthAdoptionFindingsandComparisons.as
px.
[6] F. Lau, M. Price, J. Boyd, C. Partridge, H. Bell, R. Raworth. Impact of electronic medical record on
physician practice in office settings: a systematic review. BMC Med Inform and Decis Mak 12 (2012),
10.
[7] Canadian Institutes of Health Research (CIHR). PowerPoint Presentation: About Knowledge Translation
at CIHR, 2009. [cited Aug 20, 2012]. Available from: http://www.cihr-irsc.gc.ca/e/39158.html.
[8] I.D. Graham, J. Logan, M.B. Harrison, S.E. Straus, J. Tetroe, W. Caswell, N. Robinson. Lost in
knowledge translation: time for a map? J Contin Educ Health Prof. 26 (2006), 13-24.
[9] R. Reardon, J. Lavis, J. Gibson. From Research to Practice: A Knowledge Transfer Planning Guide.
Toronto: Institute for Work & Health, 2006.
[10] M. Barwick, D. Butterill, D.M. Lockett, L. Buckley, P. Goering. Scientist Knowledge Translation
Training Manual. Toronto, The Hospital for Sick Children / Centre for Addiction and Mental Health,
2005.
[11] Canadian Institutes of Health Research (CIHR). Knowledge to Action: An End-of-Grant Knowledge
Translation Casebook. Ottawa, Her Majesty the Queen in Right of Canada, 2010.
[12] D.M. Berwick. Disseminating innovations in health care. JAMA. 289(15) (2003), 1969-1975.
[13] H.A. Heathfield, D. Pitty. Evaluation as a tool to increase knowledge in healthcare informatics.
MEDINFO 1998. (1998), 879-883.
[14] eHealth Observatory. eHealth Observatory to Monitor the Effects of Health Information System
Deployment in Canada. [cited Aug 20, 2012] Available from: http://www.ehealth.uvic.ca.
[15] F. Lau, M. Price, K. Keshavjee. From benefits evaluation to clinical adoption: making sense of health
information system success in Canada. Healthc Q 2011 14(1) (2011), 39-45.
[16] Canada Health Infoway. Benefits Evaluation Indicators Technical Report Version 2.0. [cited Aug 23,
2012]. Available from: https://www.infoway-inforoute.ca/index.php/resources/reports/benefits-
evaluation/doc_download/544-benefits-evaluation-indicators-technical-report-version-2-0.
[17] Agency for Healthcare Research and Quality (AHRQ). Health IT Evaluation Toolkit, 2011. [cited Aug
20, 2012]. Available from:
http://healthit.ahrq.gov/portal/server.pt/community/health_it_tools_and_resources/919/health_it_evalua
tion_toolkit/27872.
[18] F. Alvino, R. Agrifoglio, C. Metallo, L. Lepore. Learning and Knowledge Sharing in Virtual
Communities of Practice: A Case Study, In: A. D’Atri, M. Ferrara, J.F. George, P. Spagnoletti (Eds.),
Information Technology and Innovation Trends in Organizations. Berlin, Springer, 2011.
[19] K. McCartney, D. Hooker, J. Cordiero, H.N. Lauscher, K. Ho. Practical Considerations in Building an
Electronic Community of Practice for Health Care: Lessons from the Literature, In: K. Ho, S. Jarvis-
Selinger, H.N. Lauscher, J. Cordeiro, R. Scott (Eds). Technology Enabled Knowledge Translation for
eHealth Principles and Practice, New York, Springer, 2012, 325-337.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-263
Complex Interventions in Healthcare and

Health Informatics: A Scoping Review
Julie KIMa,1
a
Abstract. Complex interventions are pervasive in healthcare. There is a need to

make sense of complex interventions to support their better development,
implementation and evaluation. This paper summarizes the results of a scoping
review undertaken to identify attributes of, and logistical considerations for
complex interventions in healthcare. Results suggest five distinct attributes that
can assist researchers to identify, conduct, and appraise complex interventions
studies. Considerations for applying results to evaluate complex health informatics
interventions are discussed.
Keywords. Complex interventions, health informatics, health services research
Introduction
Complex interventions are increasingly becoming a focus in health services research.

They are common in many areas of healthcare such as complementary and alternative
therapies, health promotion and rehabilitation [1]. In fact, many health services
activities may be considered complex [2]. The most commonly cited definition of
complex interventions indicates that they are comprised of several interacting
components [3]. The main objective of this scoping review is to identify key attributes
of complex interventions, and in so doing provides additional definitional elements to
the term. Recognizing complexity in the development, implementation and evaluation
of healthcare interventions is important to ensure that they are addressed with due
diligence and consideration. For instance, the lack of impact found by a study may not
mean the complex interventions are ineffective. It may reflect implementation failure,
which may arise from a lack of fully accounting for complexity [3].
The overall aim of this scoping review was to understand the current state of
knowledge on complex healthcare interventions, including complex health informatics
interventions. These complex interventions attributes identified can be used planning,
conducting, reporting and critiquing complex interventions studies. A second objective
was to identify logistical considerations for operationalizing complex interventions.
The results from this scoping review should help researchers and practitioners identify,
conduct, and appraise complex interventions studies.
1
264 J. Kim / Complex Interventions in Healthcare and Health Informatics: A Scoping Review
1. Methodology
A scoping review addresses broader topics than a systematic review and determines
what literature exists on a topic to summarize what is known and to identify gaps. It
does not synthesize the evidence the same way that a systematic review does. It is a
literature review that examines the extent, range and nature of activities in a domain,
then summarizes and disseminates the findings [4].
1.1. Search Strategy and Sources
The search strategy involved the identification of literature through MEDLINE. This
was supplemented with select articles found in a previous search on complex health
informatics interventions, and a prior collection. Search parameters used were
‘complex interventions’ in the title, English language publications, and publication
between the year 2000 and June 2012.
1.2. Selection
As is common with scoping reviews, some exclusion criteria were devised post hoc
based on increasing familiarity with the literature [4]. Articles were excluded if they
were not available online. Interventions on dental care, alternative or Chinese medicine,
or sports medicine, interventions for research and physical interventions such as stents
were excluded to limit the review to health service interventions in traditional
healthcare settings. Literature was selected for final inclusion on the basis of a full text
review.
1.3. Charting the Data, Collating, Summarizing and Reporting the Results
Charting the data is equivalent to data extraction as carried out in systematic reviews. It
is “a technique for synthesizing and interpreting qualitative data by sifting, charting and
sorting material according to key issues and themes” [4, p. 15]. For this scoping review,
a key theme was the attributes of complex interventions, which emerged from data
charting iteratively in a ground-up approach. Data about complex interventions
logistics was also charted and analyzed. The scoping review involves no assessment of
the quality of literature.
2. Results
2.1. Selected Articles and Characteristics
The MEDLINE search yielded 154 articles; 99 were selected for the review. An
additional 19 were added from the previous search and prior collection. This led to 118
articles included in the study. Of these, forty-six articles were identified for the
synthesis. They included frameworks, commentary on a broad range of complex
interventions topics, editorials and reporting criteria.
J. Kim / Complex Interventions in Healthcare and Health Informatics: A Scoping Review 265
The remaining 72 articles included 16 systematic reviews and meta-analyses, 27

primary studies, seven pilot studies, eight study protocols, and 14 intervention
development studies. These are to be examined in a subsequent review.
2.2. Common Attributes
Synthesis from the review led to the identification of five common attributes that would
identify interventions as complex interventions and the extent of their complexity.
These attributes support a multi-dimensional definition of the term ‘complex
interventions’, and a view that complexity should be seen as existing along a
continuum. These attributes are described below.
Complex interventions are comprised of multiple components [3]. The lack of
specificity of those components is problematic in conducting and reporting studies as
some view complex interventions as an overall ‘black box’. This hinders the ability to
examine effects of each ‘unpacked’ component of complex interventions and determine
effectiveness [5]. Specific types of intervention components have been proposed for
health service interventions [6] and health informatics interventions [7]. One or more
components may be ‘active ingredients’ within the intervention, and it is important to
understand how they exert their effects [3]. It is possible that it can only be speculated
which components are (most) active [8].
Interactions, whether among components, the behaviors of those delivering the
intervention or the behaviors of those receiving the intervention [3] are another
attribute of complex interventions. Where people and organizational levels are involved,
the intervention can be viewed through multiple levels of abstractions and
sociotechnical lenses [9].
The more complex an intervention, the more it is necessary to have a theoretical
foundation [10]. The theory underlying the intervention should provide an
understanding of how the intervention causes change so that weak links in the causal
chain can then be identified and strengthened [3]. It should also ideally articulate
distinct groups affected by the intervention and how they differ in this regard, as well
as the mechanisms by which this occurs [11]. A number of theories, frameworks, and
models were found in the complex interventions literature [3][12].
Complexity can also lie in context [8], which includes the environment in which
complex interventions exist. Context may be the most significant aspect of complexity
[13]. The investigation of how intervention effects are influenced or modified by
context is a recently emerging focus for community intervention trial research [14].
The components of a health informatics intervention can contain many elements of
context. They are: technical components, organizational factors, logistical factors,
behavioral factors and informational components. Some of these may facilitate delivery
of the intervention and others may be barriers to implementation. These components
and factors should be pre-identified and accounted for when conducting studies of
complex health informatics evaluations [7].
Measurement is the final attribute of complex interventions. Complex
interventions themselves must be measureable in order to be studied, and there are
multiple ways in which each component can be measured such as frequency and
intensity. The effects of complex interventions must be measured, whether on people,
groups and organizations [3]. One way to distinguish between simple and complex
interventions is to consider the simplicity or complexity of causal chains and processes
[15]. It may be that the use and interpretation of 'outcomes' as they are traditionally
viewed in medical research such as clinical trials of pharmaceutical products, is not

appropriate for evaluations of complex interventions [1].
2.3. Logistical Considerations
In the development of complex interventions it is important to identify the

evidence base, which can be accomplished through a literature review, an existing
systematic review or having to conduct one [3]. Although a valid framework for
methodologically developing complex interventions may not yet exist to increase
understanding of how and why complex interventions work [16] there are some
existing frameworks that can be used to develop and test complex interventions [3]. A
lack of standardization in classifying complex interventions is a central reason for the
poor reporting of complex interventions evaluations [3]. Authors for one study
developed their own classification system, or taxonomy, to describe interventions used
to prevent falls among the elderly [17].
The importance of implementation, or putting complex interventions into practice,
is simple: interventions that are not implemented will not improve health or healthcare.
The ease with which this may occur can be evaluated prior to the commencement of a
study as an explicit consideration, but this is rarely found in practice [12]. In some
cases, adherence to a strict protocol to ensure standardization of intervention
implementation is appropriate. In other cases, standardizing interventions across study
sites may ‘over-control’ the intervention [10] and standardization can undermine
evaluations of complex interventions so that they may not lead to intended effects [18].
With some interventions, it may be more effective if adaptation to the local setting is
allowed [3].
Evaluation of complex interventions seeks to understand the underlying nature of a
clinical problem and also test theories underlying the intervention [19]. Three aspects
of evaluations of complex interventions are: assessing effectiveness, understanding
change processes, and assessing cost-effectiveness [3]. Pilot and feasibility studies are
recommended as a precursor or initial stage [3], and process evaluations can be
conducted alongside, or embedded within, the main study. Challenges in evaluating
complex interventions include: systematically using relevant research evidence to
develop intervention components, improving the definition and measurement of
complex interventions outcomes, and using appropriate research designs [20].
3. Discussion
In this review we have identified five common attributes that can be used to define and
operationalize complex interventions. Some of these attributes, such as behaviors of
those receiving the intervention and of those delivering the intervention, are mentioned
in the literature without definition or suggestion of how they would be implemented in
practice. Even for those attributes that have been discussed in the literature, such as
intervention components and theoretical foundations, there are different ways to
interpret and define complexity, based on the discretion of the researcher. Examples
include the importance of the context and environment and whether to base the
theoretical foundation at the individual or broader level.
Few specific references to, or distinctions made for, complex health informatics
interventions were found. It could be that they are not different than complex
J. Kim / Complex Interventions in Healthcare and Health Informatics: A Scoping Review 267
healthcare interventions. Alternatively additional consideration for, and study of,

complex health informatics interventions is required. This may reveal additional or
different attributes of complex health informatics interventions and nuances for
consideration when developing, implementing and evaluating them.
The common attributes identified can be further refined to assist in designing and
evaluating complex health informatics interventions. Intervention components may
include hardware and infrastructure as well as applications of information systems.
Theoretical foundations could include, as examples, the Delone and McLean
information systems success model [21], or the diffusion of innovations theory [22].
The context and environment could include technology governance, policies, and other
systems.
There is no single all-encompassing guide or framework for complex interventions.
Additional research and analysis is required to make sense of complex interventions to
guide and support researchers in development, implementation and evaluation.
References
[1] C. Paterson, C. Baarts, L. Launsø, M.J. Verhoef, Evaluating complex health interventions: a critical
analysis of the 'outcomes' concept, BMC Complementary and Alternative Medicine 9 (18) (2009), 1-11.
[2] N.C. Campbell, E. Murray, J. Darbyshire, J. Emery, A. Farmer, F. Griffiths, et al, Designing and
Evaluating Complex Interventions to Improve Health Care, British Medical Journal (2007) 455-459.
[3] P. Craig, P. Dieppe, S. Macintyre, S. Michie, I. Nazareth, M. Petticrew, Developing and evaluating
complex interventions: new guidance, Medical Research Council, 2008.
[4] H. Arskey, L. O'Malley, Scoping studies: towards a methodological framework. International Journal of
Social Research Methodology 8 (1) (2005), 19-32.
[5] M. Petticrew, When are complex interventions 'complex'? When are simple interventions 'simple'?
European Journal of Public Health 21 (4) (2011), 397-399.
[6] N. Wolff, Randomised trials of socially complex interventions: promise or peril? Journal of Health
Services Research & Policy 6 (2) (2001), 123-126.
[7] I. Shcherbatykh, A. Holbrook, L. Thabane, L. Dolovich, Methodologic Issues in Health Informatics
Trials: The Complexities of Complex Interventions. Journal of the American Medical Informatics
Association (2008) 575-580.
[8] P. Hawe, A. Shiell, T. Riley, Theorising Interventions as Events in Systems. American Journal of
Community Psychology 43 (2009), 267-276.
[9] B. Kaplan, P. Flatley Brennan, A.F. Dowling, C.P. Friedman, V. Peel, Towards an Informatics Research
Agenda: Key People and Organizational Issues. Journal of the American Medical Informatics
Association (2001), 235-241.
[10] P. Hawe, A. Shiell, T. Riley. In response to Spillane V., Byrne M.C., Byrne M., Leathem C.S.,
O’Malley [11] M. & Cupples M.E. Monitoring treatment fidelity in a randomized trial of a complex
intervention. Journal of Advanced Nursing 60 (3) (2007), 343–352.
[11] M. Mackenzie, C. O’Donnell, E. Halliday, S. Sridharan, S. Platt, Evaluating complex interventions: one
size does not fit all. British Medical Journal 340 (c185) (2010), 401-403.
[12] E. Murray, S. Treweek, C. Pope, A. MacFarlane, L. Ballini, C. Dowrick, et al, Normalisation process
theory: a framework for developing, evaluating and implementing complex interventions, BMC
Medicine (2010), 1-11.
[13] A. Shiell, P. Hawe, L. Gold, Complex interventions or complex systems? Implications for health
economic evaluation. British Medical Journal 336 (2008), 1281-1283.
[14] P. Hawe, A. Shiell, T. Riley, L. Gold, Methods for exploring implementation variation and local
context within a cluster randomised community intervention trial, J Epidemiol Community Health 58
(2004), 788-793.
[15] D. Rickles, Causality in complex interventions, Med Health Care and Philos 12 (2009), 77–90.
[16] S. Treweek, Complex Interventions and the Chamber of Secrets: Understanding Why They Work and
Why They Do Not. Journal of the Royal Society of Medicine 98 (2005), 553.
[17] S.E. Lamb, C. Becker, L.D. Gillespie, J.L. Smith, S. Finnegan, R. Potter, et al, Reporting of complex
interventions in clinical trials: development of a taxonomy to classify and describe fall-prevention
interventions, Trials 12 (125) (2011), 1-8.
[18] K. Seers, Evaluating Complex Interventions. Worldviews on Evidence-Based Nursing, Second Quarter,
2007.
[19] R. Emsley, G. Dunn, I.R. White, Mediation and moderation of treatment effects in randomised
controlled trials of complex interventions. Statistical Methods in Medical Research 19 (2010), 237-270.
[20] B. Blackwood, Methodological issues in evaluating complex healthcare interventions. 2006.
[21] W.H. Delone, The DeLone and McLean model of information systems success: A ten-year update.
Journal of Management Information Systems 19 (4) (2003), 9-30.
[22] E. Rogers, Diffusion of Innovations. New York: Simon and Schuster, 2008.
Consumer Informatics
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-271
Forumclínic: the shaping of virtual communities

to assist patients with chronic diseases
1
Immaculada GRAUa, , Francisco J. GRAJALES IIIb, Joan GENE-BADIAc,d,e, Antoni
SISOf and Marc DE SEMIRa
a
Hospital Clínic de Barcelona, Spain
b
eHealth Strategy Office, Faculty of Medicine, UBC, Vancouver, Canada
c
Universitat de Barcelona, Spain
d
Consorci d’Atenció Primària de Salut de l’Eixample, Spain
e
Institut Català de la Salut, Spain
f
Gesclinic, Spain
Abstract. Information and communication technologies (ICTs) provide new

opportunities to complement traditional care while enhancing patient autonomy.
With the objective to supplement patient care, a group of health professionals at
the Hospital Clínic de Barcelona created Forumclínic, an online networking
website in Spanish and Catalan. In 2008, seven web- and DVD-based chronic
disease portals (Diabetes, Schizophrenia, Cardiac Ischemia, COPD, Depression,
Breast Cancer and cardiovascular risk) were created with the following resources:
multimedia patient education material; physician-specialist transcribed research
(articles) news; an open question forum (for clinician-user and user-to-user
interaction); and patient and specialist interview videos on the progress of disease,
common diagnosis and treatment procedures; and information on the best or worst
prognoses. Using data from Google Analytics, server logs were used to observe
online behaviour patterns and user postings. This data combined with a mixed
methods approach were used to evaluate the development of a virtual community
(VC). A virtual community was developed when the number of forum visits was
greater than those in the disease portal (definition). While nearly half of the
visitors were from the Americas, the Schizophrenia, Breast Cancer, Depression
and COPD forums met the criteria for and developed a virtual community.
However, the Diabetes and Cardiac Ischemia forums did not reach VC status. It is
also interesting to note that users in their late thirties and early forties were
primarily women. The development of four virtual communities in Forumclínic
seems to support the self-care needs of virtual patients. Users also reported
appreciating the increased interaction with experts online and commonly
collaborated with the forum moderator to guide and support other users with
similar conditions in managing their health. Thence, we believe that Forumclínic
is a good model to complement traditional patient care. A formal evaluation of
this adjuvant form of care, from both the users’ and moderators’ perspective, is
currently in its final stages.
Keywords. e-health, Virtual Communities, Patient Communication, patient

education
1
Corresponding author. Hospital Clinic de Barcelona, Villarroel 170, 08031Barcelona, Spain.
[email protected], +34 93 2275690
272 I. Grau et al. / Forumclínic: The Shaping of VC to Assist Patients with Chronic Diseases
Introduction
Information and communication technologies offer a promising opportunity for

complementing the information provided by health care professionals in a clinical
interaction and enhance patient autonomy [1]. It has been suggested that people with a
high level of autonomy make a more intensive use of the Internet; with this use, patient
autonomy increases [2]. For this reason, the Internet is a powerful tool for
complementing the health care process in patients with chronic diseases. At present,
there is paucity in the literature about how this approach may be of benefit (or risk) to
health [3].
Facebook, YouTube and Wikipedia are all examples of what Tim O'Reilly
originally described as Web 2.0, virtual environments that promote collaboration with a
rapid exchange of information among users [4]. This same approach, applied by health
information users, allows some groups to become working virtual communities or
“networks of interpersonal links that provide sociability, support, information, a sense
of belonging, and social identity”, as described by Barry Welman [5].
A group of health care professionals at Hospital Clínic de Barcelona launched
Forumclínic, a Web 2.0-based health programme for patients with chronic conditions in
the Spanish and Catalan languages. Our aim was to test a model that would support the
clinical management of chronic disease, by providing patient education materials (e.g.,
videos and lay-transcriptions of emerging medical research), and peer support-
moderated forums. Professionals from: 1) Hospital Clinic de Barcelona, a university
hospital and leading research institution; and 2) associated health centres moderate
each of the forums, according to their specialty. The Banco Bilboa Vizcaya Argentaria
(BBVA) Foundation has recently renewed the funding for this programme. Fundación
BBVA is the philanthropic non-profit arm of the BBVA bank.
1. Objectives
This report evaluates the absence or presence of a virtual community in the

Forumclínic patient portal through a mixed methods approach.
2. Methods
The following data were used to analyze the presence or absence of a virtual
community:
Google Analytics’ downloadable data.
Sociodemographic indicators reported during user registration. (This allowed
crossreferencing to server logs, in order to evaluate general behaviour; such
as the number of times a user logged into the system, posted, etc.)
Forum postings.
A virtual community was defined to exist when the number of user visits to a
chronic disease forum was greater than those for the portal’s main page. Each portal
has information classified according to the following categories: 1) general summary of
I. Grau et al. / Forumclínic: The Shaping of VC to Assist Patients with Chronic Diseases 273
the disease; 2) weekly news, where recent research articles are transcribed into lay
language by our board-certified health professionals; 3) specialist-moderated forums;
and 4) patient, health provider, and 3D biological animation videos which explain in
simple terms the pathophysiology, treatment, and rehabilitation of disease. This
structure facilitates a dynamic relationship between patients and health professionals,
and aims to foster autonomy. The proceeding results correspond to data from the 2008
calendar year. More recent data was not used due to a server upgrade problem. The
necessary ethics approval was acquired and data was de-identified in accordance with
[6].
3. Results
Table 1 displays overall web site activity. As can be seen, the data indicate that nearly
half of visitors accessed information from the Americas (primarily South America).
Schizophrenia and Breast Cancer were the most-accessed portals. It is also intriguing to
note that cardiovascular risk was not as popular as the other forums, despite its
prevalence in Hispanic communities.
Table 2 displays the activity of forum users in the five portals that were
evaluated for the status of virtual community along with their corresponding user
profiles. In total, a third of Forumclínic’s visitors accessed the forums. Proportionately,
this was as high as 62% for the Breast Cancer portal. The overall percentage of forum
visits when compared portal visits was 23.25%. Chronic Obstructive Pulmonary
Disease (COPD), with a percentage of 20.75%, is considered a VC but does not reach
this average.
It can be seen that users are in their late thirties or early forties and, are primarily
female, except for cardiovascular disease. Family members, rather than patients
suffering the condition, are increasingly present in the Schizophrenia and COPD
communities (self-reported in the forum). Italics and the colour red were used in the
table to show forums which did not meet the definition for a virtual community.
4. Discussion
The data support the Forumclínic portal model as an effective and social approach for
chronic disease information consumer engagement.
Table 1. Activity at the Forumclínic web in 2008
Total Users Visits Pages Americas Pages Seen
/visit (% Visits) forum
Schizophrenia 44,488 63,622 5.11 40.4 122,106 37.6
Diabetes 51,345 59,490 4.21 42.1 21,192 8.5
COPD 40,039 49,299 4.32 41.0 44,172 20.8
Cardiac Ischemia 47,005 54,773 3.62 48.2 20,554 10.4
Breast Cancer 41,874 59,715 4.86 36.1 180,028 62.0
Depression 35,195 45,717 3.95 37.7 58,329 32.3
Cardiovascular 14,291 18,910 2.41 46.7 4,208 9.2
risk factors
Home page 99,229 103,563 5.00 56.0 11,479
Forumclínic 373,466 455,089 4.37 44.6 462,068 23.3
274 I. Grau et al. / Forumclínic: The Shaping of VC to Assist Patients with Chronic Diseases
Table 2. Activity at the Forumclínic moderated forums in 2008

Registered users %
% visit posts
Regist- % average % Post
from
to forum age
Users ered men patient Topics per
the
from all (year) topic
users mod-
visits
erator
Schizophrenia 44,488 37.6 214 45.4 35.8 39.2 188 8.0 17.6
Breast Cancer 41,874 62.0 274 7.3 38.2 77 240 5.0 31.9
Cardiac Ischemia 47,005 10.3 39 73.3 41.3 64.1 30 2.0 60.5
Depression 35,195 32.4 121 33.1 37.1 68.8 77 13.5 26.6
Diabetes 51,345 8.5 45 33.3 37 71.1 24 6.3 37.5
The large number of visits in the cancer and schizophrenia portals, despite the lack
of a virtual community also advocated the need for high quality information in the area.
Of interest is the demographic distribution of forum users, which is not congruent in
web-traffic with the prevalence of disease. Cardiac ischemia patients, for example,
tend to be older and uncomfortable using the Internet. However, forum postings reveal
that the digital divide is not a major obstacle for elderly people (women, for the most
part) who are using younger family members as a proxy for information access. The
impact and determinants on access by these users however requires further
investigation as in Latin societies, women (mothers and daughters) generally act as the
health agents for the family. They take care of the ill, manage the relationship with the
health system, and seek information (on the Web) for older family members.
When compared across other forums and its own portal, the diabetes forum had an
amount minimal views (<10%). We speculate that this may be due to: a) the large
number of diabetes resources available in the Spanish language; (a simple Google
search limited to Spanish reveals over 2.5 million sites; b) the potentially younger age
of participants (preliminary analyses of the user registration database reveal that this
characteristic); and c) the increasing use of social media for patient support (supported
by the former and likely plausible given that younger diabetics, Type I diabetics, tend
to be more involved in social media tools).
Through unsolicited emails and forum postings, Forumclínic’s users have
expressed the importance of moderator engagement for their interest and participation.
The moderator filters malicious content, cleans the discussion board of unwanted
publicity or inappropriate material and also redirects new users to previously answered
questions. This service, along with the portal’s structure, as been successful at engaging
consumers and appears to be responsible for the high level user satisfaction.
There are a number of limitations in this study. First, the sociodemographic data
used in the characterization of users is self-reported. Second, the Internet Protocol
tracking database used does not allow cross referencing with usernames, which means
that one person could have posted in the forums with multiple computers or web
aliases, without the corresponding adjustment. Lastly, the qualitative method used for
observing forum participants is subjective, depending on the viewpoint from which it is
evaluated.
Future research will be required to determine the quantitative impact (e.g., Quality
Adjusted Life-Years gained) and other contextual incentives for participation in the
Forumclínic portal. Using a similar dataset and a new server log, we are currently
implementing a multivariate analysis model that will contribute to our understanding of
moderator-user interaction dynamics. We also aim to study other factors that lead to the
I. Grau et al. / Forumclínic: The Shaping of VC to Assist Patients with Chronic Diseases 275
development of a virtual community.

The results of this report are homogenous with other studies of virtual
communities in the health domain [7]. In particular, a recent study by Frost and
Massagli reported that sharing health data with a community fosters patient
engagement and self-management of disease. At the time this report was written, we
believe that this element is also an important factor for success. To evaluate this more
objectively, we are currently in the final stages of two phenomenological evaluations
that characterize the emotional aspects of portal participation (one for clinicians and the
other for target users).
5. Conclusions
The development of four virtual communities confirms that patients need participative
role health-related web sites. Patients appreciate an increased degree of interaction
with experts and fellow patients which allows for more collaborative self-
management.
The portals that were successfully characterized as virtual communities in this
study continue to grow and be populated by a high proportion of female users. The
Depression and Schizophrenia portals were particularly useful in providing access for
communities that are commonly stigmatized.
Forumclínic is a living project that is growing both in activity (e.g. visits) and
participation (e.g. number of registered users engaging in forums). We have been
successful in creating functional virtual communities by providing health information
access through a model of patient education resources and user interaction. Future
research will increase our understanding on the contextual determinants of success for
user engagement in the Spanish and Catalan-speaking chronic disease community.
References
[1] G. Demiris, L. Afrin, S. Speedie, K. Courtney, M. Sondhi, V. Vimarlund, C. Lovis, W. Goossen, and
C. Lynch. Patient-centered applications: use of information technology to promote disease
management and wellness. A white paper by the AMIA knowledge in motion working group.
Journal of the American Medical Informatics Association 15.1 (2008), 8-13.
[2] M. Castells, La galaxia internet. Reflexiones sobre internet, empresa y sociedad, Editorial in Plaza
y Janes, Barcelona, 2002.
[3] G. Eysenbach, J. Powell, M. Englesakis, C. Rizo, A. Stern, Health related virtual communities and
electronic support groups: systematic review of the effects online peer to peer interactions, British
Medical Journal 328 (2004), 1166.
[4] O’Reilly T. What is Web 2.0? Design patterns and business models for the next generation of
software. O’Reilly Media, 2005.
[5] B. Wellma, J. Boase, W. Chen, The networked nature of community: online and offline, It & Society
1 (2002), 151-165.
[6] I. Hrynaszkiewicz, Me. Norton, A. Vickers, D. Altman. Preparing raw clinical data for
publication: guidance for journal editors, authors, and peer reviewers. British Medical Journal 340
(2010), 181.
[7] J. Frost and P. Massagli. Social uses of personal health information within PatientsLikeMe, an
online patient community: what can happen when patients have access to one another's data.
Journal of Medical Internet Research 10.3 (2008), e15.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-276
The PLU Problem: Are We Designing

Personal ehealth for People Like Us ?
Chris SHOWELL a,1 and Paul TURNER a
a
eHealth Services Research Group, University of Tasmania
Abstract: The near-pervasive introduction of ehealth systems, and the more recent
implementation of systems intended for patient use offer patients the opportunity
to participate in their own care. Unfortunately the design of these systems means
that they may work better for “People Like Us’ rather than for those on the wrong
side of the ‘digital divide’. This paper looks at the professional, practical and
ethical implications of this conundrum.
Keywords: electronic health records, disadvantage, information systems
Introduction
This paper argues that the current approach to the design of personal ehealth systems
may serve to accentuate the gap between privileged and disadvantaged end users and
healthcare recipients, rather than improve equity of access to health care services.
The problems facing healthcare services in the developed world are well
documented and understood. They include financial challenges from increasing
treatment costs [1], resource scarcity with an aging workforce, and an increasing
burden of chronic disease [2]. Where healthcare is publicly funded, Governments also
face the politically sensitive challenge of determining appropriate healthcare service
levels in an environment of increasing citizen demands and expectations for care.
Health reform remains a critical area for policy debate and has led to a range of
solutions to address the problems faced. In Australia, ehealth systems are seen as one
set of solutions to these problems. Unfortunately many of these ehealth systems have
delivered mixed results or have not generated the predicted savings as a result of
designs that under-estimate the complexity of healthcare practice.
One response to these design challenges has been to focus on personal ehealth
systems tailored and customised to the needs of individual health users. One example is
Australia’s personally controlled electronic health record (PCEHR) which was
launched in July 2012. However, differences in individual knowledge, skills and
inclination to use such systems may be resulting in system designs primarily suitable
for a privileged group of literate and motivated end users, rather than for those most
urgently in need of improved health service delivery.
1
[email protected]
C. Showell and P. Turner / The PLU Problem: Are We Designing Personal eHealth for PLUs? 277
1. Methods
To explore the argument that the design of contemporary personal ehealth systems
maybe problematic, a literature review was conducted to better understand research on
the relationship between literacy and personal ehealth. Using a combination of search
terms ehealth/personal ehealth and literacy/health literacy 100 papers available through
PUBMED were identified and reviewed. 17 of these papers were also identified as
providing insights on the characteristics of healthcare recipients.
2. Analysis
From an analysis of the literature, a conventional approach to segmentation of the

population of healthcare recipients is to use socioeconomic status (SES) along a linear
scale. However, there was also evidence of other factors and as a result this population
was viewed as a collection of groups or clusters that could not easily be plotted along
the SES scale. Two clusters providing simplistic archetypes of healthcare recipients
were identified to focus the analysis of ehealth design issues in the literature. These
two clusters were differentiated according to their willingness and ability to use a
personal ehealth record. One cluster includes people who understand healthcare and
health issues, take care of their own health, are literate, well to do, tech-savvy, and hold
a tertiary qualification. These are the People Like Us (PLUs). The other cluster
includes people disinclined to take exercise for its own (or their own) sake, or to eat
sensibly. They are not textually, technically or health literate. They struggle financially,
and may not have finished secondary education. We characterized them as
disempowered, disengaged and disconnected (DDDs).
The literature review highlighted a general enthusiasm for personal ehealth
systems and their capacity to improve the ‘quality’ of healthcare by reducing cost,
improving safety, facilitating access to health services, and ensuring timely care.
Australia’s National E-Health Transition Authority (NEHTA), for example, suggests
that a Personally Controlled Electronic Health Record will improve: “…the self-
management of stable chronic diseases…communication between clinicians and
individuals…[and] decision making by…individuals...” [3]. Pagliari and colleagues,
writing in the British Medical Journal, suggest that personal electronic health records
“…have the potential to empower patients through greater access to personal data,
health information, and communications tools” [4]. Broadly, the justification for the
development and implementation of personal ehealth solutions relies on the notion that
they will enhance one or more measures of the ‘quality’ of healthcare services.
Significantly however Clarke and Leigh [6] highlight how differences in life
expectancy between demographic groups in Australia provide stark evidence of
continuing health inequality. They identified significantly increased mortality
associated with income (odds ratio 1.88), education (1.25) and a low socioeconomic
index (1.32). According to the Australian Institute of Health and Welfare [7] those
living in the least advantaged areas of Australia are more likely to smoke, be physically
inactive or obese, have diabetes, behavioural problems, asthma, heart disease or
arthritis, and have higher mortality across most chronic conditions. Adult literacy also
presents a problem for many in Australia. The Australian Bureau of Statistics reported
in 2006 that 16.7% of adults were at prose literacy Level 1 (trouble completing a basic
form; may find some information on a medicine label), and 29.7% at Level 2 (may not
278 C. Showell and P. Turner / The PLU Problem: Are We Designing Personal eHealth for PLUs?
be able to summarise text) [8]. Health literacy is also a problem. The US Agency for
Healthcare Research and Quality found that poor health literacy was “...associated with
increased hospitalizations, greater emergency care use…and, among seniors, poorer
overall health status and higher mortality.” [9] The disadvantaged are less likely to
have home internet access, and there is evidence that living in a rural area and having a
medical condition will make home internet use even less likely. [10] The inter-
relationship between all of these factors is complex, and it can be difficult to separate
cause from effect. However, it is apparent that there is an association between low
income, poor literacy (textual, technical and health), chronic disease, and poor health
outcomes.
For better or worse, healthcare systems are developed predominantly by white,
educated middle class professionals (PLUs) who design systems and processes which
they see as being appropriate and user-friendly. This tendency to design for ourselves
(the PLU problem) results in patient instructions using complex language; patients
receiving complex verbal descriptions of health issues using clinical terminology; and
the explicit assumption that online tools are an important and appropriate option for
healthcare service delivery.
Catwell and Sheikh [11] considered the evaluation of ehealth systems, and argued
that the evaluation should be continuous and systemic. They cautioned that “[l]arge
investments in eHealth may, by diverting resources result in a shortfall in funding for
basic infrastructure, equipment, and staffing elsewhere in the system. …investing in
developments such as telemedicine, which are only likely to be accessible to a minority,
would exacerbate the digital divide and existing health inequities.” [11]
Figure 1: Risk of unintended consequences – the disadvantaged may end up worse off
Any diversion of health budgets towards generic personal ehealth systems is likely
to deliver benefits to PLUs, while leaving the ‘quality’ of healthcare for DDDs
unchanged. In fact, there is a risk that diversion of health budgets to ehealth will be
used to justify and facilitate a reduction in funding from other, more conventional
healthcare services. The potential for these unintended adverse consequences from
ehealth investment for DDDs is illustrated in Figure 1.
3. Results
Many of the processes, systems and technologies intended to provide supportive

care and self-care are tailored for a potential user who is very like the designer. Patient
focused ehealth initiatives may well be suited to a demographic cohort that is well off,
C. Showell and P. Turner / The PLU Problem: Are We Designing Personal eHealth for PLUs? 279
tech savvy and street smart, but may be less helpful in areas of healthcare where the
individual’s needs are great, and the cost of meeting those needs is high.
PLU systems will place additional resources at the disposal of those who are
adaptable and capable — those who probably do well with existing systems and
services. Making reasonably healthy people a little bit healthier is not necessarily the
most effective use of scarce health resources. Better cost benefits in terms of health
outcomes are likely to be achieved by focusing on the DDDs whose health is worst,
rather than designing and implementing for the PLUs.
A report from the European Union [12] comments on the introduction of Ambient
Intelligence (AmI) in healthcare. The report notes:
“AmI claims to be particularly people-oriented, implying that it will also be
inclusive - providing, of course, it lives up to its promises of being user-friendly,
unobtrusive and controllable. But…the fundamental question remains…[will] AmI …
include the majority of people or…benefit mostly young, urban and mobile techno-
freaks. In the latter case, AmI could become an additional source of exclusion in
society.” The report cautions that “[t]he relation between social exclusion and health
status is well known. Digital divides…can negatively affect health.” [12]
The greatest challenge (arguably) is providing healthcare services to those who
lack the motivation and capacity to make positive changes in their lives. They show
little interest in adopting a healthy diet; regular exercise doesn’t interest them; if they
smoke, they’ll probably continue to do so. Their poor literacy will make it hard for
them to adapt to a routine of regular medication, and they will struggle to monitor their
symptoms effectively. They may benefit from some form of mentoring, possibly with a
technology component, but it will need to be carefully targeted at their capability, skills
and worldview. The solution to the challenge of poor uptake (by DDDs) is often
framed as requiring better solutions, including eHealth, Internet access, or training. But
many of the processes and systems designed for supportive care and self-care are
tailored for the designers and PLUs. This paper argues that there is a greater need in the
design of personal ehealth systems to recognize the constraints imposed by a class of
potential users who face very high barriers to technology adoption and effective use.
4. Discussion
Solutions to healthcare problems should be targeted at those with the greatest need.
However, a PLU-designed health service, or a PLU designed ehealth system is unlikely
to deliver equitable benefit to all groups of patients and has the potential to further
disadvantage some groups. The needs of the DDDs should be considered as a special
case, and explored in some detail. Once those needs are clearly understood, they should
either be incorporated into the overall design, or provided for through a focused
alternative, that may not initially require citizen/patient use of an eHealth system.
It may be more appropriate to design and implement solutions tailored for the
capabilities of DDDs, and to then extend implementation progressively to other groups
until marginal benefits become too small to fund. Designers should focus on user
centred design, as many do, but in doing so should take care to identify and specify the
class of user to which the initiative is targeted. The sense of disempowerment among
DDDs will make them reluctant to participate in design activities. They may struggle to
express themselves, or to see their opinions as having merit. Because of the many
categories of limited capability which interfere with the use of technology by DDDs,
280 C. Showell and P. Turner / The PLU Problem: Are We Designing Personal eHealth for PLUs?
this approach is likely to be more difficult to implement. These solutions may also be
less interesting for designers, developers and academic researchers due to the lack of
feature and functional complexity. Indeed the design activities may produce systems
that their creators and implementers might not want to use. It is argued however, that in
terms of overall cost benefit, approaches like these could be potentially very rewarding
as illustrated in Figure 2.
Figure 2: Preferable to adopt a specialised focus on those with greatest need

A modified design process could include: a positive effort to engage with “lowest
common denominator” users; using observation as a way to understand how DDDs
interact with systems and with healthcare; matching prompts and documentation to the
reading level of users; and using pictograms (‘IKEA instructions’) to enhance
understanding. Implementers should also be clear about whether there is a workable
“non-e-” alternative to the personal ehealth option for those who cannot or will not use
the technological solution. Can it be provided at the same or lower cost? And what is
the overall cost to the health system to provide that service to those who need it most?
5. References
[1] Holman H (2005) Chronic disease and the healthcare crisis. Chronic Illn. 1, 265–274.
[2] ABC News, Healthcare crisis highlighted, Last updated August 17, 2009, Accessed on August 17,
2009.
[3] Benefits of a PCEHR - nehta.gov.au, Last updated November 20, 2011, Accessed on November 20,
2011.
[4] Pagliari C, Detmer D, Singleton P (2007) Potential of electronic personal health records. BMJ 335,
330–333.
[5] Bourdieu P (1987) Distinction: A Social Critique of the Judgement of Taste, Harvard University Press.
[6] Clarke P, Leigh A (2011) Death, Dollars and Degrees: Socio - economic Status and Longevity in
Australia. Economic Papers: A journal of applied economics and policy 30, 348–355.
[7] Chronic Diseases and Associated Risk Factors in Australia, 2006, Australian Institute of Health and
Welfare.
[8] ABS (2008) Australian Social Trends 2008, Australian Bureau of Statistics, Canberra.
[9] Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Viera A, Crotty K, Holland A, Brasure M, Lohr
KN, Harden E, Tant E, Wallace I, Viswanathan M Health Literacy Interventions and Outcomes: An
Updated Systematic Review, AHRQ, Rockville, MD.
[10] Wang J-Y, Bennett K, Probst J (2011) Subdividing the Digital Divide: Differences in Internet Access
and Use among Rural Residents with Medical Limitations. Journal of Medical Internet Research 13,.
[11] Catwell L, Sheikh A (2009) Evaluating eHealth Interventions: The Need for Continuous Systemic
Evaluation. PLoS Med 6, e1000126.
[12] Cabrera M, Burgelman J-C, Boden M, da Costa O, Rodriguez C (2004) eHealth in 2010: Realising a
Knowledge-based Approach to Healthcare in the EU.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-281
Islamic E-Health: Definitions,

Applications, and Challenges
Mowafa HOUSEH a,1
a
Abstract. The purpose of this exploratory study is to introduce the concept of

Islamic E-health. The study defines Islamic E-health and discusses various
applications of this concept. Data collection methods used for the study included
interviews, Facebook, Google, and iTunes searches using a variety of Islamic E-
health-related terms. The results show that some Islamic E-health applications
focus primarily on spiritual health, followed by Hajj systems for surveillance and
monitoring and the use of electronic medical records to monitor the blood glucose
levels of Muslim patients who fast during the month of Ramadan. Future research
and research limitations are also discussed.
Keywords. Islam, e-health, spirituality, healing, Ramadan, Hajj
Introduction
Information and communication technologies play a pervasive role in our lives. They
impact how we communicate and interact with our families, loved ones, colleagues,
friends, government, healthcare, and private institutions. In the healthcare domain, the
use of information and communication technologies (ICTs) to improve clinical care
and administrative decision-making has been referred to as e-health [1]. The primary
goal of e-health is the use of the Internet and related applications to improve access to,
efficiency of, and quality of care within healthcare with the goal of improving the
health status of patients [2]. The discipline of e-health has existed for many years and
includes an increasing number of multifaceted interventions that are aimed at
improving health status through the use of information technology. One relatively
unknown area of research within this domain is referred to as spiritual e-health.
Spiritual e-health is the use of the internet and related technologies to improve the
spiritual well-being of a patient [3]. Several studies have focused on the concept of
spiritual e-health, but none have researched the role of e-health applications within the
context of the Islamic spiritual belief system [3].
Spirituality plays a large part in Muslim daily life. Muslims are required to pray
five times a day, fast during the holy month Ramadan, give to charity, and make a
pilgrimage to the holy city of Mecca. Each of these religious obligations is a
1
[email protected]
282 M. Househ / Islamic E-Health: Definitions, Applications, and Challenges
manifestation of the spiritual Muslim belief system. As part of the healing process,
Muslims believe that prayer, supplication, recitation from the holy book, the Quran,
and other religious rituals can help in improving health. Muslims are also taught
through prophetic teachings that fasting during Ramadan can help to improve health.
There are many public health issues during the month of Hajj, when millions of
Muslims from around the world gather in Mecca to fulfill their religious obligations.
The country of Saudi Arabia, the birth place of the Islamic faith, has significant
concerns about Muslim daily life. For example, all public hospitals in Saudi Arabia
have Islamic religious affairs departments that help to provide religious guidance,
education, and spiritual healing to patients. In addition to establishing various places of
worship, these religious departments use technologies such as web pages, SMS
messaging and web-based information to assist in the patient care process. Furthermore,
users on social networking sites such as Facebook have developed spiritual healing
groups for Muslims.
The holy month of Ramadan and the Hajj pilgrimage are two important annual
Islamic events that affect the spiritual and physical health of Muslims. During the holy
month of Ramadan, it is obligatory for Muslims to fast. Patients who decide to fast are
likely to be monitored by their primary care physicians using technologies such as
electronic medical records to monitor blood sugar levels. During the Hajj season, when
more than two million worshipers converge on Mecca from more than 180 countries,
public health issues related to communicable diseases have become a significant
concern, and various public health information systems are employed to monitor
disease outbreaks.
Given the uses of information and communication technology to support Islamic
spiritual practices, the purpose of this exploratory paper is to define Islamic e-health
and to describe its applications and future development.
1. Definition of Islamic E-Health
This paper proposes the following working definition of Islamic e-health:

“The application and use of information and communication
technologies to monitor and support Islamic spiritual health
practices with the goal of improving Muslims’ spiritual, mental, and
physical health status.”
The above definition takes into consideration the use of information and
communication technologies, such as the internet, electronic medical record systems,
public health information systems, and mobile health technologies that can be used to
help monitor and support the spiritual, mental and physical health status of practicing
Muslims.
2. Methodology
Data collection, including interviews and internet searches, began in November 2011
and ended in January 2012. The interviews included one interview with hospital
religious affairs personnel in Saudi Arabia that lasted 120 minutes, a brief 10-minute
interview with one family care physician, and two separate 10-minute interviews with
one public health expert and one health informatics expert in Saudi Arabia.
M. Househ / Islamic E-Health: Definitions, Applications, and Challenges 283
Internet and social media sites, such as Facebook and the Apple iTunes Store, were
searched using terms such as Islamic healing, ruqya (spiritual healing), fatwa (Islamic
Law), Hajj, Ramadan, informatics, electronic medical record, public and health
informatics. Various results were returned through Facebook groups, through the
Google Search engine and through the Apple iTunes store.
3. Results: The Application of Islamic E-health
3.1. Spiritual Healing
Most of the Islamic e-health interventions that were identified were related to education
and information on spiritual healing. In Islam, spiritual healing is referred to as Ruqya
and is a form of supplication that is supposed to protect and heal. A Muslim can
perform Ruqya on himself or on others [4]. Ruqya was used by the Muslim Prophet
Muhammad, Peace Be Upon Him (This is a supplication that a Muslim performs when
mentioning the name of the Muslim Prophet), and Ibn Alqayyim, a 13th-century
Muslim scholar, said, “Ruqyah is one of the greatest remedies that the believer should
use regularly.” [4] A prophetic tradition collected by Muslim, a famous collector of
authentic Muslim prophetic sayings, narrates the following story [4]:
Uthman complained to the Messenger of Allaah (Peace be Upon
Him) about pain that he had felt in his body from the time he had
become Muslim. The Messenger of Allaah said to him: “Put your
hand on the part of your body where you feel pain and say
‘Bismillaah (in the name of Allaah) three times, then say seven times,
‘I seek refuge in the glory and power of Allaah from the evil of what I
feel and worry about’.”
Google web searches identified various sources of information in English on how
to perform Ruqya, such as Islam Q&A, which provides detailed descriptions of how to
correctly perform Ruqya, its benefits, and some common misconceptions. Several
Facebook groups on Islamic spiritual healing were found, such as the Islamic Spiritual
Healing group. Various iPad applications, such as Ayat Ruqya, list religious
supplications and how to perform them. Furthermore, within Saudi Arabia, religious
affairs departments perform Ruqya on patients who request these services. However,
patients are informed that the service is complementary to the medical treatment they
receive.
3.2. Hajj
Hajj is a religious obligation for Muslims that was prescribed by God at the time of the
Prophet. Muslims believe that its roots date to the time of the Prophet Abraham. Every
Muslim who is financially and physically able to perform Hajj is required to do so once
in his or her lifetime. Every year, approximately 3 million Muslims converge in Mecca,
Saudi Arabia, to fulfill this important religious obligation. In the Quran, God says,
“And Hajj (pilgrimage to Makkah) to the House (Ka‘bah) is a duty that mankind owes
to Allaah, those who can afford the expenses (for one’s conveyance, provision and
residence) [5].”
Much of the information related to Hajj was found through Google and through
interviews conducted with various public health and informatics experts within the
284 M. Househ / Islamic E-Health: Definitions, Applications, and Challenges
field. The results show that health information systems are used intensively during the
Hajj period to control the flow of pilgrims, to track movement, and for health
surveillance. Yamin discusses a framework for the use of information systems to
improve Hajj management [6]. The author suggests the use of information technology
for 1) collecting information on pilgrims and providing them with RFID tags; 2) processing
pilgrims and collecting additional matching information; 3) installing wireless remote
sensors; and 4) processors and display screens. The author argues that these methods will
help to reduce wait times at airports, track the movements of individuals, contain the spread
of disease, improve future planning and assist in tracking illegal immigrants.
3.3. Ramadan
Ramadan is a holy Muslim month in which it is obligatory for every healthy Muslim to
avoid eating from sunrise to sunset. God revealed the following in the Quran [7]:
“The month of Ramadan [is that] in which was revealed the Qur'an,
a guidance for the people and clear proofs of guidance and criterion.
So whoever sights [the new moon of] the month, let him fast it; and
whoever is ill or on a journey - then an equal number of other days.
Allah intends for you ease and does not intend for you hardship and
[wants] for you to complete the period and to glorify Allah for that
[to] which He has guided you; and perhaps you will be grateful.”
Participants in this study noted that Muslims living with diabetes are significantly
affected by fasting during the month of Ramadan. The participants noted that diabetic
Muslim patients are encouraged to fast and follow a prescribed regimen in which the
patient takes a full dose of medication prior to breaking the fast and one half of a dose
prior to beginning the fast before sunrise. During this month, diabetic patients are
monitored, and their blood test results are transferred to an electronic medical record
that helps their family physicians track fasting patients’ blood glucose levels. The
interviewed participants did not mention any other benefits to patients from the use of
information technology during Ramadan.
4. Discussion
This exploratory study defined the concept of Islamic e-health and surveyed some of
the existing applications that support this concept. This study revealed that the primary
focus of Islamic e-health has been spiritual health, followed by Hajj systems and
Ramadan. Information technology may provide significant benefits for Muslims’
connections for spiritual guidance, improvements to Hajj systems and the use of
technologies such as the EMR to monitor Muslim patients who fast during Ramadan.
This research is exploratory and examines some of the relevant issues related to Islamic
e-health.
5. Limitations and Future Research
The study has several limitations worth mentioning. The numbers of interviewees were
small and could have been expanded to include other stakeholders, such as patients and
M. Househ / Islamic E-Health: Definitions, Applications, and Challenges 285
clinicians. Furthermore, the searches on Google and Facebook were performed in

English. Arabic searches would reveal more information on this topic.
Future research should focus on developing and studying the impacts of Islamic e-
health applications on health-related outcomes. Other research could focus on the
development of Islamic health privacy and confidentiality guidelines, the impacts of
virtual Ruqya on spiritual and mental health, the use of Hajj and Umra information
systems, and the use of telehealth for Hajj and Ramadan. Furthermore, to improve
future Islamic e-health services, religious authorities should monitor and develop
guidelines for the development of Islamic e-health applications. This process will
provide credibility to the field and will assist in advancing this research discipline.
Acknowledgements
I would like to thank Sheikh Dr. Ahmad Naji Ali, NGHA Religious Affairs, who spent
several hours helping me develop this idea. I would also like to thank Mr. Osama
Alghanmi and Drs. Wesley Rohrer, Munir Abuhelaleh, and Abdulrahman Alsughayr
for their valuable input. Furthermore, I would like to thank the King Abdullah Institute
for Medical Research for help in editing this document.
References
[1] H. Oh, C. Rizo, M. Enkin, A. Jadad. What is eHealth (3): A systematic review of published definitions.
JMIR, 7(1) (2005), e1.
[2] HIMSS. HIMSS E-Health SIG white paper. [cited Jan 10, 2012]. Available from:
http://www.himss.org/content/files/ehealth_whitepaper.pdf .
[3] J. Van Nooten, H. Oh, B. Pierce, F.J. Koning, A.R. Jadad. J Pastoral Care Counsel. 60 (4) (2006), 387-
94.
[4] S. Al Munajid. The Virtues of ruqyah and du’aa’s to be recited therein. [cited Jan 10, 2012]. Available
from: http://islamqa.info/en/ref/3476 .
[5] The Holy Quran. 3, 97.
[6] M. Yamin. A framework for improved hajj management and future research. School of Information
Sciences and engineering, University of Canberra, Australia. [cited Jan 7, 2012]. Available from:
http://hajjmanagement.com/attachments/documents/Yamin_Jedda_Conf.pdf .
[7] The Holy Quran. 2, 185
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-286
A User-centred Methodology for Designing

an Online Social Network to Motivate
Health Behaviour Change
Noreen KAMALa, and Sidney FELSa
1
a
Electrical & Computer Engineering, University of British Columbia, Vancouver,
Canada
Abstract. Positive health behaviour is critical to preventing illness and managing

chronic conditions. A user-centred methodology was employed to design an online
social network to motivate health behaviour change. The methodology was
augmented by utilizing the Appeal, Belonging, Commitment (ABC) Framework,
which is based on theoretical models for health behaviour change and use of online
social networks. The user-centred methodology included four phases: 1) initial
user inquiry on health behaviour and use of online social networks; 2) interview
feedback on paper prototypes; 2) laboratory study on medium fidelity prototype;
and 4) a field study on the high fidelity prototype. The points of inquiry through
these phases were based on the ABC Framework. This yielded an online social
network system that linked to external third party databases to deploy to users via
an interactive website.
Keywords. health behaviour change, online social network, user-centred design,

ABC framework, prevention
Introduction
Leading a healthy lifestyle and making positive health behaviour changes have been
found to be key to preventing illness and managing chronic diseases. In fact, self-
management of ones health has been shown to be of key significance in achieving
positive health outcomes for all people, healthy and sick [1, 2, 3]. Furthermore, we
intuitively understand that our life choices are heavily influenced by family, friends,
colleagues and other connections we have. A significant factor in health behaviour is
one’s close and distant social networks, which have been found to be a contributing
factor to health outcomes, where ones social networks can be used to improve health
behaviour through facilitating social integration and social support [4].
There is recent interest in the field of Human-Computer Interaction (HCI) to
explore online social networks and health behaviour change [5, 6]. Further, HCI
researchers have looked at designing technologies to promote a more active lifestyle [7,
8] and a more nutritious diet [9].
1
Corresponding Author: PhD Candidate, Electrical & Computer Engineering, Vancouver, BC, Canada;
E- mail: [email protected]
N. Kamal and S. Fels / A User-Centred Methodology for Designing an Online Social Network 287
Existing research has not looked specifically at designing online social network
and online social games to motivate health behaviour change. For this reason, we
explore the design space for an online social network system for positive health
behaviour change through a user-centred design process. User-centred design is a
methodology used in Human-Computer Interaction (HCI) as a process of interaction
design [10]. We base the user-centred design process on existing theoretical models on
health behaviour change and participation in online social network. These theoretical
models provide specific behavioural determinants that yield the Appeal Belonging
Commitment (ABC) Framework. The determinants from the ABC Framework provide
the points of inquiry throughout the user-centred design methodology. We use this
methodology by first completing an initial user inquiry and developing prototypes with
increasing fidelity for an online social network system that we call VivoSpace. The
prototypes that were developed included a low-fidelity paper prototype, an interactive
medium-fidelity prototype and fully functional high-fidelity prototype.
1. User Centred Design Methodology
The user-centred design methodology that we used to develop VivoSpace is shown in

Figure 1. The methodology starts with a literature review of existing theoretical models
for motivating health behaviour change and participating in online social networks to
yield the ABC Framework. The determinants of behaviour then provide the points of
inquiry throughout the methodology. The user-centred design method begins with
initial user inquiry that is evaluated through questionnaires and interviews to better
understand motivations of health behaviour change and participation in online social
networks. Based on the results of the initial user inquiry, paper-prototypes are
developed that are evaluated through interviews. The design is iterated and a medium
fidelity interactive prototype is developed and evaluated through a lab study. Finally,
the design is iterated again into a working high-fidelity prototype that is evaluated
through a field study.
1.1. Appeal Belonging Commitment (ABC) Framework
The ABC Framework was developed through the distillation of 13 theoretical models.
The theoretical models for health behaviour change included in the ABC Framework
are Health Belief Model, Social Cognitive Theory, Theory of Reasoned Action, Theory
of Planned Behaviour, Common Sense Model, and The Transtheoretical Model. The
theoretical models for participation on online social networks included in the ABC
Framework are Uses and Gratification Theory, Common Identity Theory, Common
Bond Theory, Social Identity Theory, Organizational Commitment Theory, Behaviour
Chain for Online Participation and social network threshold. These theories yield a
framework that provides: individually-based determinants (appeal) that include self-
efficacy, knowledge, social enhancement and expectations about outcomes; socially-
based determinants (belonging) that include sense-of-belonging, subjective norms and
social categorization; and temporal stages (commitment) [11, 12].
288 N. Kamal and S. Fels / A User-Centred Methodology for Designing an Online Social Network
Figure 1. User-centred design method by using the ABC Conceptual Framework to form the inquiry through
the iterative evaluation techniques.
1.2. User-Inquiry
The initial user inquiry for this methodology used online and paper questionnaires. The
questionnaires inquired about demographic information such a gender, age and ethnic
identity. The inquiry then used a 5-point Likert Scale to inquire each respondent’s
agreement to the determinants from the ABC Framework. The results yielded good
agreement to the ABC Framework. Generally respondents felt that they were living a
healthy lifestyle, understanding how to live healthy, eating healthy food and exercising
regularly. However, interestingly, the majority of respondents felt that they are capable
of living a healthier lifestyle. Most respondents also recognized the social influences
on their health. Furthermore, older respondents (over 65 years old) felt that they ate
healthier food more than young adults. Furthermore, Chinese and South Asians used
online social networks for social enhancement (a determinant from the ABC
Framework) more than Canadians. Generally, respondents felt stronger about
connecting with similar individuals than belonging to a group or community, but both
showed strong motivation [13, 11].
1.3. Paper Prototypes
Based on the determinants from that ABC Framework, an initial design for
VivoSpace was developed. The paper prototypes were developed using Adobe
Illustrator. There were 14 pages in total. The paper prototype was evaluated through
individual in-person interviews with 11 participants. They were shown each page of
the prototype and described the key functionality of the design. Participants felt that
they did not want to have their health data in the same place as other personal digital
assets. However, they liked the idea of social interaction with their health information
and the dashboard. They also wanted to see a greater focus on goal setting and
gamification [13].
N. Kamal and S. Fels / A User-Centred Methodology for Designing an Online Social Network 289
Figure 2. VivoSpace application’s system architecture.
1.4. Medium Fidelity Prototypes
From the results of the interviews conducted on the paper prototype, an interactive
medium fidelity prototype was developed for VivoSpace. The medium fidelity
prototype was developed using HTML, CSS, Javascript and jQuery with the vision to
develop a realistic interactive representation of our system. In total there were 32
HTML pages, 1 CSS file, and 2 javascript files. The medium fidelity prototype was
evaluated in a laboratory with 36 participants. There were 6 groups of tasks that
participants were asked to complete; after each task group, they were asked to complete
a questionnaire that was based on the ABC Framework. Belonging was evaluated
using indirect inquiry through the adaption of the Helping Game Experiment from
behavioural economics. Similarly, Commitment was evaluated through the In-Group
Experiment from social psychology. The results showed good agreement with the ABC
Framework; however, the design should provide greater motivation for users to
provide information [14].
1.5. High Fidelity Prototype
Based on the previous designs and evaluation, a fully functional high fidelity prototype
has been developed for VivoSpace. The system architecture for the high-fidelity
prototype is shown in Figure 2. The online social network system was developed in
PHP using the CodeIgniter [15] web application framework. The CodeIgniter web
application framework enforces a Model-View-Controller (MVC) development pattern,
which separates the application logic from the presentation: the model represents the
data structures; the view contains the code for presenting the information to the user;
and the controller contains the bulk of the application logic and processes. The
database used was MySQL, and nutritional information for food was obtained from an
external database, Wolfram Alpha [16], through an Application Programming Interface
(API). The high fidelity prototype is then used to allow people to use VivoSpace in
290 N. Kamal and S. Fels / A User-Centred Methodology for Designing an Online Social Network
their day-to-day lives.
2. Conclusions
The user-centred methodology augmented by the ABC Framework provided a means

to engage users in the design through a methodology that has theoretical underpinnings.
This methodology can be used for similar systems that have an end objective beyond
usability, as it allows the inquiry to be based on a conceptual framework that is
validated in the context of the system being designed.
References
[1] E.B. Fisher, C.A. Brownson, M.L. O’Toole, G. Shetty, V.V. Anwauri and R.E. Glasgow, Ecological
approaches to self-management: the case of diabetes, American Journal of Public Health 95(9) (2005),
1523–1535.
[2] T. Bodenheimer, K. Lorig, H. Holman and K. Grumbach, Patient self-management of chronic disease in
primary care, The Journal of the American Medical Association 288(19) (2002), 2469–2475.
[3] E.H. Wagner and T. Groves, Care for chronic diseases, The British Medical Journal 325(7370) (2002),
913–914.
[4] S. Cohen, Social relationships and health, American Psychologiest November (2004), 676-684.
[5] M.E. Morris, S. Consolvo, S. Munson, K. Patrick, J. Tsai and A.D.I. Kramer, Facebook for health:
opportunities and challenges for driving behavior change, Proceedings of the 2011 annual conference
extended abstracts on Human factors in computing systems, CHI EA’11, (2011), 443–446.
[6] M.W. Newman, D. Lauterbach, S. Munson, P. Resnick and M.E. Morris, It’s not that i don’t have
problems, i’m just not putting them on facebook: challenges and opportunities in using online social
networks for health, Proceedings of the ACM 2011 conference on Computer supported cooperative
work, CSCW ’11, (2011), 341–350.
[7] S. Consolvo, D.W. MadDonald and J.A. Landay, Theory-driven design strategies for technologies that
support behavior change in everyday life, Proceedings of the 27th international conference on Human
factors in computing systems, CHI ’09, (2009), 405–414.
[8] J.J. Lin, L. Mamykina, S. Lindtner, G. Delajoux and H.B. Strub, Fish’n’Steps: encouraging physical
activity with an interactive computer game, Proceedings of the 8th international conference on
Ubiquitous Computing, UbiComp ’06, (2006), 261–278.
[9] A. Grimes, V. Kantroo and R.E. Grinter, Let’s play!: mobile health games for adults, Proceedings of the
12th ACM international conference on Ubiquitous computing, Ubicomp ’10, (2010), 241–250.
[10] H. Sharp, Y. Rogers and J. Preece, Interaction Design: Beyond Human-Computer Interaction, Wiley,
West Sussex England, 2007.
[11] N. Kamal, S. Fels, M. Blackstock and K. Ho, The ABCs of Designing Social Networks for Health
Behaviour Change: The VivoSpace Social Network, In E. Kranakis (ed.) Advances in Network Analysis
and its Applications, (2012) Springer Berlin / Heidelberg. 307–337.
[12] N. Kamal, S. Fels and K. Ho, Online social networks for personal informatics to promote positive
health behavior, Proceedings of second ACM SIGMM workshop on Social media, WSM ’10, (2010),
47–52.
[13] N. Kamal, S. Fels, M. Blackstock and K. Ho, VivoSpace: Towards Health Behavior Change using
Social Gaming, In J. C. Anacleto, S. Fels, N. Graham, B. Kapralos, M. S. El Nasr, and K. Stanley
(Eds.), Entertainment Computing ICEC 2011, Lecture Notes in Computer Science 6972 (2011), 319-
330.
[14] N. Kamal and S. Fels, Determining the Determinants of Health Behaviour Change Through Online
Social Networks. In M. Bang, E.L. Ragnemalm (Eds), Persuasive 2012, Lecture Notes in Computer
Science 7284 (2012), 1-12.
[15] CodeIgniter, [cited Sep 14, 2012]. Available from: http://codeigniter.com .
[16] Wolfram Alpha, [cited Sep 14, 2012]. Available from: http://www.wolframalpha.com.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-291
Social Media and Patient Self-

Management: Not all Sites are Created
Equal
Leonie ELLISa,1, Chris SHOWELLa and Paul TURNERa
a
eHealth services Research Group, University of Tasmania
Abstract. This paper compares two social media sites that aim to support patients
to enhance self-management. The first site, PatientsLikeMe is a well established
global site designed to allow peer-to-peer communication between people with
similar conditions. The second, HealthShare, is a recently developed site for
Australians described as “Australia’s Social Health Network”. The comparison
conducted examines the purpose, ownership, and design of both sites as well as
how the data they collect is used. Analysis highlights that PatientsLikeMe actively
facilitates patient self-management, while HealthShare is revealed to be a
professionally moderated health information portal presented as a social
networking site. While the impetus for the development of PatientsLikeMe is
clear, the motives underpinning HealthShare are less obvious. With increasing
patient interest in connecting with, and sharing information with one another,
awareness of the nature and motivations underpinning sites that provide these
services is of increasing relevance.
Keywords. social media, self-management, PatientsLikeMe, HealthShare
Introduction
Rising costs of health care and an aging population are placing strains on most health
care systems. Alternative mechanisms are being sought to provide support for patients,
especially those with chronic disease.
Self-management has been described as “…a dynamic process incorporating
an individual’s capability and confidence to engage in activities which enable them to
deal with the impact of living with a chronic condition on all aspects of their life” [1].
Opportunities for self-management have recently increased with the development
of Web 2.0 [2]. Social Media provide opportunities for patients to support each other
outside the formal medically controlled environment by sharing personal health
information and experiences. In some instances, these opportunities are changing the
traditional ‘Patient-Physician’ relationship to one where patients interact with one
another to acquire mutual support, learning and self-management [3]. Technology now
provides a mechanism to create person-to-person support groups online, providing
patients with peer support from others with similar diseases, along with ‘expert’ advice
from the patient perspective [4].
1
[email protected].
292 L. Ellis et al. / Social Media and Patient Self-Management: Not All Sites Are Created Equal
In order to better understand how these identified trends are working in practice,
two social media sites offering support for patient self-management were identified for
comparison. The two sites were: PatientsLikeMe [http://www.patientslikeme.com/] and
HealthShare [http://www.healthshare.com.au/].
1. Method
In comparing the self-management support offered by both sites a simple content

analysis was conducted that examined each site’s stated purpose, ownership, design
and the information provided on how any data collected is used. While the comparison
could have adopted the HON Code criteria [5] the aim was to primarily focus on how
these sites described and subsequently supported patient-to-patient interactions to
support self-management, rather than to assess the quality of any information they
provided per se.
2. PatientsLikeMe
PatientsLikeMe was co-founded in 2004 by three MIT engineers, Ben Heywood, Jamie
Heywood and Jeff Cole, as a means of providing support for Stephen Heywood, the
brother of Ben and Jamie, who had been diagnosed with amyotrophic lateral sclerosis
[ALS]. Their aim was to use Web 2.0 technology to provide a mechanism for people
like Stephen to connect and share information with other ALS patients [6].
The aim of the site is for patients to use it in conjunction with other patients who
have the same disease. Patients report their own health information which is then
presented graphically in their profile and shared with other patients, providing
opportunities for active dialogue amongst patients [3]. The goal of the PatientsLikeMe
website is to help patients answer the question: “Given my status, what is the best
outcome I can hope to achieve, and how do I get there?” [7].
Used responsibly PatientsLikeMe can also be an important tool for medical
research [6]. Through member participation information is gathered to better
understand disease trajectories, medical treatments and medications. Data provided by
patients is used to discover trends and evaluate new possible treatments [8].
PatientsLikeMe conducts research by investigating the patient data. Researchers
administer surveys to groups on the site to gather additional patient self reported data.
This has resulted in numerous publications that have contributed useful information to
the broader medical community [7, 9, 10].
The PatientsLikeMe site provides clear information about the founders, and how
and why the site was originally developed, including a declaration that the company is
‘for-profit’ but not one with a ‘just for profit’ mission. It states “We follow four core
values: putting patients first, promoting transparency [“no surprises”], fostering
openness and creating ‘wow” [11].
While the site offers privacy settings and indicates that patients can volunteer their
"personally identifiable information," PatientsLikeMe retains the right to share the de-
identified data with their partners and with other patients [12].
L. Ellis et al. / Social Media and Patient Self-Management: Not All Sites Are Created Equal 293
3. HealthShare
HealthShare, was established in 2009 by Darryl Jackson and Gavin Solsky, it is owned
by Adin Holdings Pty Ltd. Jackson has expressed the view that “…technology [can]
change and improve the 'old world' way of sharing healthcare experiences and
recommendations” [13]. The site was founded to help Australians with life saving
medical advice. The death of a 25 year old fitness instructor by overdosing with
paracetamol was given as an example of the reason for establishing the site by helping
people by providing information to them [14]. HealthShare’s vision is to become the
leading online presence in Australia for health consumers, professionals, community
organisations and service providers to collaborate. The co-founder Darryl Jackson
previously worked for the Australian Self-management Industry [ASMI], the peak
body representing companies involved in the manufacture and distribution of consumer
healthcare products in Australia. The site claims that “HealthShare’s program delivers
enhanced public perception of your brand” and “trust and an emotional connection with
health consumers and practitioners” [15].
The site is designed as a set of moderated communities, each dealing with a single
disease. The site includes a range of health communities covering over 350 health
topics. HealthShare has the support of health organisations such as the Heart
Foundation, Diabetes Australia and Alzheimers Australia. HealthShare identifies itself
as “Australia’s Social Health Network” where people can find other people with similar
health experiences. The structure of HealthShare is similar to Wikipedia, but with
credible moderated information, a type of ‘Facebook’ for health. The site was designed
to provide information for a range of communities commencing with the major health
issues affecting many Australians and then subsequently growing to added those issues
affecting smaller numbers of Australians. Overall the site displays a strong focus on
chronic diseases and topics related to healthy living.
Although the site is privately funded, the founders have a sponsorship program that
involves content partners working with HealthShare to have the opportunity to sponsor
particular health communities through the site [16]. HealthShare also provides the
opportunity for patients to recommend healthcare professionals. This is not a rating
system, and there is a need to explain why you would recommend that particular
healthcare professional. The CEO states that the average Australian needs more
information when choosing a health practitioner. HealthShare has a statement
identifying that the owners take privacy seriously by providing complete privacy
protection and unparalleled security. Personal information will not be released to any
other party.
4. Results
Web 2.0 has provided opportunities to support communication and information sharing
in a way that has not been available before. PatientsLikeMe has taken advantage of
these new opportunities by providing a social networking site focused on the patient
and her individual health. From initial registration onwards the focus is the patient as
an individual (your health, your current status, and the medication you are taking and
how this is affecting you). The site then offers individuals the opportunity to see other
people with similar health issues, taking similar medication or having similar reactions.
PatientsLikeMe also allows a patient to identify multiple concurrent conditions for
294 L. Ellis et al. / Social Media and Patient Self-Management: Not All Sites Are Created Equal
example registering for breast cancer and menopause together. The site is moderated
and professional healthcare information is available. Forums are grouped into higher
level categories such as Women’s Health and Pregnancy thereby facilitating broader
discussion of cross-related issues.
In contrast, although HealthShare has expressed its aim as being “Australia’s
Social Health Network” the site’s functions and features mean it is actually a
conventional web-portal that is tightly moderated to primarily provide health
information from healthcare professionals. When registering for HealthShare the focus
is on the particular medical condition, not the patient. Users are unable to link medical
conditions together into one community and instead, individual conditions are
presented in medically defined condition specific silos. For example, a patient with
breast cancer and menopause would be required to access different parts of the site to
acquire information on these conditions and would not be able to easily integrate the
separate streams of information and advice provided, even though both conditions
affect the same individual patient. HealthShare primarily operates with an ‘information
push’ model such that information is sent out to the user. Most of the content on the
site appears to be provided by numerous healthcare provider moderators. Where a
question is posted on the site it is usually answered by the healthcare moderator, rarely
by another patient user.
5. Discussion
The US based, PatientsLikeMe is a global site for all to use and share. HealthShare is a
site expressly designed for Australians. The HealthShare perspective is that a localised
service is necessary to avoid a situation of information not being relevant to Australians
(such as the unavailability in Australia of a particular drug being prescribed for a
particular disease). This perspective highlights the different underlying assumptions in
the two sites about the role of social media in supporting patient self-management and
how the advice, experiences and support available from patients for patients is valued.
Significantly, while both sites are owned and operated by private companies and
both purport to be about supporting patients to enhance self-management, there appears
to be a strong contrast in the nature, motivations and interests these sites and their
owners display in relation to patients.
PatientsLikeMe transparently declares that the business is a profit making
enterprise. The site uses the data collected from the site to support research and clinical
trials. PatientsLikeMe have their own research team and a brief scan of the current
literature soon reveals that these industry researchers also work in conjunction with
academics to investigate the data. While some recent discussions have begun to
question the use of this data [17] to date, PatientsLikeMe have continued to be
‘transparent’ in declaring how data is used and by whom.
In contrast HealthShare provides no clear information about whether or how any
de-identified data is used. There is no clear information relating to the profit status of
the company, or its business models and prospective business plans. Is HealthShare an
altruistic initiative motivated by a concern for Australian patients, or is it an example of
Astroturfing – a commercially driven activity leveraging patient interest in social
networking to generate profit from health profiling and data usage of its user base?
From the information available through the site it is difficult to tell definitively!
L. Ellis et al. / Social Media and Patient Self-Management: Not All Sites Are Created Equal 295
The rapid emergence of health focused social media has leveraged the strong
interest and appetite amongst many ‘internet empowered’ patients to connect with one
another and to openly and interactively share knowledge, experiences and attitudes
about their health and well-being. Simultaneously, on-going debates on data protection
and the security of electronic patient records as well as risks associated with personal
health profiling and business ethics related to harvesting patient data for profit have
emerged.
As the comparison of the two sites above illustrates, different health focused social
media sites support and value patients in very different ways. These differences
however are most evident, not in the health information provided by each site per se,
but rather in how patients are able to interact about their conditions with other patients.
Understanding and identifying the nature and motivations underpinning health related
social media sites is another dimension for patients to consider when seeking internet
support to self-manage.
References
[1] H. Cameron-Tucker, A study of supervised exercise with a chronic disease self-management programme
for people with COPD, University of Tasmania, PhD Thesis 2009, Australia
[2] T. O'Reilly, What is Web2.0- Design Patterns and Business Models for the Next Generation of Software.
2005 [cited 2012 January], Available from: http://oreilly.com/web2/archive/what-is-web.html.
[3] J.H. Frost, M.P. Massagli, Social Uses of Personal Health Information Within PatientsLikeMe, an Online
Patient Community: What Can Happen When Patients Have Access to One Another’s Data. Journal of
Medical Internet Research 10(3) (2008).
[4] D. Hoch, T. Ferguson, What I've Learned from E-Patients. PLoS Med 2(8) (2005),e206.
[5] Health on the Net Foundation (HON Code) [cited October 2012]; Available from:
http://www.hon.ch/HONcode/Patients/Visitor/visitor.html.
[6] L.J. Trevena, PatientsLikeMe and the tale of three brothers. Medical journal of Australia 195(5) (2011),
258-9.
[7] P. Wicks, M. Massagli, J. Frost, C. Brownstein, S. Okun, T. Vaughan, et al, Sharing Health Data for
Better Outcomes on PatientsLikeMe. Journal of Medical Internet Research 12(2) (2010).
[8] Anonymous. UCB to Mine Social Media Website for Data. Pharmaceutical Representative 39(8) (2009),
8.
[9] J. Frost, S. Okun, T. Vaughan, J. Heywood, P. Wicks, Patient-reported outcomes as a source of evidence
in off-Label prescribing: Analysis of data from PatientsLikeMe, Journal of Medical Internet Research 13(1)
(2011).
[10] G. Burnett, M.L.A. Lustria, J. Cortese, M. Kazmer, J. Frost, J.H. Kim, et al. PatientsLikeMe: Social
norms and information values in an online health support group. Proceedings of the American Society for
Information Science and Technology 46(1) (2009), 1-4.
[11] J. Heywood. PatientsLikeMe. 2005 [cited Oct 2012]; Available from:
http://www.patientslikeme.com/about.
[12] Anonymous. PatientsLikeMe. 2005 [cited January 2012]; Available from: www.patientslikeme.com.
[13] Darryl Jackson HC. HealthShare. 2011 [cited Jan 2012]; Available from:
http://www.healthshare.com.au/docs/DOC-6685.
[14] Channel 9 Television Australia, Kerri-Anne Morning Show Interview 13th October 2011. [updated 2011;
cited Oct 2012] Available from: http://www.youtube.com/watch?v=YKm_3qtSU7k.
[15] Darryl Jackson HC. HealthShare Pty Ltd [updated 2012; cited Oct 2012]; Available from:
http://www.healthshare.com.au/questions/44932-how-can-our-company-sponsor-a-healthshare-community.
[16] HealthShare Pty Ltd [cited Oct 2012]; Available from:
http://www.healthshare.com.au/sponsor-profile-benefits.
[17]BiolArtist2012 Science-Based Medicine: Exploring issues and controversies in the relationship between
science and medicine [cited Oct 2012] Available from:
http://www.sciencebasedmedicine.org/index.php/the-perils-of-patient-driven-clinical-research/comment-
page-1/.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-296
Empowering Saudi Patients: How do Saudi

Health Websites Compare to International
Health Websites?
Mowafa HOUSEHa, Abdulrahman ALSUGHAYARa, Maha AL-MUTAIRI a,1
a
Abstract. Little information is known about how Saudi health websites compare
internationally. The purpose of this paper is to compare two leading Saudi health
websites with leading international health websites. The study was conducted as a
convenience sample at a graduate health college in Saudi Arabia. A total of 42
students participated in the study. The study found that, in general, English
websites have higher levels of performance with regard to quality of information,
authority and objectivity, coverage and currency, and design. However, the
respondents considered Saudi health websites to be superior with regard to
maintaining privacy and security. The results indicate that much more work is
needed in designing Saudi Health to make them more trustworthy and credible.
The limitations of this work and future research directions are also discussed.
Keywords. consumer informatics, Saudi Arabia, health websites, and patient

empowerment
Introduction
For years, the development of websites with trustworthy health information within the
Arab world has been neglected. Much of the available Arabic health information has
focused on a particular healthcare organization’s goals, services, and policies [1]. A
study conducted by Al-Tuwaijri determined that, of the 122 Arab health websites
evaluated in terms of providing high quality health information, only five provided
trustworthy health information to the Arab consumer [1]. As a result of this research,
various healthcare organizations within the Arab world began to develop health
information websites that are trustworthy and credible to improve health literacy and
empower patients. A recent study on how Saudis use online health information found
that 58% of those surveyed used the internet to search for health information [2]. In the
study, most patients discussed the health information found online with their physician
and believed that sharing such information improved patient-physician relationships
[2].
1
[email protected]
M. Househ et al. / Empowering Saudi Patients 297
With the growing need for trustworthy and credible health information in Saudi
Arabia, various healthcare organizations have developed health information websites
for the Saudi public. This paper evaluates two leading healthcare information websites
sponsored by public healthcare organizations within the country. It compares two
leading Saudi health websites to leading English websites in Canada, England, and
Australia. The study evaluates various dimensions related to providing trustworthy
health information online. The King Abdullah Bin Abdulaziz Arabic Health
Encyclopedia (KAAHE), one of the largest initiatives for providing credible online
health information within the Arab world, was not included in this study because it was
not available online when the study was conducted.
1. Methodology
A convenient sample survey was conducted using male and female students at the
College of Public Health and Health Informatics at King Saud Bin Abdul Aziz
University for Health Sciences, Riyadh, Kingdom of Saudi Arabia from November
2011 to January 2012. The College of Public Health and Health Informatics is the first
college in Saudi Arabia to offer a graduate program in Public Health and Health
Informatics within the country of Saudi Arabia. The program accepts applications from
the various regions within the country as well as the Gulf region. The sample represents
some of the most highly educated groups within the country of Saudi Arabia.
1.1. Data Collection
The survey used by Ali et al. to evaluate Egyptian health websites was used in the
present study to evaluate two leading Saudi health websites [3]. The tool consists of
one technical category and one ethical category. The technical category evaluates six
items related to authority, objectivity, coverage, currency, design, privacy and security.
The ethical category evaluates the quality of information, informed consent, and
professionalism.
The administered questionnaire included 49 questions and took approximately 3
hours to complete. The questionnaire was e-mailed to each participant with a Microsoft
Excel attachment. The Microsoft Excel attachment included survey questions and clear
instructions on how to input data and return the survey. The participants were provided
one week to complete the survey. Participation in the study was voluntary, although
completing the survey constituted 20% of the student’s final grade.
The participants were asked to evaluate two Saudi websites using the survey.
Through random selection, two students were asked to evaluate a leading English-
language health website. A total of 21 English websites were included in the
evaluation: 9 from the U.S.A, 5 from the United Kingdom, 3 from Australia, and 3
from Canada. In addition, after they completed the survey, the participants were asked
to rank all 21 English health websites from 1 to 10, with 1 being the least trustworthy
and usable and 10 being the most trustworthy and usable. The researchers then added
the scores and divided them by 21 to give a total score between 1 and 10. Any number
between 1 and 4 was classified as having low trustworthiness and usability; health
websites that scored 5-7 were classified as having medium trustworthiness and
usability; and websites with scores of 8-10 were classified as having high
298 M. Househ et al. / Empowering Saudi Patients
trustworthiness and usability. In this study, the Saudi health sites were compared to the
four highly rated and trustable English websites (as ranked by the respondents).
1.2. Analysis
Data was analyzed by descriptive data analysis and graphs using Microsoft Excel.
2. Results
All 42 students returned their survey within one week of receiving it. The participants
in the convenience sample included 31 females and 10 males. Their ages ranged
between 24 and 55 years. Most of the students were in health sciences programs
encompassing various professions such as health informatics, medicine, nursing,
pharmacy, medical sciences (laboratory technologists, x-ray techs, etc.), and dentistry.
2.1. Quality of information
Every respondent noted that all highly rated English sites provided a higher quality of
information than Arabic websites and mid-to-low quality English websites. Each
respondent also noted that medical professionals provided the health information
available on top rated English websites, while nearly half the respondents noted that
Saudi health websites contained information that was written by medical professionals
(52% and 55% for the two Saudi websites considered). Furthermore, every respondent
noted that the information provided by highly rated English websites included scientific
studies, while this was the case for only 43% and 36% of the two Saudi websites. All
respondents indicated that the language was clear for the top rated English websites; in
this regard, high ratings were also given to the Saudi websites (98% and 95%.) Only
63% of respondents found that the highest ranked English websites provided a
publication date, while a much lower percentage respondents found that the Saudi
websites did so (33% and 26%).
With regard to the use of references, 88% of respondents noted that the highly
ranked English websites provided information on their sources, while much lower rates
for Saudi websites were reported (24% and 7%). With respect to HONCODE
certification, which certifies credible and trustworthy websites, 75% of respondents
reported that highly rated English websites have HONCODE certification, while none
of the Saudi websites were found to have HONCODE certification.
2.2. Authority and Objectivity
This study also considered the authority and objectivity of each website. When
considering the highly ranked English websites, 88% of respondents reported that
authors were identified. Much lower rates of author identification were reported for the
Saudi websites (31% and 26%). Three-quarters (75%) of respondents found that the
highly ranked English websites clearly stated institutional affiliations on their website;
on the other hand, most respondents noted a lack of clarity in this regard on the Saudi
health websites considered (67% and 57%). Respondents also found that most of the
highly rated English health websites contained the authors’ contact information (75%),
while much lower rates were indicated for the Saudi health websites considered (33%
and 24%).
Regarding the websites’ purpose(s) and objectives, all highly ranked English
websites were found to clearly state their objectives. Most respondents reported that the
purposes of the Saudi health websites were clearly stated (71% for both sites).
Respondents also noted that, for all highly rated English websites, the material
presented was in line with the objectives. Most respondents also noted that the material
presented on Saudi health websites corresponded with the objectives (75% and 62%).
2.3. Coverage and Currency
All respondents noted that the top English websites satisfied their information needs.
Similarly, almost all respondents (95%) noted that one Saudi health website satisfied
their information needs, but the other Saudi website had a much lower rate (63%) for
satisfying the informational needs of the respondents.
The majority (88%) of respondents found the highly ranked English websites to be
current and up to date. This measure was lower for Saudi websites, 79% and 33%. All
respondents noted that all highly rated English websites were up to date, while only
86% considered one Saudi website to be up to date and only 48% considered the other
to be current.
2.4. Design
All respondents noted that the highly rated English websites could be accessed and
navigated easily. The respondents also indicated that both Saudi websites were easily
accessed and navigated (86% and 83%). Almost all respondents (88%) noted that the
highly ranked English websites used visuals to convey information. Students reported
that one Saudi website was better than the English websites in using visuals to convey
information (95%). For the other Saudi website, only 74% of respondents reported the
use of visuals to convey health information.
All respondents reported that high quality graphics were used in the highly ranked
English websites, while the respondents indicated lower satisfaction with Saudi
websites (81% and 62%). The respondents also found that all highly ranked English
websites were useful, while a much lower number of respondents (76% and 52%)
found the Saudi health websites to be useful. Most of the respondents (88%) noted that
the highly ranked English websites had interactive features that were easy to use, while
the respondents reported lower satisfaction with the Saudi health websites in this regard
(74% and 64%).
Regarding software updates, all respondents reported that the highly ranked
English websites could be accessed without additional viewers or plug-ins. Only 86%
and 81% of the respondents noted that the Saudi websites could be accessed without
software plug-ins. With regard to informing or providing support for downloading the
software required, both Saudi health websites ranked higher (52% and 29%) than the
high-ranking English language websites; only 25% of the respondents noted that the
highly ranked English websites provided information regarding where to download the
plug-ins required.
300 M. Househ et al. / Empowering Saudi Patients
2.5. Privacy
Regarding privacy, almost all of the respondents (95% and 79%) considered the Saudi
websites to be more secure than the highly rated English websites (which had a support
rating of 75%). Regarding unauthorized data access, most of the respondents (79% and
75%) considered both Saudi websites to be better than the highly rated English
websites (which had a support rating of 75%).
3. Discussion
This study assessed and compared two highly ranked Saudi websites with highly
ranked English websites. Overall, the study found that English websites performed
better with regard to quality of information, authority and objectivity, coverage and
currency, and design. However, the respondents also indicated that the Saudi health
websites were better in maintaining privacy and security. The results indicate that much
work is needed in designing Saudi Health websites to be more trustworthy and credible
compared to English sites.
A current initiative is attempting to address these shortcomings. The King
Abdullah Bin Abdulaziz Arabic Health Encyclopedia (KAAHE) was launched on the
Internet in Saudi Arabia in April 2012. This initiative, led by the Saudi National Guard
Health Affairs organization, aims to provide credible and reliable health information to
the Arab world [4]. It is, therefore, different from the websites evaluated in this study,
because the websites in this study are more local and focus on specific populations or
disease groups. By providing a credible and trusted source of health information,
KAAHE is expected to improve health literacy and to indirectly improve patient safety
in Saudi Arabia and throughout the Arab world.
4. Limitations and Future Research
The present study compared two highly ranked Saudi websites with highly ranked
international English websites. This research was limited by its use of a convenience
sample, which may limit the results’ generalizability and validity. The sample used in
this study was composed of students who were highly educated and familiar with the
use of technology and health websites. Future research should be conducted in
hospitals using a wider and more diverse population and should assess the performance
of the KAAHE project within Saudi Arabia and the Arab world. Another limitation of
this study is the use of descriptive statistics; future plans include providing more
detailed statistical analyses.
References
[1] E.M. Borycki, M.S. Househ, A.W. Kushniruk, C. Nohr, H. Takeda, Empowering Patients: Making
Health Information and Systems Safer for Patients and the Public. Contribution of the IMIA Health
Informatics for Patient Safety Working Group, Yearb Med Inform 7(1) (2012), 56-64.
[2] K. Alghamdi, N. Moussa, Internet use by the public to search for health-related information.
International Journal of Medical Informatics 81(6) (2012), 363-373.
[3] N.M. Ali, A. Gamal, A.A. Mohamed, A Pilot Study For Evaluating Egyptian Health Websites: Technical
And Ethical Perspectives, Bangladesh Journal of Bioethics 1(3) (2010).
[4] M.M. Altuwaijri, Empowering patients and health professionals in the Arab World. The King Abdullah
bin Abdulaziz Arabic Health Encyclopedia on the Web. Yearb Med Inform (2011), 125-30.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-302
Taming Mental–Health-Focused Popular

Literature: A Crazy Idea?
Robert T. ZOZUS, JR a, Audrey BRICKER b, Rob LUNBLADc, Dennis
ELIAS d, Meredith NAHM e
a
Private Practice Clinical Psychologist, Durham, NC
b
Private Practice Clinical Psychologist, Ann Arbor, MI
c
Private Practice Clinical Psychologist, Salem, OR
d
Private Practice Clinical Psychologist, Phoenix, AZ
e
Duke University Center for Health Informatics, Durham, NC
Abstract. Providing tailored and easily accessible health information for mental
health clinicians and patients can be enabled through Information Technology and
Communications (ITC). The literature is mixed regarding the quality, utility and
accessibility of health information in the popular press for this purpose. However,
there is consensus that mental health information in the popular press is readily
available, easily comprehended by patients, and is continually updated. We report
the process by which mental–health-focused articles in the popular press are
identified, screened, and disseminated to a large network of doctoral level
psychologists (the PsyUSA network). We analyze 4-year article distribution and
access data, and conclude that the distribution of mental–health-related popular
press articles prompted article access. We leverage this experience to formulate a
model for direct access to clinician-vetted mental–health-related popular press
through a curated web based archive.
Keywords. consumer health, information sharing, information retrieval, patient

education, knowledge representation/organization, consumer informatics
Introduction
As early as 2003, 80% of American consumers searched for health information on the
web, with 21% searching mental health-related topics [1]. A vast array of health
information is readily available online via newspapers, magazines, television news
outlets, and press releases; i.e., “the popular press.” Health information may be
provided via such outlets through reported research results, review articles, news
articles, exposés, expert interviews, book reviews, tips and advice. This information
educates and entertains, increasing awareness of mental health-related issues.
A major advantage of popular press psychological resources is that they are written
in laymen’s terms and can be understood by most. Further, articles in the popular press
are often available for free or nominal charge. However, health information delivered
by popular media can be compromised by quality, accessibility, and utility issues [2-7].
Relevant information needed to evaluate article quality often is missing [3, 8], and
articles can “disappear” or be sequestered behind pay walls.
The increasing focus upon patient-centered care warrants increasing the
availability of applicable, reliable, and valid information related to patient needs.
R.T. Zozus Jr. et al. / Taming Mental–Health-Focused Popular Literature: A Crazy Idea? 303
The PsyUSA network, a network currently consisting of 428 doctoral-level clinical

and counseling psychologists, offers insight into clinician access of popular-press
psychological information. One of the PsyUSA network’s main functions is to increase
psychologist awareness of new articles of psychological import. Network clinicians
have noted using these articles in patient education, as well as for increasing their
breadth of knowledge and keeping abreast of new developments in the field.
1. Background
The PsyUSA network initiated operations on October 28, 1995, with a goal of
providing a forum for doctoral-level psychologists to exchange information, interact,
broaden exposure to psychological topics and stay current. Toward the lattermost goals,
mental–health-related articles are disseminated by listserv (push technology) via email
“posts,” wherein brief portions of the article (as well as relevant links connecting to the
article in its entirety) are sent to psychologists by email each day. From May 1, 2006
through September 1, 2012, a total of 10,603 article snippets and links have been
distributed though the PsyUSA listserv—an average of 4.64 articles posted per day.
Volunteer doctoral-level clinical psychologists, referred to as “Newshounds,”
identify and disseminate posts to the network; i.e., upstream versus downstream
filtering [9]. Today, three Newshounds (first three authors) work for the network in this
capacity on a rotating weekly basis and spend an estimated 1.5 to 2 hours to prepare
weekly article postings. This time is spent in identifying, reading, and discerning which
popular psychology articles should be distributed, in formatting the information sent to
comply with copyright and listserv requirements, and in obtaining shortened universal
resource locators (URLs) to access the full article.
Identifying articles focused on popular psychology for dissemination to the
PsyUSA network is done manually, using different article identification strategies.
They include utilizing website aggregators of psychology-focused articles, Rich Site
Summary (RSS) or general website aggregators to track websites with frequent fresh
mental health related content; and setting up alerts to notify Newshounds of new
articles.
Screening articles for PsyUSA network distribution is based on Newshound
experience, educational background, and personal interest. Articles are screened in a
two-part process. Initially, titles and brief portions of articles are scanned. Based on
perceived interest to PsyUSA network members, Newshound interest, and perceived
quality of the article source; articles are selected for further review. The second phase
of the screening process begins as articles judged potentially worthy of dissemination
are read in full. Criteria considered include presentation of novel information or ideas
likely to stimulate discussion and thought, completeness of topic coverage,
transparency of information sources and assumptions (or full disclosure of primary data
or evidence), methodological rigor and relationship of the article’s discussion of
relevant research. Language use, general clarity, and organization are also considered
when judging article quality.
Selected articles are then prepared for dissemination. Fair Use provisions of U.S.
copyright law dictate that only the title, attribution, and short snippet from copyrighted
articles be distributed in a post. A web link to the original article is always provided.
304 R.T. Zozus Jr. et al. / Taming Mental–Health-Focused Popular Literature: A Crazy Idea?
2. Methods
This observational study reports analysis of data pertaining to one PsyUSA network
Newshound (the first author). All articles posted by this Newshound Since July 12,
2009 were assigned links using the Bit.ly URL-shortening service (www.bitly.com) for
all non-press release articles; full press releases are intended for public dissemination
and thus were distributed directly in the post. Because Bit.ly has been reported as the
most popular URL-shortening service since May of 2009 [10, 11], it offers the best
indirect means of estimating article popularity for the general public.
The Bit.ly service provides one Bit.ly link for each article. Standard Bit.ly metrics
break down the total number of clicks by referrer (in this case, the Newshound), as well
as the total number of clicks on Bit.ly to the article from any referrer. The Bit.ly
metrics were analyzed to explore full article access by PsyUSA network members.
These analyses were conducted on aggregate web statistics, from which individual
behavior cannot be determined. Additionally, the first author kept records of article
screening, e.g., articles that he read through fully but did not disseminate because of
quality concerns (using a service called Instapaper <http://www.instapaper.com/>).
This research was submitted as Pro00041763 to the Duke University Institutional
Review Board and was granted exemption.
3. Results
From July 12, 2009 to August 21, 2012, the first author sent 1,170 posts to PsyUSA
members (approximately 324 per year). Bit.ly observational data during this period
shows that each post to the PsyUSA network generated on average 7 clicks (median of
3) to the full article. During the same period, PsyUSA members logged an average of
215 clicks per month on these posted articles. In the general population of non-PsyUSA
members, each of the Bit.ly links generated an average 320 clicks per article (median of
18) and 9,843 clicks on average per month.
Figure 1. Access by Year for PsyUSA Network Members
Figure 1 shows the average number of clicks per month on posted articles
disseminated by the first author, by year. The drop in Bit.ly referenced articles from
2010 to 2011 corresponds to an increase in press release distribution starting in 2011.
In 2009 and 2010 an average of 6.5 and 4.9 press releases were distributed each month
respectively; whereas in 2011 and 2012, the Newshound distributed 25 and 29 press
releases each month respectively. Recall that the full press releases text is distributed;
obviating the need to use Bit.ly and therefore lowering the average Bit.ly clicks per
month. A drop in PsyUSA network membership from 2007 to 2012 of about 75
members may have also contributed to the decline in Bit.ly accessed articles.
4. Discussion
Full access to psychologically focused articles as indicated by Bit.ly metrics was robust,
especially considering that articles were popular press, and “pushed” to psychologists
without regard to individual need or interest. We interpret clicks on Bit.ly links
disseminated by Newshounds as an indication that the recipient found the article of
sufficient interest to follow the link to the full article. We leave the reader to form
opinion as to whether or not exposure to psychological information of interest to a
practicing psychologist is of use to them in their practice. Network clinicians have
noted using these articles in patient education, as well as for increasing their
psychological breadth of knowledge and staying current in the field. There has been no
formal assessment of the uses and perceived helpfulness of distributed articles to
network members. Future survey research with PsyUSA network members could
provide such insight.
Given the article access with the “push” model of information dissemination in the
PsyUSA network, we posit that a “pull” model could be even more useful- e.g., a
searchable archive of vetted mental health focused articles accessible by patients with a
clinician-gated recommendation, or even the general population on demand. A recent
report that consumers receive a majority of their health information from “traditional
media and interpersonal sources” [12] underscores this need.
For such a “pull” model to work, a comprehensive, organized, easily searched web
archive of high quality popular press information must be created and maintained.
Finding relevant articles for inclusion in the database can be accomplished by
augmenting current methods with intelligent-search web crawlers to automate article
identification, application of heuristic algorithms to identify articles most likely to be
topically pertinent; and application of existing heuristics for health information quality
on the internet. While Newshounds currently operate informally to assess article
quality (described above), heuristics exist for assessing the quality of health
information on the web [13] that can also be applied and semi-automated. Further
other indications of article quality such as user feedback and rating may be employed.
Article tagging to facilitate information retrieval could be accomplished by
leveraging the Library of Medicine’s Medical Subject Headings (MeSH) database. It
uses common language, has natural language definitions, and provides a hierarchical
structure that could aid organization and search. Additional research is needed to
clarify patient and caregivers search and navigation strategies, and to ascertain the
usefulness of including a brief abstracted article summary along with the basic
information now provided of article an article snippet, title, attribution, and link to full
article.
Copyright considerations figure prominently in constructing such a database.
Because of the non-profit, educational nature of this work; inclusion of the article title,
306 R.T. Zozus Jr. et al. / Taming Mental–Health-Focused Popular Literature: A Crazy Idea?
as well as one or two article paragraphs should be allowed under Fair Use provisions of
U.S. copyright law, and Fair Dealing Doctrine of the Commonwealth Nations.
Maintenance of such a web-based archive would be needed. Such maintenance
would include ensuring that articles continue to be available at their cited web location,
and that they still are available freely (or at nominal fee).
As with all web based health information, users of this data base should be clear
that although psychology-focused popular media articles may offer advice and tips
(often by experts in the field), the information they provide should not be construed as
professional advice suited to an individual’s psychological health issues and needs.
Further research should include content analysis of high-use popular press to
discern which mental health articles generate the most consumer interest- and why.
Additional research may also inform how popular media psychology focused articles
can best be used for education purposes.
5. Conclusion
Mental–health-related popular press articles are accessed at a high rate by PsyUSA

network members. Our experience has highlighted necessary considerations for
increasing the scope of such information dissemination from doctoral level clinicians
only, to a patient resource open to the general public; and for changing the model from
a push of articles without regard to individual interest, to am on-demand resource.
6. Limitations
Due to the observational nature of this work, we cannot attribute cause. And we cannot
directly link clinician access to usefulness. Further, generalizability of our conclusions
is limited because PsyUSA network membership have self-selected based on perceived
value of receipt of popular-press articles. These findings are further limited by the
retrospective nature of data collection that relied on aggregate counts from Bit.ly as an
indicator of article use and popularity.
References
[1] S. Fox, D. Fallows, Internet Health Resources, 2003, Available from:

http://www.pewinternet.org/pdfs/PIP_Health_Report_July_2003.pdf.
[2] K.M. Griffiths, H. Christensen, Quality of web based information on treatment of
depression: cross sectional survey, BMJ 321 (2000), 1511–1515.
[3] L. Iaboli, L. Caselli, A. Filice, G. Russi, E. Belletti, The unbearable lightness of health science
reporting: A week examining Italian print media, PLoS ONE 5 (2010), e9829.
[4] M. Walji, S. Sagaram, D. Sagaram, F. Meric-Bernstam, C. Johnson, N.Q. Mirza, E.V. Bernstam,
Efficacy of quality criteria to identify potentially harmful information: a cross-sectional survey of
complementary and alternative medicine web sites, J Med Internet Res 6 (2004), e21.
[5] B. Kuriya, E.C. Schneid, C.M. Bell, Quality of pharmaceutical industry press releases based on original
research, PLoS ONE 3 (2008), e2828.
[6] D.B. Friedman, J.N. Laditka, S.B. Laditka, A.E. Mathews, Cognitive health messages in popular
women’s and men’s magazines, 2006-2007. Prev Chronic Dis 7 (2010).
[7] J.E. Stryker, Reporting medical information: Effects of press releases and newsworthiness on medical
journal articles' visibility in the news media, Prevent Med 35 (2002), 519–530.
[8] L. Alpay, J. Verhoef, B. Xie, D. Te’eni, J.H. Zwetsloot-Schonk, Current challenge in consumer health
informatics: Bridging the gap between access to information and information understanding. Biomed
Inform Insights 2 (2009), 1–10.
[9] G. Eysenbach, “Credibility of Health Information and Digital Media: New Perspectives and
Implications for Youth." In: Digital Media, Youth, and Credibility. Ed. M.J. Metzger, A.J. Flanagin.
The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning. Cambridge,
MA: The MIT Press, 2008. 123–154.
[10] J. Wortham, Bit.ly eclipses TinyUrl on Twitter, New York Times–Technology. Bits. [cited Sep 15, 2012],
[updated May, 2009], Available from: http://bits.blogs.nytimes.com/2009/05/07/bitly-eclipses-tinyurl-
on-twitter/.
[11] SiteOnn. Top 3 URL shorteners you should use. Available from http://siteonn.com/top-3-url-shorteners-
you-should-use, accessed on September 15, 2012.
[12] E.W. Maibach, D. Weber, H. Massett, G.R. Hancock, S. Price, (2006). Understanding consumers’
health information preferences: Development and validation of a brief screening instrument. J Health
Commun 11 (2006), 717–736.
[13] A. Jadad, R. Alejandro, A. Gagliardi, Rating health information on the Internet: Navigating to
knowledge or to Babel? JAMA 279 (1998), 611–614.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-308
Considerations for Personal Health Record

Procurement
Helen MONKMANa,1 and Andre KUSHNIRUK a
a
Victoria, British Columbia Canada
Abstract. Patients with chronic illnesses require tools and resources to facilitate
self-management. Personal Health Records (PHRs) are a promising option for
delivering these tools and resources to patients with chronic illnesses. As such,
many organizations are becoming interested in PHR procurement. However,
traditional procurement methods may not ensure the system success and adoption.
In this study a group of subject matter experts discussed the possibility of
converting a paper-based PHR into an electronic tool. These discussions resulted
in generation of several important criteria for assessing commercially available
PHR solutions and other considerations related to PHR procurement. These
considerations should be contemplated and discussed with stakeholders prior to
PHR procurement. In order to realize the benefits PHRs, it is imperative that the
appropriate selection is made. Prior to purchase commitment, a trial period can
prove extremely useful for performing usability analyses and ensuring
interoperability. Supplementing traditional procurement methods with these
preliminary user evaluations will increase the likelihood that the selected system
best matches the needs of users and purchasers. Moreover, the risk of system
failure and the risk of limited adoption of the PHR by the public will be reduced as
a result of adopting these methods.
Keywords. Personal Health Record (PHR), procurement, usability, chronic illness
Introduction
Chronic illnesses (e.g., arthritis, cancer, chronic obstructive pulmonary disease,

diabetes, heart disease, high blood pressure) cannot be cured and often occur in
conjunction with other chronic illnesses. Many of those suffering from chronic illness
do not receive sufficient care from their primary care providers. [1]. Chronic illnesses
place a heavy burden on the resources of the health care system and the demands on
healthcare are going to be exacerbated by the aging population [1].
Patients are the most important figures in chronic disease management because
they must deal with their illnesses on a daily basis [2]. Further, patient self-
management is an integral component to treatment to help mitigate both the physical
and psychological effects of chronic illness and improve the effectiveness and
efficiency of care [3].
An individual with chronic illness requires a set of tools tailored his or her
condition(s). Solomon [2] asserted that “by providing people with chronic disease the
1
Corresponding Author: Helen Monkman, email: [email protected]
H. Monkman and A. Kushniruk / Considerations for Personal Health Record Procurement 309
tools and support to self-manage their conditions, individuals become more

knowledgeable about their health situation and are more apt to adhere to self-care
practices that lead to better outcomes” (p. 400). One application that shows promise for
providing customized toolkits to patients with chronic illnesses is the Personal Health
Record (PHR).
The Medical Library Association (MLA) / National Library of Medicine (NLM)
Joint Electronic Personal Health Record Task Force [4] defines a PHR as:
A private, secure application through which an individual may access, manage,
and share his or her health information. The PHR can include information that is
entered by the consumer and/or data from other sources such as pharmacies,
labs, and health care providers. The PHR may or may not include information
from the electronic health record (EHR) that is maintained by the health care
provider and is not synonymous with the EHR (p. 244).
The adoption of PHRs reflects a paradigm shift from the patient as a passive
recipient of health care to one who is actively engaged in the health care process by
collaborating with providers and making decisions [5] as part of a patient-centric care
model. Moreover, although chronic illness may be the catalyst for launching PHRs,
healthy populations benefit can from their use as well.
Despite the praise for their potential to transform healthcare, patients have proven
to be reluctant to adopt PHRs on their own accord. In Canada (n = 2 304), merely 6%
of consumers already maintain a PHR; however, 61% would be receptive to their
physicians, hospitals and/or the government to providing them with a PHR [6]. The
majority of the usefulness of a PHR hinges upon its interoperability with EHRs [7] and
thus, Canadians may be waiting to be offered a PHR that is integrated with the systems
their providers use. Deloitte’s [6] survey also revealed that majority of Canadians
would like to access a family member’s PHR and would appreciate online tools to help
them assess, monitor and manage their health (e.g., risk assessment questionnaires,
health diaries). In summary, despite low adoption rates, Canadians support the
functional capabilities of PHRs and would be receptive to a PHR selected for them by
their health provider or organization.
Given their potential to benefit patients with chronic illnesses, many
organizations are considering PHR procurement. The selection of a PHR to be used by
citizens in a large region within Canada is an important and critical task in attempting
to achieve the potential advantages that might result from widespread use of such
systems. In this paper we describe how conventional procurement methods can be
combined with plans for conducting usability analyses of candidate PHR systems in
order to increase the chance of successful deployment. It is argued that a combination
of methods will be needed to reduce the risk of system failure and the risk of limited
adoption of the PHR by the public once the selected system is deployed.
1. Methods
A chronic disease interest group was developing a paper-based PHR and was interested
in also offering their PHR to patients in an electronic format. The purpose of study was
to generate a set of criteria to assess commercial-off-the-shelf (COTS) PHR products as
potential options for converting the existing paper-based PHR into an electronic tool.
310 H. Monkman and A. Kushniruk / Considerations for Personal Health Record Procurement
1.1. Team Members
The principal investigator coordinated and chaired the meetings. Two subject matter
experts from different arthritis groups provided insight on the paper-based PHR,
clinical and patient requirements. During the development of the paper-based PHR
patients were questioned and consulted about their needs and preferences for an
electronic PHR; however, given that this study was only preliminary, patients were not
included on the team. Two other representatives were subject matter experts in
information technology and provided insight on functional and technical requirements.
1.2. Procedure
The group met on multiple occasions to discuss and come to consensus on a set of
criteria to compare commercial-off-the-shelf (COTS) PHRs with the paper-based PHR.
Additionally, criteria were developed to assess issues unique to electronic tools (e.g.,
privacy and security). Criteria ranged from user focused aspects of PHRs (e.g.,
functionality, usability) to criteria important from the perspective of the dispensing
agency (e.g., vendor stability, pricing).
In conjunction with the identification of criteria, questions were generated to
assess these criteria. Some questions were adopted from PHR evaluation questions
developed by the former California Regional Health Information Organization
(CalRHIO). However, many of these questions were not well suited to the Canadian
context so they were supplemented with questions created by the project team.
To improve the likelihood that an appropriate PHR selection was made, limitations
of conventional procurement processes were considered through discussion to arrive at
extensions and new approaches.
2. Results
In addition to the criteria that were developed (summarized in Table 1), other important
considerations for PHR procurement discussed by the group are outlined here.
2.1. Potential for Integrating Complementary Apps and Resources
Integration of specialized mobile apps is especially appealing because of their

desirability among younger demographics and their potential to facilitate monitoring.
For chronic illnesses where consistent monitoring plays an important role in
management mobile apps are ideal because they:
• Are designed simplistically which facilitates quick and easy data entry.
• Are built for devices that users frequently carry with them.
• Are capable of setting timed reminders to prompt users to enter data.
Existing health resources could be integrated into the PHR that have been designed
with health literacy considerations specifically for health promotion and education. For
example, for British Columbians, HealthLinkBC (www.healthlinkbc.ca) would be an
excellent resource to integrate with a PHR. HealthLinkBC provides trusted health
information on over 5,000 health topics, symptoms, medications, medical tests and tips
for maintaining a healthy lifestyle. The site can be also used to find local health
services. Nurses are available 24 hours a day by phone and dieticians and pharmacists
are available during specific times. By integrating existing resources such as
HealthLinkBC with a PHR, users could be presented with credible, personalized
information according to a multitude of variables (e.g., gender, weight, illnesses,
location). Not surprisingly, personalized GFWECVKQPCNKPHQTOCVKQPJCUDGGPUJQYPVQDG
RQRWNCTYKVJRCVKGPVU=?
Table 1. Summary of Criteria for PHR Procurement

Criteria Description
Functionality The range of personal health information stored and
the health management and health promotion tools
offered.
Usability The effectiveness, efficiency and satisfaction with
which the PHR can be used to achieve specified
goals in a particular context.
Communication Tools The ability to securely exchange messages and
information using the PHR.
Shared Access / Permissions The capacity for patients to share aspects of their
records with health care providers and family
members.
Health Literacy The ability for users to understand the content,
apply the information and use health resources.
Language Flexibility The availability of multiple languages and the
capacity of the PHR to translate between languages.
Accessibility & Portability The ability to protect users against loss of access to
the service and their records.
Mobile Accessibility The capacity for the PHR to be accessed on
computers, tablets and mobile phones and apps
developed specifically for mobile devices.
Interoperability & Standards The ability of the PHR to communicate with other
clinical information systems (e.g., EMRs, lab
systems, pharmacy data etc.).
Compatibility with Medical Devices The ability to connect with and upload data from
other devices (e.g., glucometers, pedometers,
scales).
Privacy & Security The ability to protect users from unauthorized
access or use of records.
Support The ability for users and the dispensing agency to
get help with how to use the PHR and problems
they encounter.
Development Feasibility The ability to create and/or augment tools.
Vendor Stability / Sustainability of The ability to archive data and keep a long-term
Data detailed record assured that vendor remain in
operation.
Data Accessibility The ability to access data and know where it is
located.
Scalability The ability to handle growing amounts of work and
the ability for the infrastructure to be expanded to
accommodate that growth.
Ability to Customize The ability to tailor the PHR based on specific user
& provider wants and needs.
Pricing Expenses for deployment, maintenance and user
subscriptions.
Usage Auditing The ability to log use pattern data.
312 H. Monkman and A. Kushniruk / Considerations for Personal Health Record Procurement
2.2. PHR Flexibility
It was imperative that any COTS PHR considered was generic enough to support the
self-management of any chronic illness, but could be tailored according to the specific
condition(s) of individual patients. This strategy would be most effective for
integrating the entirety of a patient’s health information as well as streamlining the data
entry process for users. That is, users would likely be deterred from using different
PHRs for their different conditions. Thus, a common foundation of tools for all
patients (healthy and ill) with complementary modules for specific illnesses would be
the ideal solution. This would bolster the appeal for a variety of chronic illness special
interest groups. Additionally, the core components could be used by all health
consumers not only those affected by chronic illness.
2.3. Procurement Commitment
Before committing to purchase a PHR, the dispensing agency was encouraged and
advised to negotiate a trial period (e.g., 6 months). This trial period allows for the PHR
to be evaluated with usability methods and tested for interoperability with other health
information systems (e.g., e-prescriptions, test results, electronic health records). This
would involve testing with a wide range of participants from the public. Given that
usability is a key factor in system adoption and interoperability plays a fundamental
role in the usefulness of the PHR, it is imperative that the selected PHR demonstrate its
capabilities in these areas before a purchase commitment has been made. This is
following a framework described by Kushniruk and colleagues for reducing risk of
system failure and adoption by carrying out usability testing of candidate systems prior
to final purchase of regional health IT systems [5].This trial period also provides an
opportunity for users to provide feedback on their likelihood of system adoption and
the advantages and disadvantages of the product itself.
3. Discussion
There are a variety of criteria to consider before purchasing a PHR solution.

Furthermore, these criteria may be prioritized differently depending from the
perspective taken (i.e., user vs. dispensing agency). It is also important to weigh the
advantages and disadvantages of selecting a COTS PHR versus a custom development
solution. Moreover, it is imperative that as many groups as possible collaborate to
deploy a common PHR foundation to improve the likelihood of adoption.
One limitation of this study is that no users were included team as it was still the
preliminary phase of the project. However, it is encouraged that patients are involved in
the evaluation of a shortlist of candidates. Furthermore, their opinions are perhaps
among the most important of all stakeholders, given that patients will ultimately decide
whether to adopt or reject the selected system.
Learning from the challenges associated with EHR adoption, ideally, the
government should dispense a single, standardized PHR. Although a national initiative
would benefit the maximum number of Canadians, a provincial strategy is more
realistic, given the healthcare infrastructure. This approach would likely result in a
variety of implementation, financial and user benefits. For example, the challenges
associated with interoperability would be mitigated because all supporting systems
would be required to interface with a single system. Given that the core functionality
was developed, the costs of creating specialized modules would be minimized.
Additionally, this strategy would ensure the dispensing agency had the most bargaining
power regardless of whether a COTS or custom developed PHR solution was selected.
A provincial PHR is also likely to gain favour from Canadians who are waiting for a
PHR endorsed and dispensed by the government. Earlier this year, the Province of
Alberta adopted this provincial PHR strategy and it will be interesting to watch the
progress and outcomes of this approach [10]. Until all of the provinces have adopted a
PHR solution approach, individual chronic disease groups are likely to continue
pursuing PHRs procurement on their own.
Although the demands of chronic illness care are likely to motivate the deployment
and adoption of PHRs, people without chronic illnesses could also benefit from their
use. Furthermore, PHRs becoming more common and users demanding digitized
information may be the driving force behind ubiquitous EHR adoption. However, in
order to achieve these benefits, it is imperative that new and improved procurement
methods are adopted to ensure the selected systems best match the needs of their users
and purchasers.
References
[1] Health Council of Canada. Helping patients help themselves: Are Canadians with chronic conditions
getting the support they need to manage their health? Canadian Health Care Matters, bulletin 2
[Internet], 2010. [cited 2010 Sep 3] Available from www.healthcouncilcanada.ca/tree/2.17-
AR1_HCC_Jan2010.pdf
[2] Solomon MR.. Information technology to support self-management in chronic care: A systematic
review. Dis Manage Health Outcomes, 2008;16(6): 391-401. doi: 10.2165/0115677-200816060-00004
[3] Holman H, Lorig K. Patient self-management: A Key to effectiveness and efficiency in care of chronic
disease. Public Health Reports 2004;119(3): 239-243. doi: 10.1016/j.phr.2004.04.002
[4] Jones DA, Shipman JP, Plaut DA, Selden CR. Characteristics of personal health records: findings of the
medical library association/national library of medicine joint electronic personal health record task
force. Journal of the Medical Library Association: JMLA 2010;98(3):243-249. doi: 10.3163/1536-
5050.98.3.013
[5] Shortliffe EH, Cimino JJ. Biomedical informatics computer applications in health care and
biomedicine. 3rd ed. Springer, New York, 2006.
[6] Deloitte Consulting, Deloitte Center for Health Solutions. 2009 Canadian health care consumer survey
report: Treating patients as consumers. Washington DC,[Internet] 2009. [cited 2012 Sep 3] Available
from www.deloitte.com/assets/Dcom-Canada/Local%20Assets/Documents/Public%20Sector/
ca_en_healthcare_consumersurvey_oct09.pdf
[7] Tang PC, Ash JS, Bates DW, Overhage JM, Sands DZ. Personal health records: definitions, benefits,
and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006;13(2):121-126. doi:
10.1197/jamia.M2025
[8] Jones R, Pearson J, McGregor S, et al. Randomised trial of personalised computer based information
for cancer patients. BrMed J 1999;319(7219):241-1247.
[9] Kushniruk A, Beuscart-Zephir MC, Grzes A, Borycki E, Watbled L, Kannry J. Increasing the safety of
healthcare information systems through improved procurement: Toward a framework for selection of
safe healthcare systems. Healthcare Quart 2010;13:53-58.
[10] Alberta awards PHR contract to TELUS-led group. Canadian Healthcare Technology. January 26, 9(2)
[Internet](2012). [cited 2012 Sep 5] Available from http://www.canhealth.com/News1878.html
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-314
}> ``

^ #

¢¥ }}\}!}}

¢

(
!

<!(
!

|<
<
<

<

<

(
(
:

<
^ *

* ` <
`

| *
! <
`
! `|
| *

} !
` * *

} !

Z}* **

* <
*

!|
`
} ! ¾

< #

!
*` *

`!}
** !

*
}

<
}
* } |
< ^
*
* *
`
¾ ¾ `

< *

* \|

| !

` `
< ^
!
|\|
!

} !
*
}
Z

<

* } ³* ³ ³

^ *

} * |
*
}!

*
*
<

*

! ! }

| ` *

} `
! ** * |

| < *
*!
` ! !
* * **
*
<

*
}<
}*

*
}!
|

| `
<

* } <#*
} "`

^
! }`}"}^
! }^³ ¿* <<
M. Clayton et al. / An Initial, Qualitative Investigation of Patient-Centered Education in Dentistry 315

} !
}*

}` *

! `
<|`}*
|`

! <
| *

`

!
<
"¦ } <¾ *
` |

< * ` } |
<¾
<
*

} <¾
*

!<
|

"`

*
¾ *

`

!` <*

`

}
\ *

| |}<
| !
*

<

* *
*
`
*
¾ `< ¾ <

\ ¾}

¾ `
¾ |

*
¾`
¾<^ |

*`

¾}

* ¾<

"}} { ¾ *
*
*

`

*

<
** `

*
*

| *¾}

*

¾| `
¾< ¨
} * *
*
|
! | ¾ ¾

<
!
* |\|
} | `

! `|
| *

}
!
` * *

<

|

`*
Z} * **

* <

\

* |

! | | | < ^

*
! }

}
|
*
*

`
< ^
`|

}|
* }
` *
<

} | !
` *
`
<
!
`
*
<<
}

} `
}

*
< !
`
| !
*

!
** *

<} } <

* * !
`

`

!
`
<
`|* *

`
* *` *

^
! "`
^
! # ! <
316 M. Clayton et al. / An Initial, Qualitative Investigation of Patient-Centered Education in Dentistry
`|
| *` *

}

} *
}
| * `|
| |
*
<# `|
``

*
<
!
`
}|!
`

}

*` *
*
` `

<

*
| Z ` !
`

* <
Z\ `

* `

! ` ``|<*
|

|

* `|}* À^* !
`
} * À^ ! "`

^
!
`|
<
/ #
* `|!
`
**
<
Q Q YZ !

#

^` #
`!

*

*

`*

^

`| *

#

#

^
*
^

*

}
!
` ^` #

}

} *
`

}

*

`

*

`|} | ` < |
* !
}
}
|
} !

< ! *`
``| *

` <

*
¾
³¾

` < | `
** !

| |
< *

* !

!#}\*}

}!
¤ <
| *

!
`! * !
< | *

` *

|<
**

`
*
} `

!
* **
< | *

*
}
\

}

< } *

}
}

<
^ `|
`
* ** *
< *
**
} !

**
} ¨ `
}

< ^
* * !

*
*
}* *

}

* *
<¤* !

*
|
M. Clayton et al. / An Initial, Qualitative Investigation of Patient-Centered Education in Dentistry 317
* !

<
!
*
|

<
|
**

} !
}

<
\ !
*

}
*

| *
*

*
!
}
}

<
!
`* **
* * }| |

` `

¾ ** } ` <

`

¾ ** } ` <

<

}|

*` ¾ *
*
¾<
/<^
`

`<

X"

/ Z
" "

#
\%

Z %
QX \ '

** ` `
! *
* <!
`
}
|
*
!
`<

}

`! *

*
}

| * **
}

*
|}
* *
*
<

^ * *
**

*
`
<
*
** `

< |`} ` |
*
\ *
¾

}

* |
! *

`
}

<
!
*
|

}|

|

}|

*

`*

<

}

`! *
}
}

*

**

|
|
*
*
< |`} *

|
*

| *
*
`
< } |

}
` !
318 M. Clayton et al. / An Initial, Qualitative Investigation of Patient-Centered Education in Dentistry
!

* `

<

|
<
} |
| `
*
*
}
!
^
! <

Z<}|
`

<|`}
` !
`
**|
\
|
!
!

<

*
| | *

`` | *
!

*

<*

* \|

|!
`
`
< } *
!
|\|
!

} !
*
}
Z

<
"
} {\ }

£< * * *

*
<%(
?
#³<
¤

#}|\
}! }^ }}¤Z}<*
* }* !

**

<-%
:
<£³
<
} ^ ` } ! #} ^ } < ^ ** |
`
* <*<!³<
| < ` *
*
`|< :-%<
³<

}
¤<

!

*

< %(
<<³<
£} ¥ #} ¤ } £\
< `!
*
\|
*
<%(
<¥`³<
}£
¦}\}¦*}} <#

*
` *
* `

< %(
<<³
<
} ¦}
£< ^` !
¥| * < (

!
< ³<

} ¦}
£}
< !*` |

<(
!
< ³<
}`
}\\¦<

`
`*` * <%(
<<*³<
¥Á ÁÂ } < # *

< (
%< # ³
<
£} } } Ã Ã} } Ä ¢} ¢} Ä< ^

|

|!!
` *
* `

< %:
<³<

¤}
¦< ^*
|
*
< (
%<
!³}

<
¥ } } `< #

|

|

*

*

%
(
<³<
\ } } ! } ¤ £} } ¦< * *

<%(
<£³<
`
\ } ¦
£< ** ` *

*
<|(
%<£³<
£
} | Z } ¤! £} ¦! <

\

Z <%(
<!³<
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-319
Tailored Care Management with Patient-

Centered Web-Based Portal in Primary
Health Care: Sustaining a Relational
Context
James P. RONAN a1
a
Abstract. While many innovative information and communication technologies

have been offered as solutions to primary care management challenges, few have
been shown to be effective and or sustainable over time. Information technology
approaches have been narrow in focus, relying on designs that enhance usability,
interoperability and adaptability by delimiting the traits, attributes, and
characteristics of individual communication processes. It is increasingly
understood in primary health care settings that relational communication
continuity between the patient and the health care team is essential for optimizing
co-determined treatments, interventions, and self-management strategies.
Successful utilization of a patient-centered web-based portal must account for
essential proximity of the relational aspect of care between the patient and the
immediate health care team.
Keywords. informatics in primary care, relational care, tailored health

communications, patient-centered web-based portal
Introduction
Health information is often conceptualized as some form of generic communication to

the patient (such as brochures or public service announcements) that is expected to
elicit a desired behavior change. This idea draws on advertisement practices, with the
assumption that the more exposure to the health message, the more likely the desired
outcomes will be elicited. Unfortunately, this model of patient education, while
indicative of most current primary health care practices, falls short of the state of the
science of communication and behavioral theory and research. Based on current
literature one can theorize that there are prerequisites to effective health
communication and optimal behavioral change: 1) establishing and maintaining a
relational context between the patient and the primary health care team; 2) tailoring
health communication and achievable care management strategies; and 3) enacting and
coordinating information and communication technologies as facilitative tools. The
purpose of this paper is to discuss relational contexts as an antecedent to effective
tailored health communications, and optimal patient-centered web-based portal
1
320 J.P. Ronan / Tailored Care Management with Patient-Centered WBP in Primary Health Care
sustainability. The premise of a relational context is central to optimizing an integrated

schema for effective health communications and enhanced health outcomes.
1. Relational Context
Relational engagement embraces an ethic of shared power through cooperative

interchanges by patients and members of the primary health care team that result in
creation of proactive strategies for the purpose of optimizing health well being [1-4].
From the stand point of the literature in the last decade, the objectified patient who
obediently follows the prescriptive directions of the physician within the
biopsychosocial context has all but disappeared. Unfortunately, most primary health
care practices have not made this transition, nor have the professional medical societies
that represent physicians. In contrast, relational practice represents the underlying
metapardigm of the discipline of nursing since Nightingale [1-2, 5-9]. Recently,
multiple disciplines have discovered the benefit of relational practice and are engaged
in collaborative, interdisciplinary research, policy, and clinical guideline development
that highlight this awareness [4, 10-11].
The World Health Organization reaffirmed a commitment to relational approaches
to care in their recent documents outlining future world health needs [12-14] by
focusing on the increasing burden of chronic illness and the need for partnerships in
care spanning self-management to the community. From the stand point of quality, a
recent publication by the Institute of Medicine (IOM) in the United States reaffirms and
recommits to the central focus of relational approaches to patients [10]. Central tenants
to the IOMs recommendation regarding relational care include the following: an
approach to patient-centered care that is respectful of and responsive to individual
patient preferences, needs, and values, and ensures that patient values guide all clinical
decisions; care based on continuous healing relationships; care customized according to
patient needs and values; the patient is the source of control; knowledge that is shared
and with pertinent information flowing freely; decision making informed by evidence;
patient needs that are anticipated; and transparency of the care process [10]. In Canada,
Haggerty et al., [11] reaffirmed the definition of primary health care with a team of
primary care experts as a central premise of “relational continuity, coordination-
continuity, family centeredness, advocacy, cultural sensitivity, clinical information
management, quality improvement process, interpersonal communication, community
orientation, comprehensiveness, multidisciplinary team, responsiveness, and
integration” [11p. 336]. These operational definitions are grounded in a relational context
with patients. Likewise, health literacy based on a relational context assumes a
dialogue for the purpose of integrating knowledge into mutually derived approaches for
potential treatments, interventions, or self-management [4, 9].
2. Tailored Health Communications
Tailored health communications are defined as “any combination of strategies and

information intended to reach one specific person, based on characteristics that are
unique to that person, related to the outcome of interest, and derived from individual
assessment” [15 p. 113]. Computerized web-based static or non-relational approaches to
tailored health communications have had little positive impact when applied in primary
J.P. Ronan / Tailored Care Management with Patient-Centered WBP in Primary Health Care 321
health care settings [16]. Additionally, static tailored health messaging had negative
outcomes when initiated in a worksite setting [17]. Tailored health communication
derived through interpersonal communications has the potential for high effectiveness
when coupled with relational approaches. In a recent meta-analysis, Wanyonyi,
Themessl-Huber, Humphris and Freeman reported that “a significant and positive
effective of face-to-face tailored health messaging upon participants behaviors” [18].
For primary health care practice, relational tailored health communications are
understood to be most appropriate when accounting for communication reception and
acceptance coupled with motivational behavioral change pathways [19-20].
3. Patient–Centered Web-Based Portal
Clinical information systems are those dynamic repositories of information that are
created, stored, collated, and protected for the purpose of creating historical records of
patient encounters. In the 21st Century these clinical information systems are evolving
from static paper-based structures to electronic databases where records can be securely
produced, archived, shared, evaluated, and coordinated via complex electronic input,
throughput and output technologies. These health informatics technologies have
created an opportunity for expanding capabilities for the delivery of primary health care.
There are four primary interconnected platforms that frame clinical information
systems at the primary health care level. Presented here by reducing level of magnitude,
they are Electronic Health Record (EHR), Electronic Medical Record (EMR), Personal
Health Record (PHR), and Web-Based Portals (WBP). The EHR is a computer-based
collection of health information that has been gathered by and is managed by an
enterprise, typically a physician or hospital. EMR is a computerized platform for
managing medical information collected by a hospital or physician practice and is
typically understood as less in scope and functionality than an EHR. EMRs are
transitioning into EHRs as sophistication requirements increase over time. PHRs are
person-centered systems designed to track and support health activities across one’s
entire life experience, not limited to a single organization or provider. WBPs are static
or dynamic web portals that house secure point-in-time health information for the
purpose of education, communication, assessment, or management related to health or
illness conditions. These WBPs are interconnected to EHRs, and PHRs simultaneously
or separately. Following is a description of WBP implementation for primary health
care practice.
Within a context of patient centered design, primary health care practices establish
scalable EHR and a secure clinic practice WBP for creating expedient communication
between health care providers and patients regarding tailored health/illness
management strategies. WBPs adjacent to EHRs are desirable because of their
flexibility in communication, scalability for PHRs, as well as fire-wall security and
integrity protection of EHR content by designing customized specific interfaces.
Not only can patients access the practice web-based portal and log-on to secure
modules for access to standardized screening tools, they may also free-text symptoms
or concerns such as medication issues or sub-optimal treatment experiences directly to
the health care team for immediate action. These WBP communications are routed to
the practice EHR and imported to cue for action in the form of a pop-up notice for the
case manager and or the provider of record. An assessment and action is required by a
member of the health care team in response to the patient inquiry. Immediacy and
safety is enhanced by designing the pop-up to remain on the EHR screen until action is
taken. A tailored action plan response by a member of the health care team is
forwarded back to the patient at the web-based portal address. A text/email is
automatically forwarded to the patient for notification of a pending action plan.
Additionally, the action plan is routed to the patient’s EHR permanent record. A notice
is forwarded to the initiator of the action plan as soon as it is opened by the patient in
the web-based portal. If the action plan includes clinical follow-up it is coordinated
through the web-based portal with the office practice via electronic routing. This design
allows for a range of complexity to be managed appropriately, enhancing clinical
outcome and quality. Accommodating disadvantaged patients by developing alternative
communication pathways is mandatory. For example, use of telephone, proxy care
respondents, such as parents, children, siblings, and designated care givers are
acceptable alternatives.
Health information technology has increasingly become more accessible to the
health care team and now is one of the most important interfaces for communicating
health information. Second only to interpersonal networks, the internet is the primary
source for health information for the health conscious consumer with print media,
television, and radio becoming more passive sources [21].
Targeted designs for web-based portals is extensive with deployment in areas of
provider-patient relationship [22], health coaching for primary care patients with
chronic conditions [23], as an adjunct to personal health records [24], screening for
chronic conditions [25], promoting patient-centered preventive care [26], patient
satisfaction and the quality of office visits [27], and disseminating of health reminders
[28].
In a systematic review of web-based portals to improve diabetes outcomes, [29] it
was reported that usability studies have found that many patients are open to the use of
technology in their disease management, regardless of age. However, greater attempts
to assist disadvantaged, older, or less computer-literate populations must be designed
into any web-based portal initiative. While disadvantaged populations have shown less
inclination to enroll in patient portals programs, once enrolled, participation was no
different than all other participants [30]. Ancker, Barrón, Rockoff, Hauser, Pichardo,
Szerencsy & Calman [31] reported that during the first two years of a patient web-
based portal deployment, adoption rates for low-income populations with chronic
disease experience only low rates of disparity-related participation. It is essential to
recognize that web-based portal technologies focused on wellness/illness cannot be
dismissed because of disparate or disadvantaged circumstances for any population. A
usability strategy that focuses on an emphasis on deploying solution to include all
segments of the population must be attained that reinforces and sustains the relational
context.
Currently, the majority of designs for patient web-based portals can be classified as
static, offering information, education, questionnaires, or providing contact conduits to
the health care team. A more dynamic method is describable and would include an
approach that is more dialogical, relationship-focused for the patient with the health
care team. Such a system may improve both acute and chronic care outcomes.
Report of a recent random control trial of a dynamic WBP for chronic asthma
control is an example of the use of a nurse case manager for patient interaction and
management [32]. However, on careful examination this model design contains too
much content with excessive demands on the patient to complete daily ‘check in’ for
metric reporting, education, status assessment, hence usability will diminish long term
J.P. Ronan / Tailored Care Management with Patient-Centered WBP in Primary Health Care 323
sustainability. This model positions the nurse case manager adjacent to the point of
care instead of being integrated within the primary health care team. Langstrup [33]
reported on an asthma monitoring web-based portal implementation that failed
integration and long term sustainability wherein technology did not become a durable
portion of the primary health care practice. What was cogent in the Langstrup [33]
experience was that the primary care providers were distant to the project (delegated
responsibility) instead of engaged as partners in care with their patients.
4. Conclusion
Based on review of what is known from existing models, the author proposes that
by critically analyzing and incorporating experiences from the literature regarding
relational contexts, tailored health communications coupled with patient-centered web-
based portals present an integrated schema that will enhance effective health
communications and optimized health outcomes in primary health care practices in the
21st century.
References
[1] Hess JD. Gadow's relational narrative: an elaboration. Nursing Philosophy. 2003 Jul;4(2):137-48.
[2] Gadow SG. Relational Narrative: The Postmodern Turn in Nursing Ethics. Scholarly inquiry for
nursing practice. 1999;13(1):57.
[3] Newman MA. Theory for nursing practice. Nurs Sci Q. 1994 Winter;7(4):153-7.
[4] Duggan A. Understanding interpersonal communication processes across health contexts: Advances in
the last decade and challenges for the next decade. J Health Commun. [Review]. 2006 Jan-
Feb;11(1):93-108.
[5] Newman MA, Sime AM, Corcoran-Perry SA. The focus of the discipline of nursing. Adv Nurs Sci.
1991 Sep;14(1):1-6.
[6] Newman MA. Newman's theory of health as praxis. Nursing Science Quarterly. 1990 Spring;3(1):37-41.
[7] Rogers ME. The science of unitary human beings: current perspectives. Nurs Sci Q. 1994
Spring;7(1):33-5.
[8] Hartrick GD, Colleen V. Relational Practice and Nursing Obligations. Adv Nurs Sci. 2007;30(3):192.
[9] Loescher LJ, Ronan JP. Toward a holistic view of genetic technology as a way of knowing. Holist Nurs
Pract. 1998 Apr;12(3):1-8.
[10] Institute of Medicine (U.S.). Committee on Quality of Health Care in America. Crossing the quality
chasm : a new health system for the 21st century. Washington, D.C.: National Academy Press; 2001.
[11] Haggerty J, Burge F, Levesque JF, Gass D, Pineault R, Beaulieu MD, et al. Operational definitions of
attributes of primary health care: Consensus among Canadian experts. Ann Fam Med. [Article]. 2007
Jul-Aug;5(4):336-44.
[12] Epping-Jordan JE, Pruitt SD, Bengoa R, Wagner EH. Improving the quality of health care for chronic
conditions. Quality Safety Health Care. 2004;13:299 - 305.
[13] World Health Organization. The World Health Report 2003: Shaping the future. Geneva: World Helath
Organization; 2003.
[14] World Health Organization. Current and future long term needs. Geneva: World Helath Organization;
2002.
[15] Noar SM, Grant Harrington N, Van Stee SK, Shemanski Aldrich R. Tailored Health Communication to
Change Lifestyle Behaviors. Am J Lifestyle Med. 2011 March/April 2011;5(2):112-22.
[16] Ruffin MT, Nease DE, Sen A, Pace WD, Wang C, Acheson LS, et al. Effect of Preventive Messages
Tailored to Family History on Health Behaviors: The Family Healthware Impact Trial. Ann Fam Med.
2011 January 1, 2011;9(1):3-11.
[17] McHugh J, Suggs LS. Online Tailored Weight Management in the Worksite: Does It Make a Difference
in Biennial Health Risk Assessment Data? J Health Commun. [doi: 10.1080/10810730.2011.626496].
2011 2012/03/01;17(3):278-93.
[18] Wanyonyi KL, Themessl-Huber M, Humphris G, Freeman R. A systematic review and meta-analysis of
face-to-face communication of tailored health messages: implications for practice. Patient educ couns.
2011;85(3):348-55.
[19] Rimer BK, Kreuter MW. Advancing Tailored Health Communication: A Persuasion and Message
Effects Perspective. Journal of Communication. 2006;56:S184-S201.
[20] Bandura A. Health promotion by social cognitive means. Health Education & Behavior.
2004;31(2):143-64.
[21] Dutta-Bergman MJ. Primary sources of health information: comparisons in the domain of health
attitudes, health cognitions, and health behaviors. Health Communications. 2004;16(3):273 - 88.
[22] Zickmund SL, Hess R, Bryce CL, McTigue K, Olshansky E, Fitzgerald K, et al. Interest in the use of
computerized patient portals: role of the provider-patient relationship. J Gen Intern Med.
2007;23(Supplement 1):20 - 6.
[23] Leveille SG, Huang A, Tsai SB, Allen M, Weingart SN, Iezzoni LI. Health coaching via an internet
portal for primary care patients with chronic conditions: a randomized controlled trial. Med Care.
2009;47(1):41 - 7.
[24] Clarke JL, Meiris DC. Electronic personal health records come of age. American Journal of Medical
Quality. 2006;Supplement to Volume 21(3):5S-15S.
[25] Leveille SG, Huang A, Tsai SB, Weingart SN, Iezzoni LI. Screening for chronic conditions using a
patient internet portal: recruitment for an internet-based primary care intervention. J Gen Intern Med.
2007;23(4):472 - 5.
[26] Chou AF, Nagykaldi Z, Aspy CB, Mold JW. Promoting patient-centered preventive care using a
wellness portal: preliminary findings. J Prim Care Community Health. 2010;I(2):88 - 92.
[27] Mooney BL, Boyle AM. Power to the patient; using portals can improve patients' satisfaction and your
practice's efficiency. Med Econ. 2011:S12 - S4.
[28] Koonce TY, Giuse DA, Beauregard JM, Giuse NB. Toward a more informed patient: bridging health
care information through an interactive comminication portal. Journal of the Medical Library
Association. 2007;95(1):77 - 81.
[29] Osborn CY, Mayberry LS, Mulvaney SA, Hess R. Patient web portals to improve diabetes outcomes: A
systematic review. Current Diabetes Report. 2010:422 - 35.
[30] Goel MS, Brown TL, Williams A, Hasnain-Wynia R, Thompson JA, Baker DW. Disparities in
enrollment and use of an electronic patient portal. J Gen Intern Med. 2011;26(10):1112 - 6.
[31] Ancker JS, Barron Y, Rockoff M, Hauser D, Pichardo M, Szerencsy A, et al. Use of an electronic
patient portal among disadvantaged populations. J Gen Intern Med. 2011;26(10):1117 - 23.
[32] Ahmed S, Bartlett SJ, Ernst P, Pare G, Kanter M, Perreault R, et al. Effect of a web-based chronic
disease management system on asthma control and health-related quality of life: protocol for a
randomized controlled trial. Trials. 2011;12(260):1 - 17.
[33] Langstrup H. Making connections through online asthma monitoring. Chronic Ilness. 2008;4:118 - 26.
Ontologies, Trust and Standards
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-327
The Social Act of Electronic Medication

Prescribing
Jos AARTSa,1
a
Institute of Health Policy and Management, Erasmus University Rotterdam,
Rotterdam, The Netherlands
Abstract. Prescribing medication is embedded in social norms and cultures. In

modern Western health care professionals and policy makers have attempted to
rationalize medicine by addressing cost-effectiveness of diagnostic and therapeutic
treatments and the development of guidelines and protocols based on the outcomes
of clinical studies. These notions of cost-effectiveness and evidence-based
medicine have also been embedded in technology such as electronic prescribing
systems. Such constraining systems may clash with the reality of clinical practice,
where formal boundaries of responsibility and authorization are often blurred.
Such systems may therefore even impede patient care. Medication is seen as the
essence of medical practice. Prescribing is a social act. In a hospital medications
may be aimed at treating a patient for a specific condition, in primary care the
professional often meets the patient with her or his social and cultural notions of a
health problem. The author argues that the design and implementation of
electronic prescribing systems should address the social and cultural context of
prescribing. Especially in primary care, where health problems are often ill defined
and evidence-based medicine guidelines do not always work as intended, studies
need to take into account the sociotechnical character of electronic prescribing
systems..
Keywords. Electronic prescribing; Medication; Medicine
1. Prescribing Medicines
Prescribing medicine is a social act [1]. Through prescriptions physicians show their
patients that they recognize their complaints and are trying to help them. Where
medication is seen as the essence of medical practice, prescribing is the main thing
expected from a physician. A non-prescribing physician is seen as a contradiction.
Prescribing medications represents two sides of the same coin. On the one hand it
shows the authority of the physician by being able to solve a problem of the patient, on
the other hand the patient demands an instantaneous solution of his complaint.
Prescribing a medicine comes to the rescue of offering an immediate therapy, even its
efficacy is doubtful. A prescription functions as a legitimation of the patient’s sickness.
In some cultures a patient never goes home without a prescription, in other both
physicians and patients feel restraints [2]. It may explain why medication consumption
varies wildly in different Western countries. However, the overall consumption of
medications has grown exponentially and has become an important cost factor of health
care [3]. Health authorities and administrators have sought to contain the cost growth
1
E-mail: [email protected]
328 J. Aarts / The Social Act of Electronic Medication Prescribing
and one attempt was to make physicians aware of the notion of rational prescribing.
Not only would doctors prescribe drugs that really would work for a disease, but also
they would be encouraged to look at what costs the effect was achieved and whether
prescriptions could be filled with cheaper medicines. A similar role was attributed to
pharmacists and they were given the right to alter prescriptions to substitute with
cheaper ones. Often it would mean that generic drugs would replace brand drugs.
Rational prescribing was seen as one of the outcomes of evidence based medicine.
Based on the results of clinical trials and meta-studies guidelines and protocols would
be established to guide physicians in making diagnoses, selecting the proper course of
action and assess treatment outcomes. It would be hoped that doctors would adopt
these insights and put them to use in practice. In hospitals the costs of physician
prescribing would be borne by the organization. The cost increase prompted one
hospital in the early 70s to design and implement an electronic prescribing system for
physicians and this new class of information systems became also known as
computerized physician order entry (CPOE) systems [4]. Much later focus shifted to
patient safety. In his seminal paper in 1994 Lucian Leape wrote that the error rate in the
practice of medicine was high and that more than half of them could be attributed to
medication errors [5]. Already in the late 90s studies reported a positive effect of CPOE
on the decrease of medication errors [6, 7]. The landmark Institute of Medicine studies
on human errors in medicine and quality of care advocated the use of electronic
prescribing systems in health care [8, 9]. The adoption however has shown to be a very
slow process. Aarts and Koppel reported nine years later that in seven Western
countries they studied the adoption rate was nowhere higher than 20% of the
hospitals[10]. Another study reported that ten years after the publication of the IOM
reports efforts to reduce errors in medicine had limited success [11]. There is clearly a
gap between the expectations of professional and organizations adopting scientific
evidence and technology to improve practice and reality.
2. Electronic Prescribing
Electronic prescribing is a social act. Though electronic information systems are often
seen as “neutral” tools to help a job done, they represent in fact social norms embedded
in technology. The original Eclipsys system allowed physicians only to pick medicines
from an approved hospital formulary. The hospital administrators expected that
physicians would only select from this formulary and thus help save medication costs.
The hospital did indeed report savings and improved efficiency of nursing
documentation [12]. But embedded norms do form a clash with reality. Another norm
is the authority to prescribe, which rests with physicians. Only physicians are
authorized to enter medication orders. In many hospitals it was (and perhaps still is)
practice that nurses in night and weekend shifts could order pain medications to relieve
their patients, because of limited availability of physicians. For that purpose nursing
stations would have a pile of pre-signed prescription notes that nurses could fill out.
Physicians trusted this work practice because they knew that the nurses were
knowledgeable and experienced. The advent of CPOE made this way working
impossible and put the burden of entering orders completely on physicians. In one
instance it caused physicians to protest the increased workload [13]. In other instances,
users created workarounds to make their work doable. In a study of the implementation
of CPOE in a Dutch hospital Goorman and Berg found how the problem of only
J. Aarts / The Social Act of Electronic Medication Prescribing 329
physicians being responsible for medication orders was circumvented in case of

emergencies by an ‘agent for’ device. A nurse could on behalf of a physician enter
medication orders, which would be signed off later by the physician [14]. Embedded in
CPOE are also guidelines and protocols in the form of decision support. Guidelines and
protocols represent proper practice of medicine. A medication order may prompt the
physician to look for specific patient data, and not allow completion of an order unless
he had acknowledged at least of having seen them. In a number of specialties it might
be seen as an annoyance, delaying precious time in patient care, because using specific
pieces of patient information might be routine. This is also the case in situations where
physicians may receive reminders about drug interactions, when they intend to
prescribe additional drugs. Reminders can prevent medication errors, but if appropriate
reminders are drowning in a sea of less useful reminders, then the positive effect can be
mitigated. The number of ignored reminders runs up to over 90% [15]. Similarly it has
been proven very difficult to reduce the number of inappropriate reminders. In the first
place doctors among themselves do not agree on which can be turned off [16]. They
report that they themselves might know what to do, but that doctors from other
specialties or residents would surely need them. In the second place, the technology is
still far from perfect. Using medication cases as gold standard, van der Sijs and her
colleagues found that CPOE systems would respond differently requiring additional
pharmacy review [17]. Implementing CPOE can have unintended consequences as well.
During rounding, medication orders are verbally communicated by physicians and
sometimes corrected by nurses, who would know about the exact health status of a
patient. It was an effective way for nurses to know when administering drugs should be
started and a perfect safety net to prevent errors. A CPOE system forces a doctor to
look for a computer, often sitting in a separate office, and the doctor would lack
interaction with colleagues when entering the order. Koppel and his colleagues
identified a number of situations that CPOE would potentially induce new errors [18].
It is clear that electronic prescribing systems are being designed and implemented with
embedded intent and purpose. Often, they rather shift workarounds instead of removing
them completely. Lacking the interaction with colleagues when entering an order, new
workarounds may arise to compensate it, like writing down the order on a piece of
paper during rounding and delay entering orders until after rounding. Because of this
intertwinement with organizational context and culture, they are in essence
sociotechnical systems [19].
3. The Future
In September 2010 the director-general of the Dutch health inspectorate announced that
electronic prescribing would become mandatory on January 1, 2012. Most likely, it will
be less of a problem in hospitals. Most hospitals in the Netherlands are currently
implementing electronic prescribing technologies. The shortage of physicians have led
to the introduction of nurse practitioners and physician assistants, who have received
prescribing authority, supervised by physicians. In a way the problem of the informal
work practice of nurses filling out orders has been removed by reconsidering
prescribing authority [20]. Clinical wards are in a way a kind of a micro-cosmos, where
professionals influence each other’s behaviors. The wide-scale implementation of
electronic prescribing will be much more difficult in primary care. More often, patients
visit a primary care doctor with vague complaints. In such situations emphasis on
330 J. Aarts / The Social Act of Electronic Medication Prescribing
evidence-based medicine in the form of guidelines and protocols is problematic and

physicians prescribe medications as a magic wand to address the needs of their patients.
A point in case is antibiotics, one of the most successful drugs in medicine.
Indiscriminate use to combat infections has caused the emergence of resistant
organisms compromising their efficacy. Prescribing antibiotics is nowadays based on
practice guidelines and carefully monitored because of increased insensitivity to
infectious microorganisms. Yet, even in hospitals non-medical reasons still influence
antibiotics prescribing [21]. One need not to be farsighted to see that the problem is
much larger in primary care practice, where a physician sees such a diverse patient
population of different social and cultural backgrounds. A study of prescribing
antibiotics for sore throats in primary care reported that physicians were well aware of
the marginal effects but yet often prescribed for good relationships with patients [22].
Most studies of electronic prescribing have been done in a hospital context. Primary
care is still largely uncharted territory. Future studies are needed to understand the
sociotechnical and cultural character of primary care prescribing using electronic
prescribing systems. In a recent paper I described a number of conditions that need to
be addressed [23]. They include interoperability allowing electronic prescribing
systems to interact with other systems so that patient medication information is more
readily available, improving decision support technology, and focus on the continuity
of care, in which professionals, organizations and systems are better aligned. But
foremost future research needs to focus on the question how the practice and social
nature of health care, evidence-based medicine and technology could be better
integrated. Bosk et al report in a commentary that giving a ‘simple checklist’ to
professionals as a solution to improve patient safety is based on the mistaken
assumption that a technical solution can solve a sociocultural problem [24]. Obliging
exchange of patient medication information between primary, secondary and tertiary
care is makes sense, but I am not in favor of mandating electronic prescribing so soon.
There are a lot of issues in the practice of health care that need to be resolved before it
can become meaningful.
References
[1] van der Geest S, Whyte SR, Hardon A. The anthropology of pharmaceuticals: a biographical approach.
Annu Rev Anthropol. 1996;25:153-78.
[2] Payer L. Medicine & culture: varieties of treatment in the United States, England, West Germany, and
France. 1st ed. New York: Henry Holt; 1988.
[3] Lied TR, Gonzalez J, Taparanskas W, Shukla T. Trends and current drug utilization patterns of Medicaid
beneficiaries. Health Care Financ Rev. 2006 Spring;27(3):123-32.
[4] Hodge MH. History of the TDS medical information system. In: Blum BI, Duncan K, editors. A history
of medical informatics. Reading (MA): Addison-Wesley Publishing Company; 1990. p. 328-44.
[5] Leape LL. Error in medicine. JAMA. 1994 Dec 21;272(23):1851-7.
[6] Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of "corollary orders" to
prevent errors of omission. J Am Med Inform Assoc. 1997 Sep-Oct;4(5):364-75.
[7] Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, et al. Effect of computerized
physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998
Oct 21;280(15):1311-6.
[8] Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human, building a safer health system.
Washington, D.C.: National Academy Press; 2000.
[9] Committee on Quality of Health Care in America. Crossing the quality chasm, a new health system for
the 21st century. Washington, D.C.: National Academy Press; 2001.
[10] Aarts J, Koppel R. Implementation of computerized physician order entry in seven countries. Health Aff
(Millwood). 2009 Mar-Apr;28(2):404-14.
J. Aarts / The Social Act of Electronic Medication Prescribing 331
[11] Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates
of patient harm resulting from medical care. N Engl J Med. 2010 Nov 25;363(22):2124-34.
[12] Watson RJ. A large-scale professionally oriented medical information system - five years later. J Med
Syst. 1977;1(1):3-21.
[13] Massaro TA. Introducing physician order entry at a major academic medical center: I. Impact on
organizational culture and behavior. Acad Med. 1993 Jan;68(1):20-5.
[14] Goorman E, Berg M. Modelling nursing activities: electronic patient records and their discontents. Nurs
Inq. 2000 Mar;7(1):3-9.
[15] van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician
order entry. J Am Med Inform Assoc. 2006 Mar-Apr;13(2):138-47.
[16] van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A. Turning off frequently overridden drug alerts:
limited opportunities for doing it safely. J Am Med Inform Assoc. 2008 Jul-Aug;15(4):439-48.
[17] van der Sijs H, Bouamar R, van Gelder T, Aarts J, Berg M, Vulto A. Functionality test for drug safety
alerting in computerized physician order entry systems. Int J Med Inform. 2010 Apr;79(4):243-51.
[18] Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, et al. Role of computerized
physician order entry systems in facilitating medication errors. JAMA. 2005 Mar 9;293(10):1197-203.
[19] Berg M, Aarts J, Van Der Lei J. ICT in health care: sociotechnical approaches. Methods Inf Med.
2003;42(4):297-301.
[20] Kroezen M, van Dijk L, Groenewegen PP, Francke AL. Nurse prescribing of medicines in Western
European and Anglo-Saxon countries: a systematic review of the literature. BMC Health Serv Res.
2011;11:127.
[21] Hulscher ME, Grol RP, van der Meer JW. Antibiotic prescribing in hospitals: a social and behavioural
scientific approach. Lancet Infect Dis. 2010 Mar;10(3):167-75.
[22] Butler CC, Rollnick S, Pill R, Maggs-Rapport F, Stott N. Understanding the culture of prescribing:
qualitative study of general practitioners' and patients' perceptions of antibiotics for sore throats. BMJ.
1998 Sep 5;317(7159):637-42.
[23] Aarts J. The future of electronic prescribing. Stud Health Technol Inform. 2011;166:13-7.
[24] Bosk CL, Dixon-Woods M, Goeschel CA, Pronovost PJ. Reality check for checklists. Lancet. 2009 Aug
8;374(9688):444-5.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-332
Improving Decision Quality in Healthcare

with an Error Prevention Model (EPM)
David T.S. CHIOa,1, Andre KUSHNIRUKa, Elizabeth BORYCKIa
a
Abstract. Human factors involved in decision quality are critical issues in

healthcare. In this paper, issues related to the impact of human factors on decision
quality in healthcare are considered. Specifically, the focus is on the issue of
reducing human error as well as improving decision quality. An Error Prevention
Model (EPM) is presented for considering tools and techniques that can be used to
analyze complex errors that may be considered latent.
Keywords: Technology-induced Error, Decision Quality, STDC, EPM Model
Introduction
According to an IOM report, somewhere between 44,000 and 98,000 people die in
different American hospitals every year because of “errors committed by medical
professional” [3]. The errors caused by technology-induced errors (i.e. errors that result
from use of health information technology) are related to human factors can come in
many forms, such as slips and lapses [2]. Although some active medical errors may be
detected, current approaches may not contribute to the problem of dealing with latent
errors. Even though work has been done in preventing technology-induced error in
healthcare, latent errors are still difficult to prevent due to the complexity of healthcare
processes and work activities. Some errors may arise from conscious, preconscious,
and unconscious human factors. For the purpose of analysis, errors are conceptualized
as an iceberg consisting of both active and latent errors. Active errors are associated
with conscious factors, while the latent errors include preconscious and unconscious
factors (see Figure 1).
Figure 1. A conceptual diagram of decision errors as an iceberg
1
Corresponding author: David T. S. Chio, School of Health Information Science, University of Victoria
Phone: +250 721 8575 E-mail: [email protected]
D.T.S. Chio et al. / Improving Decision Quality in Healthcare with an Error Prevention Model 333
Decision processes can be conceptualized as selection of a right choice from a

number of available choices through the iterative STDC (See-Think-Do-Check)
decision activities of seeing, thinking, doing, and checking. Each choice represents a
decision alternative. In reality, what people see is far more than what people
understand; what lies beyond is far more than what people see. In other words, there
are a number of barriers in achieving the goal of decision quality in the process of
decision making (See figure 2).
Barrier 1: how to identify what we see (i.e., from seeing to thinking)
Barrier 2: how to transform what we think (i.e., from thinking to doing)
Barrier 3: how to evaluate what we do (i.e., from doing to checking)
Barrier 4: how to measure what we check (i.e., from checking to seeing)
Despite the advantages of Clinical Decision Support System (CDSS) in healthcare,
there are a number of clinical challenges for CDSS. One of the most critical issues is
lack of effective and efficient ways to analyze and predict latent errors caused from
human factors. In our work, the roles of CDSS are considered in reducing medical
errors involved in decision-making processes. Furthermore, we propose an Error
Prevention Model (EPM) to provide potential solutions to avoid preventable adverse
events. In next section, some papers to related human errors and decision making are
reviewed for their potential to lead to evidence-base evaluation and outcome
improvement.
1. Related Work
Provision of safe patient care is a major challenge confronting today’s health care
system [1]. Well-designed patient safety initiatives based on systematic interventions
may produce the best results in enhancing the quality of health care processes [5]. In
addition, Berner and Lande [1] summarize current data on the use and impact of
clinical decision support systems.
In order to bridge the gap between process and decision modeling, a formalized
linkage model has been proposed to facilitate the integration of strategic, decision and
process objectives within a single framework [7]. Furthermore, a number of
frameworks and models for considering technology-induced error have been compared
to show how they are practically employed in the prediction and prevention of such
error [2] (see Table 1).
Figure 2. STDC (See-Think-Do-Check) decision process

334 D.T.S. Chio et al. / Improving Decision Quality in Healthcare with an Error Prevention Model
Table 1. Category and type of error in healthcare (Extracted from [2])

Error inducing category Error source type
Government Organization policy, legislation
Model Healthcare Organization model policies and procedures, model workflows, model
terminologies
Vendor Organization requirements gathering, design, programming, software testing
Local Healthcare Organization devices, workflow, interface design, local terminologies,
policies and procedures, local testing
Individual training, support
Some evaluation models used in checking human errors are considered. A

modeling process that helps decision makers in overcoming these perceptual and
cognitive challenges was proposed to improve the decision effectiveness [6]. Elwyn et
al. [4] measure outcomes with aggregate ratings for each criterion calculated using
medians weighted to compensate for different numbers in stakeholder groups. Criteria
were given the highest ratings where evidence existed. In contrast, Scott, Bellala, and
Willett measure the rates of convergence of generalization error that can be measured
with an upper bound theory quantifying the generalization error of various large margin
classifiers [8].
Upon review of the above literatures on human errors in healthcare decision
quality, the limitations of human beings are seen as a major factor contributing to this
type of error in medical decision processes. Even though much work has been done in
preventing decision error in healthcare, latent errors are still difficult to prevent due to
the complexity of healthcare processes.
2. Error Prevention Model (EPM)
In this section, an Error Prevention Model (EPM) is proposed to provide potential

solutions to avoid preventable adverse events to reduce human errors in the processes
of decision making.
There is a need to extend error evaluation frameworks by using V-model to ensure
their effectiveness, efficiency and safety, then the standardized methods or guidelines
could be developed based on best practices from a health information industry
perspective [2]. By extending the framework for diagnosing technology-induced errors
in healthcare from Borycki et al [2], a process is proposed to reduce errors and to
improve decision quality as well in the current work (see Figure 3).
Figure 3. An extension framework to improve the decision quality by reducing the errors
D.T.S. Chio et al. / Improving Decision Quality in Healthcare with an Error Prevention Model 335
Figure 4. Error Prevention Model (EPM)
As stated in the section of introduction, decision processes can be conceptualized

as selecting a right choice from a number of available choices through the iterative
STDC (See-Think-Do-Check) decision activities. In the above framework, the four
barriers in the STDC process can be conceptualized and translated into the EPM by
combining them with four error preventive engines, i.e., recognition, investigating,
analytical, and improvement engines to which we can incorporate a number of efficient
and effective mathematical analytical tools (see Figure 4).
1. Recognition Engine to overcome the barrier 1 in STDC
2. Investigating Engine to overcome the barrier 2 in STDC
3. Analytical Engine to overcome the barrier 3 in STDC
4. Improvement Engine to overcome the barrier4 in STDC
3. EPM Applications: Improving Decision Quality in Healthcare
In complex systems, when one component of a system which serves multiple functions
fails, all of the dependent functions may fail as well. As the dependency on technology
in complex systems increases and systems become more tightly coupled in time and
sequence, so does the likelihood of accidents [3]. By using this EPM model, error
prevention can be integrated in one systematic approach to deal with the human errors
when technology changes human tasks, shifts workloads, and tends to reduce or
eliminate human decision making.
This model provides guidelines for the development of cognitive and systematic
interventions to decrease medical errors to not only identify errors, but also transform
problems into a warning system. In table 2, some available methods, tools and
techniques (T&T) can be embedded within this model for some applications.
Discussion
In this paper, frameworks for considering error prevention were considered. Some
papers related to human errors and decision making were reviewed for their relevance
to error prevention and decision outcome improvement. Upon review of literature on
human errors in healthcare decision quality, the limitations of human beings are seen as
336 D.T.S. Chio et al. / Improving Decision Quality in Healthcare with an Error Prevention Model
a major factor contributing to this type of error in medical decision processes. Even
though considerable work has been done in preventing decision error in healthcare,
latent errors are still difficult to prevent due to the complexity of healthcare systems.
An Error Prevention Model (EPM) was proposed to provide potential solutions to
avoid preventable adverse events for CDSS to reduce human errors in the process of
decision making. Error lifecycle modeling could assist a healthcare authority to achieve
strategic objectives by providing methodologies and tools to develop integrated
healthcare process models, however further work is needed on how to identify the
methods and analytic tools to fit the specific error pattern.
Such a model has potential application to have a hybrid decision support
framework and EPM model built on best practice in healthcare. Validating this model
empirically will be next step, and this will involve performing systematic experimental
studies. Further study is needed to develop and test error mechanisms and the error
prevention engines proposed in this paper.
Table 2. EPM (Error Prevention Model) and its interpretation

Phase Name Purpose Method Tools and Techniques (T&T)
I Recognition Pattern Identifying Delphi Techniques, Linear
Engine Recognition Method Programming, Goal Programming,
Network Models, Decision Analysis,
Graph Theory, Statistics Tools
II Investigating Meaningful Transforming AHP (Analytic Hierarchy Process)
Engine Transformation Method Technique, Forecasting and
Prediction, Time Series, Moving
Average, Monte Carlo Simulation
III Analytical Risk Evaluating Fuzzy theory, Sensitivity Analysis,
Engine Assessment Method Network Models, Decision Analysis,
Queuing Analysis, Markov Analysis,
Statistical Analysis
IV Improvement Performance Measuring Systems Dynamics and Simulation,
Engine Measurement Method Decision Tree, Bayesian Decision,
Multi-Criteria Decision Making
References
[1] Berner ES, La Lande TJ. Overview of Clinical Decision Support Systems. New York, NY: Springer New
York; 2007:3-22.
[2] Borycki EM, Kushniruk AW, Brender J. Theories, models and frameworks for diagnosing technology-
induced error. Studies in health technology and informatics. 2010; 160:714-718.
[3] Kopec D, Kabir MH, Reinharth D, Rothschild O, Castiglione JA. Human errors in medical practice:
systematic classification and reduction with automated information systems. Journal of medical systems.
2003:297-313.
[4] Elwyn G, O'Connor A, Stacey D, Volk R, Edwards A, Coulter A, et al. Developing a quality criteria
framework for patient decision aids: online international Delphi consensus process. BMJ: British
Medical Journal. 2006; 333(7565):417-419.
[5] Kalra J. Medical errors: overcoming the challenges. Clinical biochemistry. 2004; 37(12):1063-1071.
[6] Marquard JL, Robinson SM. Reducing Perceptual and Cognitive Challenges in Making Decisions with
Models . New York, NY: Springer New York; 2008:33-55.
[7] Neiger D, Churilov L, Flitman A. Business Objectives Modelling. Boston, MA: Springer US; 2009; 19:1-
26.
[8] Scott C, Bellala G, Willett R. Generalization Error Analysis for FDR Controlled Classification, In:
Proceedings of the 2007 IEEE/SP 14th Workshop on Statistical Signal Processing; 2007:792-796.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-337
Software as a Medical Device: Regulatory

Critical Issues
1
Sylvia PELAYO a, , Sabrina BRAS DA COSTA a,b, Nicolas LEROYa, Séverine
LOISEAUa and Marie-Catherine BEUSCART-ZEPHIR a
a
Université Lille Nord de France, CIC IT Lille-EVALAB, INSERM, EA2694, UDSL,
CHU Lille, F-59000, Lille, France
b
InterPsy-ETIC (EA 4432), Université Paul Verlaine - Metz, BP 30309Ile du Saulcy
57006, Metz cedex 1, France
Abstract. The revised Medical Device Directive has been adopted by the EU in
2010. A major change is that software for certain purposes is now considered a
medical device. This entails that a new view needs to be developed on the design,
development, evaluation and post-market surveillance of medical software that
meets the definition of a medical device. This paper identifies some issues at stake
and discusses them.
Keywords. medical device, patient safety, risk management, software, human

engineering
Introduction
In 2010 the EU adopted a revised Medical Device Directive (MDD) [1]. A rather small
change in the text – the insertion of the word “software” in the definition of a medical
device – may have significant consequences for the health IT community. Now some
standalone software are considered Medical Device (MD): a MD is “any instrument,
apparatus, appliance, software, material or other article, whether used alone or in
combination, including software intended by its manufacturer to be used specifically
for diagnostic and/or therapeutic purposes and necessary for its proper application”.
Such software are subject to the same regulation as MDs: to introduce them on the
market, it is necessary to demonstrate their safety and efficacy. The manufacturer must
carry out a conformity assessment, set up a technical file and sign a European
declaration of conformity. The MDD explicitly mentions the role of evaluation
(technical, clinical and usability evaluations) for proving that the device is safe and
meets the stated objectives.
Since its adoption, the new definition raised many questions and has been
discussed in various contexts. The aim of this paper is to explore some consequences of
the revised MDD on design, development, evaluation and post-deployment of medical
software and to discuss the challenges for the health IT community. A brief case study
1
Sylvia Pelayo – CIC IT Lille-EVALAB, CHRU de Lille, Bâtiment Hippocrate, 2 Avenue Oscar
Lambret, Lille Cedex, F-59037 FRANCE – Tel: 0033 3 20 44 48 83 – [email protected]
338 S. Pelayo et al. / Software as a Medical Device: Regulatory Critical Issues
will be presented to illustrate the purpose.
1. Difficulty to Qualify Standalone Software as MD
Most of the discussions about the MDD focused on which software meets the
definition for a MD and which software does not. According to IEC 62304 [2], MD
software is “a software system that has been developed for the purpose of being
incorporated into the medical device being developed or that is intended for use as a
medical device in its own right”. Parallel to this definition, the manufacturer is anyway
the one determining if his software is or not a MD depending on the specification of its
intended use.
But, the further specification that “software for general purposes when used in a
healthcare setting is not a medical device” has introduced a certain level of ambiguity
on the interpretation of software to be classified as MD, especially when standalone
and/or software modules are used for the transmission and storage of patient data for
medical purposes [3]. For illustration, Electronic Health Records (EHR) are considered
an electronic view of the paper file that stores data and documents on a certain patient.
Given that some modules of EHR systems are the main sources for clinicians to make
diagnosis and establish patient treatment, they could be considered as MD software.
They should also follow the rules applied to MDs and have to be evaluated considering
the risk posed to patients and users. However, the European (and Canadian) Directive
tends to exclude EHR from the definition of MD software. Conversely, the US
regulation diffuses a detailed list of MDs and seems to suggest that EHR are MD
software. At first glance, the guidance document to qualify software as MD [4]
provides clarity, but with a closer look it can become a battlefield of interpretations. In
any case, in the end, the manufacturer is the one choosing. It is left to him to argue
whether his software is a MD or not. It is also left to him to draw the boundaries and to
argue that some modules are no MDs and others are.
2. Fear From Manufacturers that Regulation Stifle Innovation
The major requirement of the MDD is that MDs should be designed and manufactured
so that they will not compromise the safety of patients and users of the device. This can
be considered the first challenge for the developers of medical software. The MDD
states that a device should be designed so that it adheres to patient safety principles and
that state of the art methods are used. Currently, such a systematic patient safety based
design is not common practice and state of the art methods are not articulated. The
question arises whether the ISO standards should be considered as state of the art or
whether additional research based recommendations are needed [5]. Some initiatives tend to
provide this kind of supports such as the Standards Australia who published a handbook
with requirements for the presentation of health data [6]. The Microsoft and the National
Health Service have also developed guidelines for a common user interface for electronic
patient record systems [7]. But these initiatives are perceived by manufacturers as
“killers” of innovation because they would lead to the standardization of the design.
Moreover, manufacturers are supposed to publish sufficient information for an
objective analysis about the trade-offs between safety benefits and harms of their MD.
But manufacturers consider as confidential the knowledge about the design of their
S. Pelayo et al. / Software as a Medical Device: Regulatory Critical Issues 339
MD, especially drawbacks and glitches. Not sharing the software problems is an
industry implicit norm to keep competitors at bay. Manufacturers feel crashed with this
“aggressive government regulations of health IT” [8] and constantly claim that
regulations will cause an exodus of IT manufacturers from the field. In honesty, one
can acknowledge that there is a high number of regulation requirements usually
associated with onerous conformity assessment procedures.
3. Usability Requirement: Methodological Challenges
There is also the important question of whether software regulation is even feasible.
Apart from evaluation, the MDD mentions explicitly the ergonomic factors that should be
considered and integrated in the design and development cycle following a standard user-
centred approach. The aim is to secure patient safety (as well as the safety of the users of
the device) by preventing usage errors. But such approaches are still not universally
applied in the development of medical software and it is an arduous challenge to
overcome.
First of all, many methodological issues are raised by the required conformity
assessment procedure. On the one hand, manufacturers should use published data on
other devices that are sufficiently similar to the device at stake. But it is very difficult
to see this applied to medical software. Let’s take the example of Computerized
Physician Order Entry (CPOE). Are different CPOE sufficiently alike that studies of
benefits and potential harm can serve as proof that a particular CPOE is safe? On the
other hand, software implementation is massively complex. It may imply many users
with different goals and may lead to resistance, work processes reorganizations and/or
frustration. It is difficult to predict all the scenarios in which software failures could
result in patient harm and liability [9]. Finally, the medical specificities often request
specific configurations and customizations. Medical rules and order sets, for instance,
are almost always locally developed and extensively modified. These adaptations may
improve the fit between software and the work processes and enhance patient safety.
But, even an “innocent” modification may have negative repercussions on the use of
the software. How to track and control all these post-hoc changes? Does one need to
prove that the order sets are safe? And what if changes are made over time?
To conclude on usability methodological issues, the standard proving conformity
with the ergonomic Essential Requirement [10] raises various problems and needs to be
improved in its current version. It is supposed to guide manufacturers for the usability
engineering process but it is almost impossible to understand and apply for non-
usability experts. The major risk is that manufacturers interpret the standard in a wrong
way. The usability process is a subpart of the risk management process (risk analysis
focused on errors of use) that manufacturers often perfectly handle. They are also
persuaded to have all the necessary information to document the usability file. But they
often identify the risks related to technical problems of the MD and not the risks of
errors of use.
4. A Brief CaseSstudy: A Mobile Application for Diabetics
A Small & Medium-sized Enterprise (SME) has designed an innovative mobile

application for diabetic patients with insulin therapy. A Smartphone app helps patients
to manage their insulin treatment determining the right dose of insulin by analyzing
ongoing treatments, glycemic index goals, previous and current days’ results, and
personal variables. A telecare platform enables physicians and nurses in the
telemonitoring to support patients using physician-approved protocols. Usability
specialists have accompanied the company in the validation phase of the usability
process (i.e. evaluation of the achievement of the safety criteria for usability) and the
documentation of the usability file, subpart of the technical file for CE marking. The
usability study in still in progress, only partial results are presented in the paper.
4.1. Global Approach of the Manufacturer
The demonstration of the safety and efficacy of their product was a lengthy and
onerous process for the company. But, the revised regulation brought the company a
potential big benefit. CE marking allows the introduction on the market. The further
step with additional assessments may allow their software to be registered in the list of
products and services qualifying for reimbursement (LPPR) to be reimbursed by the
National Health Insurance. The inclusion would give a competitive advantage
compared to the others editors of this kind of software. Aware of this, the company had
approached a large pharmaceutical manufacturer and a research team on diabetes to
create a consortium enabling to face all the challenges to attain CE marking, and the
reimbursement of their software MD.
First, a clinical study was conducted to show that the tool significantly improves
metabolic control and effectively helps patients manage their care and maintain their
desired lifestyle. Significant improvement in glycated hemoglobin levels was achieved
in six months using the app. A second clinical study was planned to confirm these
results and measure the medico-economic impact of the app on large-scale care
management. The usability intervention was decided by the in-house person in charge
of the regulatory affairs, because he realized that the validation phase for usability
needed to be performed by usability specialists.
4.2. The Definition of the Medical Software
As it suited the manufacturer that his app was a MD, the MEDDEV guiding the
qualification of standalone software [4] was interpreted in this way: the app (i) was a
computer program, (ii) performing actions on data different form storage, archival,
lossless compression, communication or simple search, (iii) its actions were for the
benefit of individual patients, and (iv) its actions were for monitoring and treatment.
The precise MD boundary raised different points of view. For the manufacturer, the
MD was only the standalone software enabling the patient to manage his insulin
treatment in his Smartphone. As the interface between the app and the telecare platform
was not yet completely finalized, the manufacturer preferred that the usability
evaluation focused on the app only. But for usability specialists, it was not acceptable
in terms of patient safety: the app cannot be used by the patient safely without the
telecare platform used by professionals to manage the monitoring. Professionals were
mandatory to help patients in their decisions to adapt or not their treatment regarding
the app suggestions. The software MD was also defined as the app integrated in the
Smartphone and linked to the telecare platform to support interactions between
professionals and patients.
S. Pelayo et al. / Software as a Medical Device: Regulatory Critical Issues 341
4.3. The Usability Engineering Process
A standard user-centred approach was performed: some interviews with patients and
professionals, a literature review of incidents related to errors of use reported with this type
of apps, a usability inspection and usability tests. The results show that while the
manufacturer was convinced he had identified and documented all risks related to the
software MD, risks of wrong usage were obviously missing. Due to superficial
usability problems, the main risk for patients is that they register wrong data without
realizing. Another problem more related to a “loss of opportunity” is the feeling for
patients of not being followed by a real professional (because the interface with
professionals was not yet finalized). Several patients stressed the importance of not
feeling isolated but feeling “there is someone on the other side, a real person”.
5. Discussion - Conclusion
This study shows that cooperation with manufacturers within the regulation procedures
is possible, provided that (i) there is a benefit to the manufacturer and (ii) for “simple”
software MDs. The manufacturer here has a particular interest in following correctly all
the procedures to be included in the LPPR. His software MD is not a complex one. It
remains today a real challenge to test systems as they are implemented in highly
networked and interfaced systems. Concerning the qualification of software as MD, the
manufacturer is the one deciding regarding his interests. But it is also to market actors
to require for instance that software has a CE marking to be sure that it was developed
according to certain standard. One could imagine that hospitals, for instance, may buy
only software MD with CE marking to reduce the risk of compromised patient safety.
Another major challenge lies in maintaining innovation during the implementation of
regulations. We have to think about a potential set of recommendations for how
regulation could be implemented with minimal impact on innovation. Future
innovations should not be stifled due to poorly thought out safety regulations.
References
[1] Council directive 93/42/EEC, [cited Sep 8, 2012], Available from: http://eur-
lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:1993L0042:20071011:en:PDF.
[2] International Electrotechnical Commission, Medical Device - Software Life Cycle (Rep. No. ISO
62304) (2006), International Electrotechnical Commission, Geneva, 2006.
[3] D. Luzi, F. Pecoraro, Medical Device Software: A new Challenge, Proceedings of the XXIV Conference
of the European Federation for Medical Informatics, 2012.
[4] J. Talmon, S. Pelayo, The medical device directive and software: the challenges for the HFE community.
Communication presented to the 5th International Symposium on Human Factors Engineering in
Health Informatics, 2011.
[5] European Commission, [cited Sep 8, 2012], Available from: http://ec.europa.eu/health/medical-
devices/files/meddev/2_1_6_ol_en.pdf.
[6] User Interface requirements for the presentation of health data. HB 306-2007. Standards Australia,
2007.
[7] National Health Services, Common User Interface, [cited Sep 8, 2012], Available
from: http://www.connectingforhealth.nhs.uk/systemsandservices/data/cui.
[8] C.A. Longhurst, M.L. Howard, Health information technology and patient safety: rigorous transparent
evaluation of software is crucial, but regulation may stifle innovation, BMJ 344:e1096 (2012), 1-2.
[9] R. Koppel, D. Kreda, Health Care Information Technology Vendors’ “Hold Harmless” Clause:
implications for patients and clinicians, JAMA 301 (2009), 1276-1278.
[10] International Electrotechnical Commission, Medical Devices- Application of usability engineering to
medical devices (Rep. No. ISO 62366), International Electrotechnical Commission, Geneva, 2007.
Nursing Informatics
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-345
Developing National Level Informatics

Competencies for Undergraduate Nurses:
Methodological Approaches from Australia
and Canada
Elizabeth M. BORYCKIa,b1, Joanne FOSTERb, Tony SAHAMAb Noreen FRISCHc,
Andre W. KUSHNIRUKa
a
b
Queensland University of Technology, Brisbane, Queensland, Australia
c
Abstract. Health information systems are being implemented in countries by

governments and regional health authorities in an effort to modernize healthcare.
With these changes, there has emerged a demand by healthcare organizations for
nurses graduating from college and university programs to have acquired nursing
informatics competencies that would allow them to work in clinical practice
settings (e.g. hospitals, clinics, home care etc). In this paper we examine the
methods employed by two different countries in developing national level nursing
informatics competencies expected of undergraduate nurses prior to graduation
(i.e. Australia, Canada). This work contributes to the literature by describing the
science and methods of nursing informatics competency development at a national
level.
Keywords. nursing informatics, health informatics, education, competencies,

methods
Introduction
Internationally, we have seen a change in healthcare delivery with the introduction of

health information systems (HIS) such as electronic patient records (EPRs), electronic
medical records (EMRs) and electronic health records (EHRs). For example, EPRs are
being implemented in acute care settings in order to streamline health care processes,
improve the quality of healthcare and reduce the number of medical errors that arise
during health care delivery. In response to this modernization of health care, nurses
are being asked to use HIS in acute care, home care, long term care, and community
settings [1]. With these changes there has developed an increased demand for nurses
graduating from college and university undergraduate programs to have already
acquired competencies associated with the use of HIS and other technologies before
completing their program of study [1, 2]. Employers (e.g. regional health authorities)
are placing demands on university and college schools of nursing to produce graduates
who are able to use these technologies during patient care [2].
1
Corresponding author: Elizabeth Borycki, email: [email protected]
346 E.M. Borycki et al. / Developing National Level Informatics Competencies
There are few opportunities for nurses in their undergraduate studies to engage
with EMRs, EPRs and EHRs [1]. Many countries around the world: (1) have not fully
described the nursing informatics competencies that will be expected of nurses upon
graduating from their undergraduate nursing programs, (2) have not developed a clear
strategy for integrating these competencies into undergraduate nursing curriculums, and
(3) have not invested in developing tools representative of real-world clinical settings
that would help student nurses to understand the underlying theory and principles
embedded within HISs in order to be able to use these technologies [1]. As a result,
many student nurses are graduating without sufficient knowledge of nursing
informatics to be able to work effectively and efficiently in clinical practice settings
such as hospitals and clinics [1]. National governments, nursing educational
organizations and nursing informaticians around the world have recognized this is an
issue [2-4,6,8-11]. Two countries that are currently engaged in the process of
developing and integrating nursing informatics into undergraduate nursing programs
are Canada and Australia. In this paper we describe and compare the methods used to
develop these competencies, and the strengths and limitations of the approaches in
terms of work involving the science of developing national level competencies.
Methods for Developing Undergraduate Nursing Informatics Competencies
Although researchers have attempted to develop nursing informatics competencies at

the undergraduate and graduate levels [7], there are few countries that have attempted
to develop nursing informatics competencies for undergraduate nurses at a country
level. Both Australia and Canada have attempted to do so. Although there are many
aspects of undergraduate nursing informatics competency development that can be
focused upon [4], we have chosen to focus upon the methods that are used to develop
undergraduate nursing informatics competencies in this paper. For the purpose of this
work, we have chosen to focus upon the following dimensions (which are contained in
Table 1): the process of competency development, the types of background work that
was done to develop the nursing informatics competencies, the type of stakeholder
involvement, and the validation processes that were used (see Table 1). We have
maintained a constant definition of competency in reviewing this work. A competency
is a complex term that describes a professional’s ability to combine knowledge,
attitudes and skills with external resources and apply these to specific practice
situations [4].There are many similarities and differences between Australia and
Canada where nursing informatics competency development methods are concerned.
For example in both Canada and Australia initial literature reviews were conducted
followed by the development of draft competencies. There are differences as well: the
validation process in Australia involved sending the competencies to all nurses in the
country for comment [3]. In contrast, in Canada a forum of 50 participants that
included individuals with diverse geographical representation from across the country
and who were nurse educators, practicing nurses, representatives from national nursing
associations, students, and other health professionals participated in competency
development and validation. Many of these individuals had interest in nursing
informatics, yet only a few were nursing informatics experts. These stakeholders were
involved in the validation process, with the competencies being sent to Deans and
Directors of Schools of Nursing for additional comments.
E.M. Borycki et al. / Developing National Level Informatics Competencies 347
Table 1: Methods of Developing Nursing Informatics Competencies
Australia Canada
Process of Development
1. Review of literature 1. Development of a Task Force and sub-task
2. Survey all Australian Nurses using an online forces.
survey. 2. Review of the literature.
3. Develop draft competencies. 3. Draft a list of national nursing informatics
4. All Australian nurses review a draft list of the competencies by the sub-task force.
competencies. Nurse informaticians validate 4. Presentation to the full task force for review
the competencies. and refinement.
5. National focus groups are conducted for further 5. Presentation to stakeholders (nursing
validation of feedback informatics experts and a diverse group of
6. Changes are made based on feedback and a interested nurses and students from across the
final draft is sent out to all Australian nurses country) for review and refinement.
and nurse informaticians 6. Presentation for review to Deans and Directors
7. Further changes were made. of the School’s of Nursing for review and
8. Final competencies are developed and validated survey completion.
by nurse informaticians. 7. Refinement by the sub-task force.
9. Competencies are submitted for publication and 8. Presentation to the full task force and
acceptance into the National Registered Nurses refinement.
Competencies Framework
Background Work
- Extensive literature review was conducted to Review of the literature that included:
ascertain existing competencies outlined in the -A review of the national and international literature
research grey and academic literatures.
- Draft evidence based competencies were -A review of nursing competencies at both the
formulated from the analysis of the literature in provincial and regulator levels.
consultation with the Project Advisory Committee,
key stake holders and expert nurse informaticians
- An on line questionnaire (34 questions) of all
competencies was constructed from the literature
review, and sent to all Australian nurses to identify
NI priorities for the profession.
- Draft competencies were developed from the
outcomes of the questionnaire
Type of Stakeholder Involvement
Project Advisory Committee A taskforce of nursing informatics experts was
- Oversaw all of project brought together to develop the methods and oversee
Key stake holders (Australian Nurses) and expert the process: The findings were:
nurse informaticians -presented at a stakeholder forum to 50 nursing
- were consulted about the competencies (from informatics experts and others
draft through to completion) -stakeholders reviewed and 20 competencies
- Focus groups were conducted with nurses (8 developed
questions) Draft competencies were also presented to Deans
and Directors of Schools of Nursing, stakeholders
and the Canadian Association of Schools of Nursing
committee. This group of stakeholders was asked to
complete an online questionnaire with the
competencies.
Validation Processes
Following draft competency list approval: Following draft competency list approval:
- Competencies were sent to all Australian - Competencies were presented at a stakeholder
Nurses and nurse informaticians for validation forum for 50 nursing informatics experts and
- Focus groups were conducted other nursing groups.
- Changes were made to the competencies from - Changes were made based on feedback
feedback - Competencies were presented to Deans and
- A final draft was sent out to all Australian Directors of Schools of Nursing, stakeholders
Nurses and nurse informaticians for final and the CASN committee. This group of
validation stakeholders was asked to complete an online
- Changes were made based on feedback [3] questionnaire with the competencies. [2]
348 E.M. Borycki et al. / Developing National Level Informatics Competencies
Discussion and Conclusions
Healthcare is an information intensive industry that requires nurses to learn informatics

competencies [5]. Yet, nurses have few opportunities to learn about nursing
informatics within the context of their undergraduate programs [1]. Both Australia and
Canada have taken the first step toward integrating competencies into undergraduate
curricula- they have developed nursing informatics competencies. The processes
undertaken by both countries have both similarities and differences. Similarities exist
in terms of: (1) developing project groups/task forces to initiate work on the
competencies, (2) using empirical literature to drive initiate competency development,
(3) initiating the process using literature reviews, and (4) engaging in stakeholder
consultation to improve the competencies through consultation. There are some
differences between the countries. One country has chosen to use the research and grey
literature as well as regulatory information to inform their work while the other has
used the research literature as their primary source of literature. In addition to this,
there are differences in the validation processes between countries. Although both
countries consulted key stakeholders (e.g. nurses with informatics expertise), the
countries differed in their decisions regarding whom to consult. Australia validated
their work by sending out a draft version of the competencies to all nurses and nursing
informatics specialists in the country, while Canada reviewed draft versions of the
competencies with nursing informatics experts, nursing students in practice with
particular interest in this area, students and other health professionals as well as Deans
and Directors of Schools of nursing [3]. CASN taskforce members were involved in the
development of these competencies. There are advantages and disadvantages to using
these differing methodologies. There are advantages to consulting Dean and Directors
of Schools of Nursing. Such consultations would enable the developers of the
competencies to determine the feasibility of implementing them in Schools of Nursing.
There are also advantages to consulting front line nurses who are currently working
with the technologies that nursing students will be expected to be working with. It will
be interesting to review the strategies of other countries developing national level
competencies for undergraduate nursing students – to learn about the methods that were
employed. For example, in the United States nursing informatics competencies at the
undergraduate level were developed using a Delphi panel of experts in the field of
nursing informatics [12]. More importantly, understanding the similarities and
differences in competency development will allow for international comparisons in not
only methods for competency development but will allow researchers to determine if
methodology of development is a factor that influences the type of undergraduate nurse
competency development.
References
[1] Borycki EM, et al. The University of Victoria interdisciplinary electronic health record educational
portal. Stud Health Technol Inform, 2005;143:49-54.
[2] Nagle L, et al., Nursing informatics: Entry to practice competencies for registered nurses. Canadian
Association of Schools of Nursing: Ottawa. 2012.
[3] Kenny A. Online learning: enhancing nurse education? J Adv Nurs, 2002;38(2):127-135. DOI:
10.1046/j.1365-2648.2002.02156.x
[4] Goudreau J, Pepin J, Dubois S, Boyer L, Larue C, Legault A. A second generation of the competency-
based approach to nursing education. Int J Nurs Educ Scholarsh 2009;6:1-15.
E.M. Borycki et al. / Developing National Level Informatics Competencies 349
[5] Kushniruk AW, Borycki EM. Human, social and organizational aspects of health information systems.
IGI Global: Hershey, New York. 2008.
[6] Conrick M, et. al. A Framework for Nursing Informatics in Australia: A Strategic Paper.[Internet] 2004
Available from
http://www.hisa.org.au/system/files/u2615/Nursing%20Informatics%20in%20Australia_final%20Versio
n1%5B1%5D.pdf
[7] Garde S, Harrison D, Hovenga E. Skill needs for nurses in their role as health informatics professionals:
A survey in the context of global health informatics education. Int J Med Inform 2005;74(11-12):899-
907
[8] Coalition of National Nursing Organisations. Position statement: Nursing and E-health, [Internet] 2008.
Available from http://www.conno.org.au/pdf/CoNNO_Position_Statement_Ehealth.pdf
[9] Carter BE, Axford RL. Assessment of computer learning needs and priorities of registered nurses
practicing in hospitals. Comput Nurs 1993;11(3):122-126.
[10] Hegney D, Buikstra E, Eley R, Fallon T, Gilmore V, Soar J. Nurses and Information Technology: Final
Report. [Internet] 2007 Available from: http://www.anf.org.au/it_project/PDF/IT_Project.pdf
[11] Foster J. Development of Australian Nursing Informatics Competencies: Final Report, 2011
Unpublished.
[12] Staggers N, Gassert CA, Curran C. A Delphi study to determine informatics competencies for nurses at
four practice levels. Nurs Res 2002;51(6):383-97.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-350
What Nurses are Talking About: Content

and Community within a Nursing Online
Forum
Kathleen ABRAHAMSON a,1, Rebekah FOX b and James G. ANDERSON c
a
Western Kentucky University, Bowling Green, KY, USA
b
Texas State University, San Marcos, TX, USA
c
Purdue University, West Lafayette, IN, USA
Abstract. There is little research on the use of online forums by nursing

professionals as a way of creating community and exchanging professional
knowledge. The purpose of this study is to analyze the content of an online
nursing forum to determine the potential of online forums to constitute a
community of practice among nurses. We examined one month of thread topics
from a nursing focused general discussion forum and categorized (294 discussion
threads) according to content. The most frequent topics of discussion were advice
regarding career planning and clinical/technical questions. The majority of posts
dealt directly with the domain of nursing. The finding that nurses are seeking
information about clinical tasks from unverified anonymous online sources raises
concerns about the safety culture of health care institutions.
Keywords. online forum, nurses, communities of practice
Introduction
Online forums, defined as websites “that provide a venue of exchange of information

between people about a particular topic” [1], allow multiple participants to contribute
and/or respond to a comment or narrative in an asynchronous manner. As such, nursing
focused online forums are potentially a rich source of information regarding the
experiences, concerns, stories, and work-worlds of practicing nurses [2]. Existing
empirical research addressing nursing and internet use has focused on issues such as
the internet as a teaching tool for students [3], internet communication of health
information from provider to patient [4], health communication using mixed samples of
physicians, nurses, and consumers [5] and the privacy risks of forums use [6]. Less
research has addressed the use of online forums by nursing professionals as a means to
create community, share narrative, and exchange knowledge within their professional
domain. The goal of this study was to analyze the content of threads within a nursing
focused online forum to 1) better understand the topics and content of nursing
discussions that occurred outside of the nursing workplace but within the nursing
community and 2) discuss the potential of online forums to constitute a Community of
Practice (CoPs) among nurses.
1
[email protected]
K. Abrahamson et al. / What Nurses Are Talking About 351
1. Communities of Practice
The word community is used to mean a wide range of relationships based on such
categories as: geography (neighborhoods, villages, towns, cities), culture (ethnic groups,
religious groups), organizations (families, kinships, associations), identity (craft, trade),
and physical/biological similarities (scientific organisms) to name a few. One particular
type of community, connected through shared knowledge, is called a Community of
Practice (CoP). A Community of Practice, in a basic sense and for the purposes of this
study, can be thought of as a group of individuals who share a body of knowledge and
work together to solve a common problem. Organizational and rhetorical scholars have
argued that CoPs demand scholarly attention for their ability to increase effectiveness
and reduce organizational costs [7,8,9]. In addition, CoPs offer possibilities to save
lives (e.g., in high risk industries), and alter oppressive conditions by giving voice to
personal and collective narratives of its members. For example, CoPs have developed
in nursing and midwifery, firefighting, and the Armed Forces to serve as forums for
practitioners who share a concern for what they do, and wish to improve effectiveness
through regular interaction.
Although the theoretical parameters of CoPs are still being constructed [10],
Community of Practice is traditionally viewed as a combination of three fundamental
elements: domain, community, and practice [9]. Domain refers to the set of issues that
bring individuals together. These domains may be based on a profession (cardiologists,
teachers, nurses, etc.) or because members face similar problems in different contexts
(handling paperwork, workplace motivation, etc.). The domain represents the topic
with which CoP members are passionately interested. Community refers to the
individuals who participate in the CoPs and the social context they create. Membership
in CoPs is dependent on the idea of reciprocation; individuals trust that membership
will be beneficial and mutual respect encourages willingness to share, expose, and
provide empathetic feedback [9]. Practice is concerned with the exchange of
knowledge that improves effectiveness in whatever domain the community shares, and
creates a shared repertoire of tools, cases, and ideas. A defining characteristic of CoPs
is that, although they utilize a variety of formats, members may come from diverse
backgrounds a long as the interest in the domain is shared [9,10].
Tacit knowledge refers to knowledge that individuals have but cannot easily
communicate. Unlike explicit knowledge, which is formally communicated through
organizational processes such as training or orientation, tacit knowledge represents the
unspoken but important knowledge that does not get formally relayed, or participants’
intuitive knowledge of the domain [9,11]. CoPs have potential to give voice to
members of a community who may not otherwise be heard within formal
organizational structures, challenge oppressive ideologies through the expression of
narratives that tell the story of their practice, and construct new worldviews.
Nursing work has historically relied upon both explicit and tacit knowledge to
inform its practice [12]. Interestingly, the emergence of online asynchronous forums
may provide a visual text of such knowledge transfer between participants. We argue
that some online forums are an example of CoPs, and that these forums are worthy of
our scholarly attention. Online forums offer a unique space for nurse dialogue because
a) the internet provides a certain level of anonymity that may allow openness, b)
personal story-telling in these forums often encourages others to share narratives and
exchange tacit knowledge with an interest in the domain, and c) internet forums are
becoming more common and further exploration is needed to investigate the
352 K. Abrahamson et al. / What Nurses Are Talking About
communication occurring in this relatively novel setting. We explored the scope and
content of a nursing-focused community of practice, specifically a nursing
asynchronous online forum, to more fully understand how and why it is being used.
Results have potential to inform health care organizations regarding the domain and
practice issues that are relevant to CoPs participants.
2. Methods
Data was collected from the General Nursing Discussion forum found on the website
allnurse.com. This website was developed in 2003 as a resource for nurses, and
provides information about job opportunities, nursing education, financial aid, and
travel opportunities in addition to the asynchronous online forums addressing a number
of respondent-initiated topic streams. As such, this site represents the everyday
discourse of a community of practice in the domain of nursing. As a fairly new source
for mining texts, online forums require several a priori decisions to be made before
analysis. For this project, the particular site for study was chosen from among many
nursing forums for three reasons; (1) the forum is public and does not require
membership approval, (2) the forums are searchable, and (3) there is no warning to
researchers that the site is off limits. Once the site was chosen, on-line forums were
searched. We categorized respondent initiated threads within the online forum in terms
of content and characteristics. The study was designed specifically to be exploratory in
nature, and thus an open-coding method was employed to discover the topics addressed
by participants. Categories were first developed by the first author, then reviewed,
discussed, and refined through collaboration between authors. Each threaded post was
assigned to a single category deemed to best fit the overall content of the thread. To set
realistic parameters for this study, one month of thread topics (May of 2012) were
examined. Thus, a limitation of this work is the potential emphasis on issues that
occurred temporally close to the time of data collection. Future research would benefit
from a more time representative design.
3. Findings
A total of 294 discussion threads were initiated during the time period of May 1-May
31, 2012. The most frequently noted category of discussion was advice regarding
career planning within the domain of nursing. Examples of threads in this category
include questions regarding employment (Should I send my resume if I haven’t passed
the NCLEX yet?), pay expectations (Can I live this lifestyle as a nurse?), interview tips
(Should I speak of another offer during an interview?), and advice regarding career
strategy (How did you know your specialty and when?). The second most frequent
category included posts seeking answers to specific clinical questions or concerns.
Interestingly, many of these questions requested information that is readily available in
policy manuals or nursing reference materials (How do I flush a PICC line?), or cited
actions taken in the clinical setting then asked, essentially, ‘did I do it right?’ Nearly all
threads remained tightly linked to nursing-specific topics. The emergence of
discussions regarding unjust treatment (21%), job-related emotional difficulties (4%),
and sharing stories (4%) highlight the potential for internet forums to allow community
narratives to develop. Additionally, practice tools in terms of technical advice and
resource sharing accounted for approximately 25% of posts, reflecting the potential of
internet forums to allow for exchange of ideas applicable to practice change. Categories
of forum thread content are noted in Table 1.
Because privacy concerns were an emergent theme in the existing literature
addressing medical professional use of online forums [6], we also examined threads for
information that would identify patients, identify a specific facility, describe a patient
situation with enough detail that the patient could potentially identify themselves in the
thread, and/or endorse specific products. Of the 294 posts none named or otherwise
identified a patient, and only 4 posts named a facility (3 were in reference to potential
employment or an employment interview). Specific patient situations (without
identifying information) were highlighted in 11 posts; these 11 posts were contained
within the categories of dissatisfaction with nursing, concerns/commentary regarding
poor or dangerous care, technical questions, sharing stories, and ethic concerns. No
thread overtly endorsed a product.
4. Discussion
The goal of this study was to analyze the content of a nursing focused internet forum to
1) better understand the topics and content of nursing discussions that occurred outside
of the nursing workplace but within the nursing community and 2) discuss the potential
of internet forums to constitute a Community of Practice (CoPs) among nurses. We
discovered that within a single nursing focused internet forum the majority of posts
dealt directly with the domain of nursing, nurses were willing to share information that
included emotion-based content, and nearly half of the posts were contained within the
two most frequent categories (advice regarding career planning and technical
questions). Additionally, apart from the two most dominant categories, thread content
varied considerably over a wide range of categories, indicating that nurses were
engaging the forum membership for both professional interaction and information
needs. Surprisingly, nurses frequently used the forum to ask information regarding the
safety or procedural steps of clinical tasks such as IV line flushing and medication
administration, information that should be readily available on a nursing unit or in an
established nursing reference. The finding that nurses are seeking such information
from anonymous online sources (there is no way for members to verify the identity,
experience, clinical capability or qualifications of fellow members) is of concern, and
may reflect poorly on the safety culture of health care institutions. Despite recent
efforts to encourage nurses to openly question procedures, admit when further
information is needed, and discuss situations where patient safety may have been
comprised within their organizational settings, some nurses are instead seeking this
type of support from internet forums.
The nature of a community of practice (CoPs) is one that encourages individuals to
share their experiences in order to solve common problems. Some threads appeared in
the form of questions, wherein nurses seek advice from other community members,
while other threads were declarative in nature and made assertions regarding personal
experiences and professional knowledge. The CoP framework draws our attention to
knowledge “as communicatively constituted in practice” [10,p.193]. We recognize that
not all nurses who participate in online nursing forums represent CoPs, but believe
there is valuable information about how knowledge, especially tacit knowledge, is
rescued and shared through this type of interaction. As theorists continue to examine
354 K. Abrahamson et al. / What Nurses Are Talking About
CoPs, they are realizing that the “enactment of CoPs can differ greatly as regards
mutual engagement, shared repertoire, and negotiation of a joint enterprise” [10 ,p.
195]. If the benefits of CoPs are to be realized, we must also take the changing nature
of community into account in order to include the explicit and tacit knowledge shared
in online communities.
Table 1. Categories of Forum Thread Content
Category n %
Advice re: career planning (hiring, pay, interview, job choice, career 75 26%
strategy)
Technical questions, clinical questions 51 17%
Advice re: formal education (course of study, coursework, schools) 24 8%
Perception of unjust treatment by administration or leadership 21 7%
Issues related to scheduling or shift work 14 5%
Handling job-related emotional difficulties 13 4%
Sharing stories (telling a story and/or asking others for their stories) 13 4%
Issues specific to nursing management 12 4%
Professional development or training/related readings/ instructional videos 11 4%
Licensure issues 9 3%
Interpersonal conflict with co-worker or colleague 9 3%
Dissatisfaction with nursing career in general 8 3%
General discussions of issues affecting nursing care 5 2%
General discussions of issues affecting personal health 5 2%
Discussion comparing practice settings without requesting career advice 4 1%
Concerns/commentary of poor or dangerous care 4 1%
Discussion regarding of public image of nursing 4 1%
Comments on forum use or the forum itself 4 1%
Questioning the ethics of a specific situation 3 1%
Using forum to gain employment or recruit nurses 3 1%
Event announcements 3 1%
Missing/unable to categorize 3 1%
Interpersonal conflict with patient 2 <1%
Using forum to recruit nurses for research studies 1 <1%
Asking where to find a specific resource 1 <1%
References
[1] PC Magazine, [cited Sep 1, 2012]. Available from: http://www.pcmag.com.

[2] E. Im, W. Chee. An Online Forum as a Qualitative Research Method: Practical Issues, Nursing Research
55 (2006), 267-273.
[3] M. Boulos, I. Maramba, S. Wheeler. Wikis, Blogs and Podcasts: A New Generation of Web-based Tools
for Virtual Collaborative Clinical Practice and Education, BMC Medical Education 6 (2006).
[4] M. Solomon, S. Wagner, J. Goes. Effects of an Online Intervention for Adults with Chronic Conditions
on Patient Activation: Online Randomized Controlled Trial, Journal of Medical Internet Research 14
(2012).
[5] E. Miller, A. Pole. Diagnosis Blog: Checking up on Health Blogs in the Blogosphere, American Journal
of Public Health 100 (2010), 1514-1519.
[6] T. Lagu, E. Kaufman, D. Asch, K. Armstrong. Content of Weblongs Written by Health Professionals,
Journal of General Internal Medicine 10 (2008), 1642-1646.
[7] J. Iverson, R. McPhee. Knowledge Management in Communities of Practice: Being True to the
Communicative Character of Knowledge, Management Communication Quarterly 16 (2002), 259-266.
[8] B. Lehaney, S. Clarke, E. Coakes, G. Jack. Beyond Knowledge Management, Idea Group Publishing,
2004.
[9] E. Wenger, R. McDermott, W. Snyder. Cultivating Communities of Practice, Harvard Business School
Press, 2002.
[10] J. Iverson, R. McPhee. Communicating Knowing Through Communities of Practice: Exploring Internal
Communicative Processes and Differences among CoPs, Journal of Applied Communication 36 (2008),
176-199.
[11] Polanyi, M. The Tacit Dimension, Doubleday and Company, 1983.

[12] S. Lake, C. Moss, & J. Duke. Nursing Prioritization of the Patient Need for Care: A Tacit Knowledge
Embedded in the Clinical Decision Making Literature. International Journal of Nursing Practice 15
(2009), 376-388.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-356
A Framework for Leveling Informatics

Content Across Four Years of a Bachelor
of Science in Nursing (BSN) Curriculum
Noreen FRISCHa1 and Elizabeth BORYCKIa,b,
a
a,b
Abstract. While there are several published statements of nursing informatics

competencies needed for the Bachelor of Science in nursing (BSN) graduate,
faculty at schools of nursing has little guidance on how to incorporate the teaching
of such competencies into curricula that are already overloaded with required
content. The authors present a framework for addressing nursing informatics
content within teaching plans that already exist in virtually all BSN programs. The
framework is based on an organization of curriculum content that moves the
learner from elementary to complex nursing concepts and ideas as a means to level
the content. Further, the framework is organized around four broad content areas
included in all curricula: professional responsibility, care delivery, community
and population–based nursing, and leadership/management. Examples of
informatics content to be addressed at each level and content area are provided.
Lastly a practice-appraisal tool, the UVIC Informatics Practice Appraisal – BSN is
presented as a means to track student learning and outcomes across the four years
of a BSN program.
Keywords Nursing informatics, baccalaureate nursing education, nursing

curriculum
Introduction
There have been several efforts over the years to establish nursing informatics
competencies for nurses, beginning with Staggers and colleagues work in 2001 and
2002 [1,2]. This work was followed in the U.S. more recently by the TIGER initiative
(Technology Informatics Guiding Educational Reform) [3-5] and work in Canada
supported by Canada Health Infoway and the Canadian Association of Schools of
Nursing [6]. The resulting competency documents provide statements describing the
knowledge, skills and attitudes needed for modern nursing practice and suggest that
nurse educators must incorporate nursing informatics content into their curricula. In
addition, the national accrediting bodies for nursing education in both countries suggest
that graduating students should be prepared with basic nursing informatics
competencies (6,7). Although these efforts indicate agreement that informatics content
is required in undergraduate curricula, faculty has little guidance on teaching such
1
N. Frisch and E. Borycki / A Framework for Leveling Informatics Content Across Four Years 357
content. For example, the TIGER initiative provides broad goal statements such as a
beginning nurse should “access data and perform nursing documentation using
computerized patient records” and should “recognize the role of informatics in nursing”
(8, p 58] but does not provide concrete examples about how or where to bring a wide
range of nursing informatics competencies into undergraduate nursing curricula that are
already challenged to cover a wide range of required content. In addition, prior work
on leveling nursing informatics content has addressed competencies and outcomes
across levels of nursing education (i.e. the undergraduate, masters and doctoral
level)[1], not leveling content within undergraduate programs. The current paper
presents a contribution to the literature as it describes how nursing informatics
competencies can be integrated across BSN curricula.
The purpose of this paper is to present a useable framework, developed by the
authors, suggesting ways that faculty can incorporate much of the needed material
into content areas already being addressed in virtually all nursing programs within the
four years of a baccalaureate nursing program. In addition, the authors present a
practice appraisal tool that faculty and students may use to track exposure and
achievement of nursing informatics competencies over the four years of a program.
Background: Organization of Nursing Curricula: Elementary to Complex
In baccalaureate programs, nursing curricula are designed to move the learner through
course content and clinical experiences that ensure the graduate is ready for beginning
professional practice. This means that in a four-year university program, the learner
begins as a novice nurse (i.e., one without nursing knowledge and experience) and ends
as a generalist ready for practice in any of nursing specialty areas. Nursing curricula
have been widely influenced by the work of Benner [9] who has documented the
development of nurses throughout their careers in her publication “From Novice to
Expert”. Though there are program and regional variations, nursing curricula take an
approach of thinking about student learning and development over the four-year
experience. Nursing curricula begin with content and experiences that are more
elementary or foundational to the discipline and move the learner through experiences
that build on that foundation to increasing levels of complexity. Thus, it is relatively
common that students provide care first to individual adults and care for families and
communities later; they build care management skills by taking responsibility for one
patient, moving to take responsibility for a group of patients, and only then take on unit
management/leadership experiences.
In order to incorporate nursing informatics awareness and competencies into
nursing curricula, the authors have selected four content areas that are addressed in one
way or another in all programs. These are: 1) professional responsibility, 2) care
delivery, 3) community nursing and care of populations, and 4)
leadership/management. These content areas are present in a BSN curriculum as
illustrated in Figure 1 as four squares. Nursing informatics represented by the arrows
moving from the centre out, depicting the integration of such competencies from
elementary to advanced.
358 N. Frisch and E. Borycki / A Framework for Leveling Informatics Content Across Four Years
Professional
Care Delivery
Responsibility
Community/
Leadership
Population- and
based Management
Nursing
Figure 1: Nursing Informatics Competencies across a Four Year Nursing Curricula

In the sections that follow, the authors suggest specific informatics content related
to each of these broad areas, provides examples of how this suggested content
addresses published competency statements, and give examples of items that could be
used in student practice appraisals to retrieve competency achievement.
Content Area 1: Professional Responsibility
The topics of professional responsibility, accountability for professional behavior, and

the legal/ethical issues of nursing practice are always addressed in the beginning of
nursing curricula. The learner must understand that nursing practice is based on a legal
scope of practice, that the use of professional nurse title (Registered Nurse) is governed
by a regulatory body, and that the nurse is legally and ethically accountable for all
nursing actions. There is opportunity for faculty to address a wide range of informatics
content in this introductory content. Privacy and confidentiality are two concepts the
novice nurse must understand in a new, professional context. Topics such as access to
digital records, password security, use of social media, cell phones, photographs, and
text-messaging can and should be addressed early. In addition, for the novice nurse
who has learned to rely on technology in his or her personal life, the possibilities of
technology failures and e-iatrogenic errors/technology-induced errors can be
incorporated into course content about use of records, decision supports and the nurse’s
accountability for all professional actions. Lastly, in the beginning courses, the topic
of the professional and therapeutic nurse-patient/client relationship is addressed. Here,
discussion of the challenges of developing appropriate supportive practice while using
technology is an important consideration. Faculty understands that technology can
readily become a focus for care and in some circumstances the technology impedes the
development of a new nurse’s ability to interact with the patient.
Addressing these areas of privacy, confidentiality, access to records, device use,
technology induced errors and development of professional/therapeutic relationships
attends to the competency requirements of the TIGER competency of “clinical
management and care responsibility” with its related skills of assuring confidentiality
and system security in information[10] and the CASN informatics competency

document [6] under domain of “professional and regulatory accountability” with
indicators related to legal/ethical standards and use of professional judgment and
accountability in all practice decisions; and under the domain of “information and
communication technologies” with indicators related to use of technology that supports
the nurse-patient relationship. Examples of performance appraisal indicators are listed
in Table 1.
Table 1: Informatics - Professional Responsibility: Selected Performance Appraisal Indicators

Uses only those technologies that ensure the privacy and confidentiality of patients.
Posts, enters or saves files (including digital photo files, text and/or audio files) on only password protected,
organizationally-sanctioned secure sites.
Reports any loss or theft of hardware, software breaches or presence of computer viruses.
Develops a practice style that incorporates technology use and positive nurse-patient relationships.
Content Area 2: Care Delivery
Students provide basic care in beginning nursing courses and most often learn clinical
skills in a nursing laboratory before carrying out care in actual clinical situations. There
are two areas of care delivery that require the learner to understand care in a
technology-enabled practice setting: 1) documentation of nursing care and 2)
medication administration.
In learning care documentation, the beginning nurse needs an understanding of the
legal status of nursing notes, signatures (paper or electronic) that indicate personal
knowledge/observation of the material for which the signature is given, and -- in digital
records -- the differences between narrative language and use of standardized
languages. Introduction to nursing and other standardized languages (such as the
ICNP®, NANDA-I, C-HOBIC, ICD-10, DSM-IV-R) is essential content. The
beginning nurse need not be an expert in all of these language systems, but needs to
understand their development and use in recording in electronic health records systems.
Providing a foundation early in a nursing curriculum with reinforcement of knowledge
throughout the program gives students an opportunity to grow in their sophistication of
use, critique, and, ultimately, improvement of digital documentation systems.
Medication administration cannot be taught without attention to paper systems,
hybrid systems and electronic order entry systems. Nursing laboratories that simulate
all three are best suited to teach the learner how to practice in current and future
systems. Paper systems provide the basics for the learner to understand how a
medication order is processed from initial writing to administration and recording.
Electronic systems involving physician order entry, pharmacist and nurse receipt,
pharmacy dispensing, and nurse administration are complicated and, when explained,
help the student understand care processes and the nursing and interdisciplinary roles in
safe practice. Decision supports related to medication dose and interactions between
medications provide opportunity for learning related to system benefits, possible errors,
need for professional judgment and vigilance in professional behaviours.
Addressing issues such as nursing documentation, standardized languages,
decision supports, accountability for professional decisions and actions, and
understanding of systems processes address the TIGER competency area of “clinical
information management” and its related skills of maintain records, capturing data,
producing summary records of care and managing medication orders and care [10]and
the CASN informatics competencies [6] under the domain of “information and
knowledge management” with indicators related to use of nursing data using
standardized languages, recording nursing data, use of hybrid and homogenous record
systems, and under the domain of “information and communication technologies” with
indicators related to understanding various components of health information systems.
Table 2 provides examples of performance appraisal items of these competencies.
Table 2: Informatics – Care Delivery: Selected Performance Appraisal Indicators

Describes the underlying workflow that supports medication ordering, dispensing and administration when
using an electronic health record that supports physician order entry, pharmacy information systems and
medication administration systems.
Communicates with physician and/or pharmacist if there are questions about a medication order
(demonstrates awareness of underlying rationale for a physician or prescriber bypassing an alert or reminder
where the patient’s health status is concerned).
Documents care according to the standardized language systems used in practice settings.
Describes which parts of the nursing record are codeable, stored, and retrievable.
Provides examples of how hybrid records could introduce opportunities for errors.
Content Area 3: Community/Population-based Nursing
Community nursing and care of populations most frequently comes later in a BSN
curriculum – often in years three or four. The student must build on knowledge and
practice experience with care of individuals to care of client groups to expand practice
to the community/population level. Areas that the student must learn have to do with
retrieval and interpretation of population-based data to determine needs, risks and
surveillance activities as well as need to consider basic privacy/confidentiality issues in
terms of transferal of data over wireless networks (from client homes to clinical record
systems), protection of client/patient data when data are held on hand-held laptops or
portable devices such as computers used in the home. The student will need to become
acquainted with population datasets, interpretation of population-based statistics and
new considerations for privacy and confidentiality. In addition, when providing care to
clients in their homes, the student needs to gain an awareness of how clients obtain
access to their personal health data and make use of personal health records. When this
new content is delivered through classroom and clinical experiences, the faculty are
addressing the TIGER competency of “management of patient groups or populations”
and related skills of use of information to support epidemiological investigations and
population health issues, and the CASN informatics competencies [6] under the
domain of “information and communication technologies” with indicators related to
care delivery to diverse populations, assisting patients and their families to use
information and technologies to manage their own health, and critical evaluation of
data from a variety of sources. Table 3 provides examples of performance appraisal
items of these competencies.
Table 3 Informatics – Community Nursing/Population-based Care Selected Performance Appraisal

Indicators
Describes how data can be transferred in the community via secure networks.
Ensures that hardware used is stored in safe and secure settings between home visits.
Teaches patients and their families how to access and use personal health information from digital records.
Accesses population-based information from available datasets.
Content Area 4: Leadership and Management
Courses in leadership and management come near the end of BSN curricula. Students
need to understand approaches to leadership and to think about their own personal
leadership styles. While a new graduate is rarely hired directly into a management
position, BSN graduates will be in positions of leading teams and directing and
supervising care soon after graduation. Further, many will move into supervisory
positions without returning to school for additional formal education. There are two
important informatics content areas that directly impact the nurses’ ability to serve in
supervisory and management positions: 1) access to data for assessments of quality and
safety, and 2) evaluation of technology and its impact on workflow. Quality and
safety have become essential parameters for evaluation of institutional performance.
While each healthcare system will have related data identified and stored in varying
ways, the BSN graduate nurse must have enough background to understand data
collection, retrieval and interpretation for these purposes. Use of aggregate data to
identify trends, ‘change points’, and progress toward institutional goals can be taught at
the undergraduate level. Again the BSN graduate will not be the expert, but must have
a basic understanding of the processes used to ensure quality and safety. Also, a nurse
supervisor needs an understanding of the well-known fact that technology impacts
workflow and behaviours [11]. Adding hand-held devices, personal paging systems,
bar code identification, or new documentation systems will impact (positively or
negatively) on the work of the nursing team. To fully participate in technology roll-
outs, a supervisor must be in a position to ask questions related to workflow, time
management and the results of pilot-tests of new technology. Courses can include this
content with case-studies from practice experiences to help the graduate learn. Use of
data for quality and safety and an understanding of technological impacts on workflow
address TIGER competency of “clinical information management” and related skills
such as data capture, information management in relation to clinical workflow tasks
[10] and the CASN informatics competencies [6] under the domain of “professional
and regulatory accountability” with indicators related to advocacy for use of
information systems to support safe, quality care; and under the domain of
“information and communication technologies” with indicators related to
understanding the ability of technology to improve health systems and quality care.
Table 4 provides examples of performance appraisal indicators.
Table 4: Informatics – Leadership and Management: Selected Performance Appraisal Indicators

Tracks data on unit performance for quality measurements.
Analyzes data collected to provide input to quality improvement activities.
Works with health informatics professionals to evaluate impact of technology on care processes.
Advocates for nursing staff where technology safety is concerned.
Practice Appraisals
Documenting student learning and tracking student learning outcomes are integral tasks
for professional nursing programs. Practice appraisal forms provide both faculty and
students the ability to document exposure to content and learning of content across the
entire four years of a program. Such forms can be used as part of end of term student
evaluations, incorporated into student portfolios of individual learning, and as measures
contributing to overall program evaluation. The authors have developed a nursing
informatics practice appraisal tool (see Appendix A) that is a form that could be used
for each of these purposes. This tool, titled the University of Victoria Informatics
Practice Appraisal - BSN is organized according to the four content areas addressed
above and provides indicators of student learning. The authors present this tool for
faculty or students who may wish to incorporate it into their own programs.
Discussion
Nursing faculty is often faced with the challenge of being asked to add content to their
courses and curricula without being able to omit content already present. This need to
add new and changing content is even more difficult when faculty themselves are not
expert in the new content. Existing frameworks for leveling nursing informatics
competencies provide information about how to integrate these technology-based
competencies across differing levels of nursing education (i.e. the undergraduate,
masters and graduate). In this work we outline a framework that can support faculty
decision making when integrating nursing informatics competencies across a four year
undergraduate baccalaureate in nursing program. The framework we outline illustrates
a place in curricula where content and learning experiences can be incorporated into
existing teaching plans and activities and provides a way forward for faculty addressing
the informatics content required for practice in our future. We also present a practice
appraisal tool for aiding faculty in their process of documenting and tracking student
learning. Our future work will include testing the framework to determine whether it
can be used to support faculty decision making in integrating such context in an
undergraduate program context. In carrying out this work (i.e. integrating nursing
informatics content into four year baccalaureate programs), faculty will be able to
demonstrate that their curricula meet current published standards for informatics in
BSN curricula and will be able to retrieve student learning of this content in practice
appraisals.
References
[1] Staggers N, Gassert CA, Curran C. Informatics competencies for nursing at four levels of practice. J of
Nurs Education, 2001;40(7);303-316.
[2] Staggers N, Gassert CA, Curran C. A Delphi study to determine informatics competences for nurses at
four levels of practice, Nursing Research, 2002;51(6); 383-97.
[3] Skiba DJ, Dulong D. Using the TIGER vision to move your agenda forward. Nurs Management,
2008;2,14-16.
[4] Sensmeier J. 2006 TIGER Summit – Evidence and informatics transforming nursing. CIN, 2007;1;55-
56.
[5] Dulong D. Informatics: the TIGER project. Online J of Issues in Nurs, 2009;13(2);(2p).
[6] Canadian Association of Schools of Nursing (CASN). Nursing Informatics Entry-to-Practice

Competencies for Registered Nurses. http://www.casn.ca/en/Whats_new_at_CASN_108/items/123.html
retrieved September, 2012.
[7] Commission on Colleagiate Nursing Education (CCNE), Essentials for Baccalaureate Nursing
Education, http://www.aacn.nche.edu/education-resources/BaccEssentials08.pdf retrieved September,
2012.
[8] Hebra T, Calderone L. What nurse educators need to know about the TIGER initiative. Nurse Educator,
2010;35(2), 56- 70.
[9] Benner P. From Novice to Expert: Excellence and power in clinical nursing (commemorative ed).
Upper Saddle River, NJ: Prentice Hall.
[10] Technology Informatics Guiding Educational Reform (TIGER), TIGER Competencies retrieved from
http://tigercompetencies.pbworks.com/w/page/22247287/FrontPage, September, 2012.
[11] Borycki EM, Kushniruk AW, Kuwata S, Watanabe H. Simulations to assess medication administration
systems (pp 144-159). In B. Staudinger, V. Hoess & Herwig Ostermann (Eds.). Nursing and Clinical
Informatics: Socio-Technical Approaches. Hershey, Pennsylvania: IGI Global. 2009.
Appendix A
University of Victoria Informatics Practice Appraisal –

BSN:
A Template for Tracking Nursing Informatics Competencies
Developed by Elizabeth Borycki, PhD, RN, School of Health Information Science and
Noreen Frisch, PhD, RN, FAAN, School of Nursing*
__________________________________________________________________
Content Area 1: Professional Responsibility
• Understands the difference between public space, private space, and the
need to maintain privacy and confidentiality when using differing types
of software and hardware
Uses only those technologies that ensure the privacy and
confidentiality of patients
Posts, enters or saves files (including digital photos files, text or
and audio files) on only password protected, organizationally
sanctioned secure sites.
Accesses patient information using organizationally sanctioned
resources
Accesses only the patient information for those patients for
whom one is responsible for
Accesses confidential patient information only during periods of
medical crises
• Recognizes limitations, potential errors, and errors in electronic
systems.
Reports any loss or theft of hardware, software breaches or
presence of viruses
Uses clinical judgment when confronted with decision support
Reports technology errors/failures and instances where the
technology does not effectively support nursing practice.
• Maintains relational practice and therapeutic communication within
technologically-enhanced environments
Develops a practice style that incorporates technology use and
positive nurse-patient relationship.
Identifies when technology is disrupting nurse-patient
therapeutic communication and modifies communication style or
technology use accordingly
______________________________________________________________________
Content Area 2: Care Delivery
2a. Medication Administration

• Understands the underlying workflow that support medication ordering,
dispensing and administration when using an electronic health record
that support physician order entry, pharmacy information systems and

medication administration systems
• Communicates with physician and/or pharmacist if there are questions
about the medication order
Is aware of the underlying rationale for a prescriber bypassing
an alert or reminder where the patient’s health status is
concerned
Questions if medication order data appears incorrect as it
appears in the medication administration system (i.e. the wrong
dose or if the medication is the incorrect medication)
2b. Electronic Documentation
• Demonstrates ability to use electronic documentation for nursing

documentation
Documents according to the standardized language systems used
in practice settings
Documents accurately on checklists, assessment forms, and
other computerized systems.
Accesses patient data from the record as needed to perform safe
care.
• Understands which parts of the nursing record are codeable, stored,
and retrievable.
• Understands how hybrid environments could introduce opportunities
for errors.
Understands the communication and information access
challenges of hybrid environments (i.e. where part of the patient
record is paper based and part of it electronic) and develops
strategies to overcome them
Reports errors or near misses arising from work in a hybrid
environment to nurse managers, the information technology
department, professional practice and to error reporting systems
______________________________________________________________________
Content Area 3: Community and Population-Based Nursing
• Understands privacy issues related to community care and electronic

documentation, transfer of information and information retrieval
Describes how data can be transferred via secure networks
Follows agency policies in all aspects of data collection, storage
and retrieval.
Ensures that hardware used is stored in safe and secure settings
between home visits.
Educates patients and families about the most appropriate
communication channels between nurses, clients, and families to
maintain privacy and confidentiality.
• Is aware of technological supports available for use in home and

ambulatory settings to manage chronic disease and/or to enable safe
and independent living.
Demonstrates (articulates) ability to meet the issues and
challenges of monitoring, independence and privacy in
community settings.
______________________________________________________________________
Content Area 4: Leadership and Management
• Has basic data and information literacy skills

Tracks data on unit performance for quality assurance and
comparisons
Analyzes data collected via the electronic health record to
provide inputs into professional practice and quality
improvement activities
Uses analyzed data to undertake evidence-based, management
decision making
Presents data to staff about units clinical outcomes
• Understands how work is embedded in the technology and how the
introduction of new technologies affects workflow (e.g. wireless IV
pumps, smart beds, wireless vital signs monitors and the electronic
health record)
Works with the health informatics professionals to optimize
workflow arising from software and placement/use of the
technology
Works with health informatics professionals to evaluate the
impact of technology upon unit processes and
clinical/organizational outcomes
• Advocates on behalf of the nursing staff when issues arise when the
hardware and software does not adequately meeting health professional
needs.
• Advocates for nursing staff where technology safety is concerned
_______________________
*The authors would appreciate feedback from colleagues using this tool for further
extension and development
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-367
Patient Safety Perspectives: The Impact of

CPOE on Nursing Workflow
Mowafa HOUSEHa,1, Anwar AHMADa, Anwar ALSHAIKHa, Fatimah ALSUWEEDa
a
College of Public Health and Health Informatics, King Saud Bin Abdulaziz University
for Health Sciences (KSAU-HS), National Guard Health Affairs (NGHA), Riyadh,
Saudi Arabia
Abstract. The purpose of this review is to explore the impact of Computerized

Physician Order Entry (CPOE) systems on patient safety from a nursing
perspective. The paper discusses the importance of safety culture within nursing,
nursing perceptions of CPOE, and the impact of CPOE on nursing workflow. The
findings indicate that the implementation of CPOE negatively impacts nursing
workflow when CPOE systems are inadequately designed. Future work is
necessary to explore the impact of CPOE on nursing workflow and the direct
impact on patient safety.
Keywords. Patient Safety, Nursing, CPOE, Medication Errors.
Introduction
In 1999, the Institute of Medicine (IOM) released a report, To Err is Human, which
predominantly focused on health care errors and revealed that between 44,000 and
98,000 Americans die from medical errors each year [1]. The report stressed the
importance of using reporting systems to collect standardized information about
adverse events for the purpose of improving patient safety. The report also
recommended the use of voluntary reporting systems for systemic failures and near-
misses to identify potential problems before harm occurs [1].
Almost 12 years later, in 2012, the Institute of Medicine produced a report on the
impact of health information technology on patient safety [2]. The IOM reported that
health information technology products are expected to improve patient safety only if
the products are well-designed and strategically implemented. Incidences in which
poorly designed health information systems caused medical errors were a concern,
particularly due to the impact on patient safety. The report encourages collaboration
between health information technology vendors, physicians, nurses, and hospitals to
improve patient safety.
According to IOM reports, Computerized Physician Order Entry (CPOE) systems
were identified as a means of improving patient safety by reducing hospital medication
errors. CPOE systems are broadly considered an essential component for health
information systems that helps improve patient safety by reducing medical errors.
CPOE provides passive and active reminders, alerts and alarms that notify medical
1
368 M. Househ et al. / Patient Safety Perspectives: The Impact of CPOE on Nursing Workflow
staff (such as nurses) of incorrect orders, and information about possible drug-drug
interactions and missed medication doses that allow medical staff to correct, modify
and rewrite incorrect entries[3]. A large number of serious medication errors occur
every year and include prescribing the wrong drug, drug over-dosage or unobserved
drug interaction and allergy. CPOE offers accurate drug information, dose auto-
calculations and support for making the appropriate decision at the point of care.
CPOE application helps decrease medication errors in multiple ways. For example,
when a nurse inputs a physician’s order into a CPOE system and there is a mistake on
the order, the CPOE system will give him/her a warning alarm. As a result, the nurse is
given the opportunity to review the prescribed medication with the physician and to
correct and re-enter the order.
A study conducted by Bates et al. showed that CPOE application reduced the
medication error rate by 55% [4]. A systematic review of 10 studies was performed:
five of the studies indicated that the use of CPOE significantly decreased the number
of medication errors, four of the studies concluded that the resulting decrease in
medication errors was not significant and one of the studies showed no change in the
number of medication errors with the use of CPOE [5]. Some studies have shown that
the application of the CPOE system has a negative impact on medication errors. A
study conducted by Koppel et al. reported that the CPOE system facilitates medication
errors and identified 22 types of medication errors that can result from CPOE
application [6].
The use of CPOE in hospitals is facilitating many changes to the processes of
prescribing, dispensing and administering medications. These changes are becoming
more apparent in the tasks of nursing staff, as more physicians are requesting nurses to
enter orders into CPOE systems [7]. In 2009, Fields et al. reported that the total impact
of CPOE on nursing can be determined by evaluating three aspects of nursing: process,
relationships and operations. Moreover, nurses’ concerns and difficulties with using the
CPOE system may potentially affect nursing workflow [8].
Little work has determined nursing perspectives on the use of CPOE and its
potential impact on nursing workflow and patient safety. In this paper, we discuss the
importance of the patient safety culture within the nursing profession, the nursing
perception of CPOE, and the impact of CPOE on nursing workflow.
1. Nursing and Patient Safety
Patient safety is defined by the IOM as "the prevention of harm to patients"[9]. There
exists much concern regarding the healthcare system’s ability to prevent errors and to
facilitate learning from errors, which requires the involvement of the hospital
administrator and medical staff [10]. Improving quality within hospitals has long been
a goal of medical professionals, especially nursing professionals. For example, in 1855,
Florence Nightingale played a major role in creating a patient safety culture when she
gathered and analyzed the mortality data of British troops and then promoted hygienic
practice [11]. In 1859, she was credited with initiating the implementation of the
world’s first hospital performance measures. Today, nurses play a major role in
improving patient safety within hospitals. The impact of nurses on patient safety is
greater than the impact of medication administration or injury prevention in health care
settings .
M. Househ et al. / Patient Safety Perspectives: The Impact of CPOE on Nursing Workflow 369
2. Nursing Perceptions Towards CPOE
One of the benefits of using CPOE in nursing practice is that CPOE supports safe drug
administration by promoting “the five rights of drug administration”; this can provide
nurses more time to focus on the professional steps of medication administration, such
as the monitoring of adverse events and the continuous assessment of patients [14].
However, nurses have resisted the introduction of information technology into hospitals.
In 2003, Trimmons reported on a study that evaluated nurse resistance to the use of
information technology [15]. The study revealed that nurses found that the use of
information technology increased their dependence on technology and therefore
undermined their critical thinking skills. The nurses in the study were also concerned
with the reliability and security of health information systems. Additionally, nurses
were concerned that they would be unable to access patient information during a
system failure.
However, Ash et al. reported that nurses were comfortable in using CPOE systems
that were well-designed. Nurses reported that easy-to-use systems improved their
workflow and that badly designed systems actually hindered workflow, especially
when they were not customized or implemented accordingly. The authors also reported
that one major success of the project resulted from strong support and leadership from
management [16].
The ways in which nurses interact with CPOE are also changing. CPOE systems
were originally designed for physicians to enter medication and lab orders. However, a
study conducted by Kazemi et al. (2010) showed that most of the CPOE systems
implemented in hospitals are not used by physicians, but rather by nurses and other
medical staff [7]. Physicians were found to be resistant to using CPOE systems and
relied on nurses and other medical staff to input CPOE orders.
The studies conducted by Trimmons, Kazemi and Ash indicate that physicians are
relying on nurses to enter CPOE orders and that, as long as the technology helps
improve patient care and is designed to support workflow, nurses are more willing to
use the technology.
3. CPOE Impact on Nursing Workflow
The full impact of CPOE on nursing tasks and patient safety has not been examined
and, therefore, remains poorly understood. Small studies are available on this subject,
but as more health information systems are implemented and used by nurses, the need
to understand the impact of these systems on nursing workflow and patient safety is
increasing. Ash et al. reported that nurses and other clinicians were willing to use
CPOE systems if they were well-designed and implemented [14]. The study noted that
CPOE implementation had a major impact on nursing, but it was unclear whether
CPOE systems would decrease or increase the duration, number or type of nursing
tasks, such as order processing and other patient care duties.
Another study, conducted by Bleich et al., reported that the implementation of
CPOE increased the time nurses spent performing tasks on the system while reducing
time spent performing patient care activities [16]. The authors recommended achieving
a greater and more thorough understanding of the CPOE system and the effects on
CPOE users, including nurses who interact with the system daily to retrieve and
manage patient orders and improve patient care and safety.
370 M. Househ et al. / Patient Safety Perspectives: The Impact of CPOE on Nursing Workflow
When a CPOE system is implemented and physicians enter orders from other
places within the hospital, nurses become dissatisfied with CPOE. Nurses must log into
the system many times to check for new orders, which increases the time spent using
the system rather than visiting bedsides. Therefore, this additional clerical task is
imposed on already-burdened professionals and is also economically inefficient [15]. A
study examining nursing time expenditure determined that nurses spend approximately
20% of their time on direct patient care, 9% on activities related to the unit, 25% on
indirect patient care, 30% on documentation, and 15% on personal activities [17]. In
fact, there exists a difference in the amount of time nurses spent on documentation
before and after using the CPOE system, indicating that more time will be available for
nurses to spend on direct patient care and improving the quality of work documentation
[18].
Successful implementation and adoption of CPOE are more likely when the
developers and implementers understand the complexities and unpredictability of the
nurses' workflow. For instance, an inability to ensure the compatibility of the system
with nursing workflow may lead to increased encounters of unintended consequences.
Clearer guidance is required from hospitals on the use of CPOE systems by physicians,
nurses, and other medical staff. Without clear policies, well-developed CPOE systems
that are well-implemented and -designed may hinder nursing workflow and impact
patient safety and patient care.
4. Discussion and Conclusions
CPOE with Computer Decision Support Systems (CDSS) can help health care
providers achieve optimum care for their patients. Raebel et al. conducted a study to
assess CPOE abilities in alerting prescribers of contraindicated medications during
pregnancy. The trial included more than 11,000 women who were enrolled in urban
Kaiser Permanente Centers. The study findings were statistically significant and
indicated that CPOE with CDSS reduced potential medication errors [19].
Although CPOE systems can improve patient care, they can negatively impact
organizational culture, workflow processes, and medical errors due to poor design. In
2006, Campbell et al. identified unintended consequences that resulted from CPOE
implementation. The study made use of an expert panel by using an iterative process in
which the list of CPOE adverse consequences was divided into categories. This study
revealed that CPOE can generate new types of errors, which include, but are not limited
to, organizational culture, workflow process, and order errors that occur when users
accidentally select an item that appears beside the intended choice. This is a result of
poor design [20].
As an increasing number of nurses enter physician orders into CPOE and
administer such orders, more work is required to understand the impact of CPOE on
nursing workflow and, in particular, the impact of these changes on patient safety and
patient care. Kazemi et al. suggest that the involvement of nurses in entering data into
CPOE serves as a form of collaborative effort between physicians and nurses that may
improve patient safety and patient care [7]. This further highlights the need to study the
impact of CPOE on nurse workflow and the direct impact on patient safety.
M. Househ et al. / Patient Safety Perspectives: The Impact of CPOE on Nursing Workflow 371
References
[1] Institute of Medicine. To Err is Human. Building A Safer Health System. November 1999. Web Access
[http://www.iom.edu/~/media/Files/Report%20Files/1999/To-Err-is-
Human/To%20Err%20is%20Human%201999%20%20report%20brief.pdf]
[2] Institute of Medicine. Health IT and Patient Safety: Building Safer Systems for Better Care. November
2011. Web Access [http://www.iom.edu/Reports/2011/Health-IT-and-Patient-Safety-Building-Safer-
Systems-for-Better-Care.aspx]
[3] Ash, J. S., Sittig, D. F., Dykstra, R., Wright, A., Mcmullen, C., & Richardson, J. (2010). Identifying Best
Practices for Clinical Decision Support and Knowledge Management in the Field, 806-810.
doi:10.3233/978-1-60750-588-4-806
[4] Bates et al (2001). Reducing the Frequency of Errors in Medicine Using Information Technology, in
Journal of the American Medical Informatics Association Volume 8 Number 4 Jul / Aug 2001
[5] Jesse I. Wolfstadt I, Jerry H. Gurwitz. The Effect of Computerized Physician Order Entry with Clinical
Decision Support on the Rates of Adverse Drug Events: A Systematic Review. University of
Toronto.Canada.2008
[6] Koppel R, Metley JP, Cohen A, Abaluck B, Localia AR, Dimmel SE, Strom BL. Role of computerized
physician order entry systems in facilitating medication errors. JAMA. 2005;293, 1197-1203
[7] Kazemi A, Fors UGH, Tofighi S, Tessma M, Ellenius J Physician Order Entry Or Nurse Order Entry?
Comparison of Two Implementation Strategies for a Computerized Order Entry System Aimed at
Reducing Dosing Medication Errors. JMedInternetRes2010;12(1):e5
[8] Khajouei,R., P. C.Wierenga, et al.(2011)." clinicians satisfaction with cpoe ease of use and effect on
clinicians workflow, efficiency and medication safety."Int J Med Inform Oh H, Rizo C, Enkin M, Jadad
A (2005). What is eHealth (3): A systematic review of published definitions. JMIR, 24;7(1):e1
[9] Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safer health system. A report
of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC:
National Academy Press; 2000.
[10] Maddox P, Wakefield M, Bull J. Patient safety and the need for professional and educational change.
Nurs Outlook, 2001;49(1):8-13.
[11]Poissant L, Pereira J, Tamblyn R, et al. The impact of electronic health records on time efficiency of
physicians and nurses: a systematic review. J Am Med Inform Assoc 2005;12:505-16.
[12] Smith J, Crawford L. The link between perceived adequacy of preparation to practice, nursing error,
and perceived difficulty of entry-level practice. JONAS Healthc Law Ethics Regul 2003;5(4):100-3
[13] Keohane, C., Bane, A., Featherstone, E., Hayes, J., Woolf, S., Hurley, A., Bates, D., Gandhi, T., & Poon,
E. (2008). Quantifying Nursing workflow in medication administration. Journal of Nursing
administration, 38(1), p. 19-26.
[14] qtd in. Teasdale C. 2008. Nursing Perceptions of a Computerized Physician Order Entry System.
Northern Kentucky University. Web Access:
[http://books.google.com.sa/books?id=FgrRNzl65AIC&pg=PA19&lpg=PA19&dq=trimmons+nursing&
source=bl&ots=Xi1dXSWdr0&sig=kqnfvCasIxKGcHFGOER1wgliEIM&hl=en&redir_esc=y#v=onepa
ge&q=trimmons%20nursing&f=false]
[15] Ash, J., Gorman, P., Seshadri, V., & Hersh, W. (2004). Computerized physician order entry in U.S.
hospitals: Results of a 2002 survey. Journal of American Medical Informatics Association, 11(2), 95-99.
[16] Bleich, M. R., Hewlett, P. O., Santos, S. R., Rice, R. B., Cox, K. S., & Richmeier, S.(2003). Analysis of
the nursing workforce crisis: A call to action. American Journal of Nursing, 103(4), 66-74.
[17] Lee TT, Mills ME, Lu MH. Impact of a nursing information system on practice patterns in Taiwan
Comput Inform Nurs. 2008 Jul-Aug;26(4):207-14.
[18] Mohamed N., and Gabr H., (2010): Quality improvement techniques to control medication errors in
surgical intensive care units at emergency hospital. Journal of Medicine and Biomedical Sciences,(5): 24
-35.
[19] Raebel MA, Carrlo NM, Kelleher JA, Chester EA, Berga S, Magid DJ. Randomized trial to improve
prescribing safety during pregnancy. JAMIA.
[20] Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related
to computerized provider order entry. JAMIA. 2006;13, 547-556.
Healthcare Modeling and Simulation
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-375
Developing a Multivariate Electronic

Medical Record Integration Model for
Primary Health Care
Francis LAU a,1, Morgan PRICE b and Mary LESPERANCEc
a
School of Health Information Science, University of Victoria, Canada
b
Department of Family Practice, University of British Columbia, Canada
c
Department of Mathematics and Statistics, University of British Columbia, Canada
Abstract. This paper describes the development of a multivariate electronic

medical record (EMR) integration model for the primary health care setting. Our
working hypothesis is that an integrated EMR is associated with high quality
primary health care. Our assumption is that EMR integration should be viewed as a
form of complex intervention with multiple interacting components that can impact
the quality of care. Depending on how well the EMR is integrated in the practice
setting, one can expect a corresponding change in the quality of care as measured
through a set of primary health care quality indicators. To test the face validity of
this model, a Dephi study is being planned where health care providers and
information technology professionals involved with EMR adoption are polled for
their feedback. This model has the potential to quantify and explain the factors that
influence successful EMR integration to improve primary health care.
Keywords. complex interventions, electronic medical record, primary health care
Introduction
Despite increased adoption of the electronic medical record (EMR) in primary health
care reported in recent years, its impact on provider performance and patient outcomes
has been mixed [1,2]. For instance, a systematic review by Lau et al. [3] of 43 EMR
evaluation studies has shown only 51.2% had positive impact, while 30.2% had no
effect. Given the recent attention to transform primary health care in Canada and the
great expectations on EMR as the enabler [4], the lack of positive evidence to date is
cause for concern. That said there are successful primary health care practices that have
shown improved care thru innovative use of their EMR. Examples are the use of
individualized decision support and reminder tools by Holbrook et al [5] to improve
diabetes care in a Canadian community, and an EMR supported quality improvement
project by Nemeth et al [6] in 99 primary care practices in the United States. Both
studies had multiple interventions in which EMR was one component. In this paper we
present a multivariate EMR integration model that is used to define, quantify and
explain factors that may affect the successful integration of EMR systems to improve

1
Corresponding Author;: Francis Lau, School of Health Information Science, University of Victoria,
P.O. Box 3050 STN CSC, Victoria BC, Canada V8W3P5
376 F. Lau et al. / Developing a Multivariate EMR Integration Model for Primary Health Care
the quality of care in the primary health care setting.
1. Purpose
This study is to develop a multivariate electronic medical record (EMR) integration

model for the primary health care setting. Our working hypothesis is that an integrated
EMR is associated with high quality primary health care. We postulate that an
integrated EMR is one where its features are well adopted and used over time in the
practice to support improvement of the care provided. Our assumption is that EMR
integration can be viewed as a form of complex intervention with multiple interacting
components that impact the quality of care [7]. These components include a set of
contextual factors that influence EMR integration in a practice. Depending on how well
the EMR is integrated into the practice, one can expect a corresponding change in the
quality of primary health care (PHC) as measured thru a set of PHC quality indicators.
2. Methods
To develop this multivariate model, we defined the parameters, provided evidence to

justify the choices where feasible, and described means by which this model could be
tested. The model is built on the EMR adoption framework by Price et al [8] since it
provides a means of rating the level of EMR adoption within a PHC practice. We also
selected relevant PHC indicators from the Canadian Institute for Health Information
(CIHI) since they are specific to the Canadian primary health care setting [9]. To
establish the evidence base for the model parameters, we searched for published EMR
literature and synthesized key concepts from relevant articles through an iterative
consensus process among the three co-authors. Where feasible we used articles that
focused on Canadian primary health care practices within the last 10 years when much
of the PHC transformation occurred in Canada. In instances where the evidence for a
model parameter is not conclusive we arrived at a consensus from our best judgment
and what would be feasible in terms of the sample size needed for model testing. We
acknowledged this model was tentative in nature and required validation by others.
3. Results
3.1. Model Parameters
First we define EMR adoption in terms of an adoption score based on the EMR
adoption framework. Second we define EMR integration in terms of the EMR adoption
score and the contextual factors that influence how well the EMR can be integrated.
Third we define PHC quality index in terms of process and outcome quality indicators,
which are derived from the PHC indicators from CIHI [9] and the Summary of QUality
InDex (SQUID) algorithm by Nietert et al [10]. Fourth we model the relationship
between EMR integration and PHC quality index. These definitions are as follows:

F. Lau et al. / Developing a Multivariate EMR Integration Model for Primary Health Care 377
EMR adoption score from 0 to 5, defined as

‫ ݊݋݅ݐ݌݋ܴ݀ܽܯܧ‬ൌ ‫ ݊݋݅ݐܽ݉ݎ݋݂݄݊݅ݐ݈ܽ݁ܪ‬൅ ‫ ݏ݊݋݅ݐܽܿ݅݀݁ܯ‬൅ ‫ ݏܾܽܮ‬൅
‫ ݏܿ݅ݐݏ݋݊݃ܽ݅ܦ‬൅ ܴ݂݁݁‫ ݏ݈ܽݎݎ‬൅ ‫ ݐݎ݋݌݌ݑݏ݊݋݅ݏ݅ܿ݁ܦ‬൅
‫ ݊݋݅ݐܽܿ݅݊ݑ݉݉݋ܿܿ݅݊݋ݎݐ݈ܿ݁ܧ‬൅ ܲܽ‫ ݐݎ݋݌݌ݑݏݐ݊݁݅ݐ‬൅ ‫ ݊݋݅ݐܽݎݐݏ݅݊݅݉݀ܣ‬൅
ܲ‫݃݊݅ݐݎ݋݌݁ݎ݁ܿ݅ݐܿܽݎ‬ (1)
EMR integration = EMR adoption as a function of the contextual variables, as
‫ ݊݋݅ݐܽݎ݃݁ݐܴ݊݅ܯܧ‬ൌ ‫ ݁ݎ݋ܿݏ݊݋݅ݐ݌݋ܴ݀ܽܯܧ‬൅ ሺ‫ ݐܿݑ݀݋ݎ݌ܴܯܧ‬൅
‫ ݊݋݅ݐܽݎݑ݂݃݅݊݋ܴܿܯܧ‬൅ ‫ ݕݐ݈݅ܽݑݍܽݐܴܽ݀ܯܧ‬൅
ܶ݅݉݁‫ ݊݋݅ݐܽݐ݈݊݁݉݁݌݉݅݁ܿ݊݅ݏ‬൅ ܲ‫ ݁݌ݕݐݎ݁݀݅ݒ݋ݎ‬൅
ܷ‫ ݈݁݃݀݁ݓ݋ܴ݊݇ܯܧݎ݋݅ݎ݌ݎ݁ݏ‬൅ ܷ‫ ݊݋݅ݐ݂ܿܽݏ݅ݐܽݏݎ݁ݏ‬൅
ܲ‫ ݊݋݅ݐܽݖ݅݊ܽ݃ݎ݋݁ܿ݅ݐܿܽݎ‬൅ ܲ‫ ݁ݖ݅ݏ݁ܿ݅ݐܿܽݎ‬൅ ܲ‫ ݐ݊݁݉݁ݒ݋ݎ݌݉݅݁ܿ݅ݐܿܽݎ‬൅
‫݁ݒ݅ݐ݈݊݁ܿ݊݅ܽ݅ܿ݊ܽ݊݅ܨ‬ሻ (2)
PHC quality index = Summary PHC quality index as a proportion from 0 to 1, as the
average of the patient level indicator targets, derived from below: (3)
E = Number of quality process and outcome indicators the patient is eligible
M= Number of quality indicator targets that the eligible patient has met; for process
quality indicators the target has been met if the process has been performed within
the specified time period; for outcome quality indicators the target has been met if
the outcome has achieved the recommended guideline
Patient level indicator target = M/E
Association between EMR integration (2) and PHC quality index (3) to be determined
by modeling PHC quality index as a function of EMR integration (4)
The parameters listed in the above equations are further defined in Table 1.
3.2. Evidence for Model Parameters
For each of the parameters in the model we searched for relevant literature to find out
how others have defined them. We have included three such parameters to illustrate the
source of evidence used. For example, our Practice organization definition is based on
five published Canadian PHC organizational performance studies [4,10-13]. Our
Practice size definition is based on five published studies from Canada, United States
and Australia [1,14,16,17]. Table 2 shows the suggested PHC indicators for our model,
which are drawn from PHC content standard development work by CIHI [9].
3.3. Model Testing
For testing, we need to collect data for the parameters identified in the model. We then
conduct statistical analysis to test the model. These steps are described below:
Data Collection – We need to collect data for the parameters from multiple sources
through environment scan, survey and data extraction methods. The variables,
sources and tools are listed below. The tools are available via the UVic eHealth
Observatory website (http://ehealth.uvic.ca/resources/tools/tools.php).
a) EMR adoption score – from care providers and support staff through face-to- face
interviews using the EMR adoption survey tool
b) EMR products and configurations – from EMR vendor, jurisdictional and Infoway
websites and published reports using the environment scan tool
c) User prior EMR knowledge and user satisfaction – from care providers and support
staff through face-to-face interviews using the user assessment tool
d) EMR product, configuration, time since implementation, provider type, size,

organization, practice improvement and financial incentive – from interviews with
lead provider and support staff using the practice characteristics survey
e) EMR data quality – from anonymized EMR data extract or query report provided
by the practice analyzed using the EMR data quality assessment tool
f) PHC quality index – from anonymized EMR data extract or query report from the
practice analyzed using the PHC quality indicator assessment tool
Table 1. Definition of the parameters in the multivariate EMR integration model.

Legends: EMR-electronic medical record; ASP-application service provider; PHC-primary health care; CIHI-
Canadian Institute for Health Information; SQUID-Summary of QUality InDex.
Parameter Definition
EMR adoption Level of EMR adoption achieved in a practice measured by EMR adoption score
EMR adoption score User reported adoption level from 0 to 5 (0 as paper-based, 5 as fully integrated)
How well EMR has been adopted in terms of adoption score and contextual
EMR integration
factors
EMR product Number of EMR software product on the market
EMR configuration Type of EMR instance installed in practice, e.g. local, single and multiple ASP
Overall rating of data quality based on sensitivity/specificity of selected data
EMR data quality elements in terms of accuracy and completeness; high 80%, medium 60%,
low <60%
Time since
Length of time since EMR went live, expressed as <1, 1-2, 3-4, 5 years
implementation
Type of primary health care providers in practice, defined as solo physician
Provider type practice, group/multispecialty physician practice, and interprofessional PHC
team
User prior EMR Average length of time that users have used EMR in the past, expressed as none,
experience or yes with <1, 1-2, 3-4, 5 years
Overall satisfaction rating from users toward the EMR, expressed in a Likert
User satisfaction
scale from 1 to 5, with 1 as highly dissatisfied and 5 as highly satisfied
Type of PHC organization/model in place, defined as fee-for-service, alternate
Practice organization
payment plan, capitation and blended payments
Practice size Number of PHC providers in the practice, defined as 1-3, 3-4, 5 providers
Practice Training and support activities in place that enhance practice management and
improvement clinical care, defined as none, EMR related, non-EMR related, and both activities
Use of government funding to improve practice, defined as EMR support,
Finance incentives
practice support, and both
The quality process and outcome indicators defined by CIHI summarized as
Indicator targets
a PHC quality index, calculated using the SQUID algorithm [x]
The subset of PHC quality indicators that relate to care processes performed
Quality process
within the
indicators
specified time period as part of the recommended guidelines
The subset of PHC quality indicators that relate to care outcomes met as part of
Quality outcome
the
indicators
recommended guidelines
Adaptation of the SQUID algorithm based on a subset of PHC indicators
PHC quality index
published by CIHI, expressed as the average of indicator targets
Data Analysis - We need to conduct univariable testing of each pair of parameters,

followed by multivariable testing while controlling for the parameter. We then use this
multivariable model for prediction
a) Univariable testing – Determine relationship between PHC quality index and each
variable separately, e.g., PHC quality index and EMR adoption score, PHC quality
index and EMR product, etc.
b) Multivariable testing – Determine relationship between PHC quality index and
EMR adoption score after controlling for the other parameters, i.e. Eq.5.
ܲ‫ ݔ݁݀݊݅ݕݐ݈݅ܽݑݍܥܪ‬ൌ
‫ ݁ݎ݋ܿݏ݊݋݅ݐ݌݋݀ܣ‬൅ ሺ‫ ݐܿݑ݀݋ݎ݌ܴܯܧ‬൅ ‫ ݊݋݅ݐܽݎݑ݂݃݅݊݋ܴܿܯܧ‬൅
‫ ݕݐ݈݅ܽݑݍܽݐܴܽ݀ܯܧ‬൅ ܶ݅݉݁‫ ݊݋݅ݐܽݐ݈݊݁݉݁݌݉݅݁ܿ݊݅ݏ‬൅ ܲ‫ ݁݌ݕݐݎ݁݀݅ݒ݋ݎ‬൅
ܷ‫ ݈݁݃݀݁ݓ݋ܴ݊݇ܯܧݎ݋݅ݎ݌ݎ݁ݏ‬൅ ܷ‫ ݊݋݅ݐ݂ܿܽݏ݅ݐܽݏݎ݁ݏ‬൅ ܲ‫ ݊݋݅ݐܽݖ݅݊ܽ݃ݎ݋݁ܿ݅ݐܿܽݎ‬൅
ܲ‫ ݁ݖ݅ݏ݁ܿ݅ݐܿܽݎ‬൅ ܲ‫ ݐ݊݁݉݁ݒ݋ݎ݌݉݅݁ܿ݅ݐܿܽݎ‬൅ ‫݁ݒ݅ݐ݈݊݁ܿ݊݅ܽ݅ܿ݊ܽ݊݅ܨ‬ሻ (5)
c) Prediction – Use the multivariable model from (b) to do prediction
4. Next Steps
HIMSS Analytics has recently released an US Ambulatory EMR Adoption Model that
measures adoption stages but it is focused on EMR functionality without any linkage to
contexts [18]. Conversely, two recent PHC evaluation studies from Ontario and Quebec
have identified many factors that influenced PHC performance but neither examined
their relationship with EMR adoption in detail [19,20].
Table 2. Example PHC indicators from CIHI.
Legends: PHC# indicator reference number; + reviewed by author.[MP] as feasible to extract from EMR, *not part of
prioritized CIHI list but reviewed as feasible
PHC Quality indicator Eligibility criteria Target PHC# Feasible+

Process Quality Indicators
Influenza immunization Age 65 Within last year 41
Congenital hip displacement,
eye and All By 3 years of age 43
hearing problem screening
Primary childhood
All By 7 years of age 44
immunizations
Colon cancer screening Within last two
Age 50 48
(Hemoccult) years
Mammography and breast Within last two
Ages 50 to 69 49
exam years
Within last three
PAP smear Ages 18 to 69 50
years
Lipid profile screening (full Within last two
Women, age 55 52
fasting)* years
Lipid profile screening (full Within last two
Men, age 40 53
fasting)* years
Within last two
Blood pressure measurement Age 18 54
years
Fasting blood sugar
Lipid profile screening (full
Age 18 with coronary
fasting) Blood pressure Within last year 55
artery disease
measurement
Obesity/overweight screening
Ages 18 to 75 with
Within last two
Eye exam diabetes 58
years
mellitus
Active ACE or ARB
ACE inhibitors or ARBs* Congestive heart failure 60
treatment
Acute myocardial Active beta blocker
Beta blocker 62
infarction treatment
Outcome Quality Indicators
Blood pressure control Age 18 with <140/90 mmHg
hypertension duration 1 40
year
We presented this multivariate model as part of a panel discussion on complex

eHealth interventions at the 2012 Canadian Association of Health Services and
Policy Research Conference in Montreal. The model was well received as a potential
way to quantify and explain the factors that influence successful EMR integration to
improve primary health care. For next steps we are planning a Delphi study in fall
2012 to solicit feedback on the face validity of this model from stakeholders involved
with EMR adoption in Canada and elsewhere. The Dephi study will consist of two
rounds of feedback from stakeholders starting with the proposed model in this paper
with revisions to improve the model after each round. The stakeholders will include
jurisdictional representatives who are part of the Canada Health Infoway benefits
evaluation forum, members of the UVic eHealth Observatory virtual community on
eHealth benefits evaluation, and researchers and practitioners engaged in primary
health care and EMR research. Then we need to test the model with the steps that have
been outlined in section 3.3 of this paper.
References
[1] C. Schoen, R. Osborn, M.M. Doty, D. Squires, J. Peugh, S. Applebaum. A survey of primary care
physicians in eleven countries, 2009: perspectives on care, costs and experiences. Health Affairs 28
(2009), w1171-w1183.
[2] CMA. National Physician Survey 2010. [cited March 25, 2012]. Available from:
http://www.nationalphysiciansurvey.ca..
[3] F. Lau, M. Price, J. Boyd, C. Partridge, H. Bell, R. Raworth. Impact of electronic medical record on
physician practice in office settings – systematic review. BMC Med Inform Decision Making 12
(2012):1-10.
[4] B. Hutchison, J. Levesque, E. Strumpf , N. Coyle. Primary health care in Canada: Systems in motion.
Milbank Quarterly 89 (2011), 256-88.
[5] A. Holbrook, L. Thabane, K. Keshavjee, L. Dolovich, B. Bernstein, D. Chan, S. Troyan, G. Foster, H.
Gerstein, for the COMPETE II Investigators. Individualized electronic decision support and reminders
to improve diabetes care in the community: COMPETE II randomized trial. CMAJ 181 (2009), 37-44.
[6] L.S. Nemeth, A.M. Wessell, R.G. Jenkins, P.J. Nietert, H.A. Liszka, S.M. Ornstein. Strategies to
accelerate translation of research into primary care within practices using electronic medical records.
Journal of Nurs Care Qual 22 (2008), 343-9.
[7] I. Shcherbatykh, A. Holbrook, L. Thabane, L. Dolovich. Methodologic issues in health informatics trials:
The complexities of complex interventions. JAMIA 15 (2008), 575-80.
[8] M. Price, F. Lau, J. Lai. Measuring EMR adoption: A framework and case study. ElectronicHealthcare
10 (2011), e25-e30.
[9] Canadian Institute for Health Information. Draft Pan-Canadian Primary Health Care Electronic
Medical Record Content Standard, Version 2.0 – Business View. Ottawa, Ontario, 2011.
[10] S. Dahrouge, W. Hogg, G. Russell, R. Geneau, E. Kristjansson, L. Muldoon, S. Johnston. The
comparison of models of primary care in Ontario (COMP-PC) study: methodology of a multi-faceted
cross-sectional practice-based study. Open Medicine 3 (2009), 149-64.
[11] J.L. Haggerty, R. Pineault, M.D. Beaulieu, Y. Brunelle, J. Gauthier, F. Goulet, J. Rodrigue. Practice
features associated with patient-reported accessibility, continuity and coordination of primary health
care. Annals of Family Medicine 6 (2008), 116-23.
[12] J.F. Levesque, R. Pineault, P. Tousignant, A. Couture, R. B. De Silva, M. Breton. Assessing the
evolution of primary healthcare organizations and their performance (2005-2010) in two regions of
Quebec province: Montreal and Montergie. BMC Family Practice 11 (2010), 95-108.
[13] G.M. Russell, S. Dabrouge, W. Hogg, R. Geneau, L. Muldoon, M. Tuna. Managing chronic disease in
Ontario primary care: The impact of organizational factors. Annals of Family Medicine 7 (2009),309-
18.
[14] C.M. DesRoches, E.G. Campbell, S.R. Rao, et al. Electronic health records in ambulatory care- A
national survey of physicians. New England Journal of Medicine 1 (2008), 50-60.
[15] D. Gans, J. Kralewski, T. Hammons, B. Dowd. Medical groups’ adoption of electronic health records
and information systems. Health Affairs 5 (2005), 1323-33.
[16] R. MacGregor, P. Hyland, C. Harvie, B.C. Lee, A. Dalley, S. Ramu. Benefits of ICT adoption and use
in regional general medical practices: A pilot study. Health Information Management Journal 35
(2006), 23-35.
[17] S.R. Simon, R. Kaushal, P.D. Cleary, et al. Correlates of electronic health record adoption in office
practices: A statewide survey. Journal of American Medical Informatics Association 14 (2009), 110-7.
[18] HIMSS Analytics. Health Information Management Systems Society - Ambulatory EMR Adoption
Model. [cited October 24, 2012]. Available from:
http://www.himssanalytics.org/emram/AEMRAM.aspx .
[19] E. Strumpf, J.F. Levesque, N. Coyle, B. Hutchison, M. Banes, R.J. Wedel. Innovative and diverse
strategies toward primary health care reform: Lessons learned from the Canadian experience. Journal of
American Board Family Medicine 25 (2012), S27-33.
[20] R. Pineault, P. Tousignant, D. Roberge, P. Lambarche, D. Reinharz, D. Larouche, G. Beaulne, D.
Lesage. Research Collective on the Organization of Primary Care Services in Quebec – Detailed
Report. [cited October 24, 2012]. Available from:
http://www.greas.ca/english/publication/pdfreport/detailed_report_english.pdf.
IOS Press, 2013
Towards Assessing The Socio-Economic

Impact of VPH Models
Rainer THIELa, Karl A. STROETMANNa1, Marco VICECONTIb
a
empirica Communication and Technology Research, Bonn, Germany; b Istituto
OrtopedicoRizzoli, Bologna, Italy
Abstract: Biocomputational modeling as developed under the European Virtual

Physiological Human (VPH) and the US NIH program on “Predictive Multiscale
Models of the Physiome in Health and Disease” is the area of ICT to most likely
revolutionize the practice of medicine. The VPH is a framework of methods and
technologies that, once fully established, will make possible the investigation of
the human body as a whole. There is considerable demand for measurable
evidences that such complex technology is actually worth the cost. How is it
possible to quantitatively assess ex ante such technologies in terms of safety,
efficacy, and socio-economic impact on our health systems? With the example of
osteoporosis management, a socioeconomic assessment framework is presented
that captures how the transformation of clinical guidelines can be evaluated.
Applied to the clinical decision support system under development in the European
Osteoporotic Virtual Physiological Human Project, a consequent cost-benefit
analysis delivers promising results, both methodologically and substantially.
Keywords: Virtual Physiological Human
The revised paper was not available at the time of publication.
1
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-383
The Use of Discrete-Event Simulation

Modeling to Compare Handwritten and
Electronic Prescribing Systems
Ahmad GHANYa,, Karim VASSANJIb, Craig KUZIEMSKYc, Karim KESHAVJEEb1
a
McMaster University, Hamilton, ON
b
InfoClin Inc, cUniversity of Ottawa
Abstract. Electronic prescribing (e-prescribing) is expected to bring many benefits

to Canadian healthcare, such as a reduction in errors and adverse drug reactions.
As there currently is no functioning e-prescribing system in Canada that is
completely electronic, we are unable to evaluate the performance of a live system.
An alternative approach is to use simulation modeling for evaluation. We
developed two discrete-event simulation models, one of the current handwritten
prescribing system and one of a proposed e-prescribing system, to compare the
performance of these two systems. We were able to compare the number of
processes in each model, workflow efficiency, and the distribution of patients or
prescriptions. Although we were able to compare these models to each other, using
discrete-event simulation software was challenging. We were limited in the
number of variables we could measure. We discovered non-linear processes and
feedback loops in both models that could not be adequately represented using
discrete-event simulation software. Finally, interactions between entities in both
models could not be modeled using this type of software. We have come to the
conclusion that a more appropriate approach to modeling both the handwritten and
electronic prescribing systems would be to use a complex adaptive systems
approach using agent-based modeling or systems-based modeling.
Keywords. Discrete-event simulation model, Complex Adaptive Systems, Agent-

based Model, Electronic Prescribing, Handwritten Prescribing.
Introduction
There are many benefits that are expected from an electronic prescribing (e-
prescribing) system for Canadian healthcare. These benefits include greater efficiency
and a possible reduction in medication errors and adverse drug events (ADEs) [1].
Canada Health Infoway (Infoway) has found some benefits from e-prescribing
implementations in Canada, such as a reduction in ADEs, greater patient compliance,
and greater productivity from providers [2]. However, since there currently is no
functioning e-prescribing system in Canada that is completely electronic, the ability of
the system to function at both a provincial and national level is not clear. As we are
unable to evaluate the performance of a live e-prescribing system, a reasonable
alternative would be to consider the use of simulation modeling for evaluation.
Simulation modeling can be used to model a proposed system, making it possible to
1
384 A. Ghany et al. / The Use of Discrete-Event Simulation Modeling
design and measure the performance of a system before it is actually built [3].
Simulation modeling of e-prescribing may be of benefit in identifying transition issues,
key productivity bottlenecks and patient safety concerns that may disrupt or delay
clinical adoption. Simulation modeling might also be able to demonstrate the potential
impact that e-prescribing would have to Canadian healthcare. Others have developed
simulation models of e-prescribing systems or closely related systems, such as
computerized provider order entry systems (CPOE). Bell et al., with the RAND
Corporation, developed and used a simulation model to predict how the
implementation of different aspects of an e-prescribing system would affect variables
such as process times and workflow efficiency [4]. Anderson et al. developed a
simulation model to test how changing certain aspects of a CPOE system would impact
variables such as medication errors, ADEs, and costs [5].
We set out to develop two discrete-event simulation models, one of the
handwritten prescribing system currently used in the out-patient setting in Canada and
one of a proposed electronic prescribing system for Canada, and then compare the
performance of these systems to each other.
1. Methods
1.1. Building the Simulation Models
The modeling software Arena was selected for building our discrete-event simulation
models. Arena has been selected by other researchers to develop e-prescribing
simulation models, such as Bell, who were able to successfully develop their
simulation models using this software [4]. Drug prescribing and dispensing processes
for both handwritten prescribing and electronic prescribing were first mapped out in
Arena using a standard modeling notation, Business Process Modeling Notation
(BPMN). Both models included processes involved in drug prescribing and dispensing.
The handwritten prescribing simulation model was built based on the clinical
experiences of a member of the research team (KK). The electronic prescribing
simulation model was inspired by a proposed e-prescribing system for Canada
developed by Canada Health Infoway [6]. The models were reviewed by a pharmacist
and two data modelers for completeness and accuracy.
Data values for both the handwritten prescribing and e-prescribing simulation
models were estimated based on the clinical experiences of a member of the research
team (KK). Data values were estimated for process times, number of prescriptions, and
decision points. Minimum, maximum, and average values for each process were also
estimated.
1.2. Testing the Simulation Models
Once the simulation models were built and populated with data, test scenarios were
developed to test the models. Test scenarios were designed to test the completeness of
the models and to check for any bugs present in the models. Sensitivity analysis was
also performed.
A. Ghany et al. / The Use of Discrete-Event Simulation Modeling 385
1.3. Comparing the Simulation Models
After the models were tested, the handwritten prescribing simulation model was
compared to the e-prescribing simulation model. This was done by running each
simulation model and then comparing the output reports for the variables of interest.
2. Results
Both the handwritten prescribing simulation model and the e-prescribing simulation
model were successfully built in Arena and were determined to be complete and
accurate by the pharmacist and data modelers who reviewed the models. Figure 1
displays a condensed version of the handwritten prescribing simulation model. Figure
2 displays a condensed version of the e-prescribing simulation model.
Running test scenarios through each model revealed that the models were running
properly and were complete. Small bugs in both models were corrected. We were
able to compare the handwritten prescribing simulation model to the e-prescribing
simulation model on several levels. We compared the number of processes involved in
each model and were able to determine that the e-prescribing model clearly contained
more processes than the handwritten prescribing model. The handwritten prescribing
model contained 34 processes, while the e-prescribing model contained 60 processes.
We found that we could compare workflow efficiency between the two models by
comparing similar processes in both models for average process times. Finally, we
were able to compare how the initial input of patients or prescriptions were distributed
throughout the handwritten system and the e-prescribing system once the simulations
were run. The results of the comparison between the two simulation models are
displayed in Table 1.
Figure1. Condensed version of handwritten prescribing simulation model.

386 A. Ghany et al. / The Use of Discrete-Event Simulation Modeling
Figure2. Condensed version of electronic prescribing simulation model.
Table 1. Comparison of Process Number and Times for Each Model

Handwritten Model E-Prescribing Model
Total Processes 34 60
Time to Review Chart Paper Chart –1.58 Minutes E-Chart – 0.84 Minutes
Select Medication 0.23 Minutes 0.22 Minutes
Generate Prescription Paper – 2.49 Minutes Electronic – 0.44 Minutes
Pharmacist Manages Issues From 0.97 Minutes 0.33 Minutes
Drug Check
Callbacks to Physicians 21.87 Minutes 28.86 Minutes
3. Discussion
Although we were able to build detailed simulation models of the handwritten

prescribing system and the electronic prescribing system in Arena, using a discrete-
event simulation modeling software to model these systems proved to be challenging.
We were only able to measure workflow efficiency, using process times, in Arena. We
could not also measure other important outcomes such as patient safety, which would
be measured using the number of medication errors or adverse drug events. We
initially viewed both the handwritten system and the electronic prescribing system as
linear processes; however, when modeling both processes we came to the realization
that there are processes in both systems that are non-linear. An example is a
prescription that is not clearly written, which affects several other processes including
the pharmacist reviewing the prescription, the pharmacist calling back the physician,
the patient receiving the wrong drug, and then ultimately having to see the physician
again due to issues with the medication. While it is possible to model non-linear
processes like this using discrete-event simulation software like Arena, discrete-event
simulation software is not the ideal choice for modeling non-linear processes [7].
Furthermore, we found that there are several feedback loops in both systems, such as
call backs to physicians from pharmacists seeking clarification and the refill process.
Again, while feedback loops can be modeled using discrete-event simulation software,
A. Ghany et al. / The Use of Discrete-Event Simulation Modeling 387
it is not the ideal choice for modeling feedback loops [7]. Finally, we found that while
Arena was suitable for modeling both systems with a sufficient level of detail, it was
not suitable for modeling the interactions between entities in both models.
In future, we plan to use a complex adaptive system approach to simulating the
handwritten prescribing system and electronic prescribing system by using agent-based
or systems-based simulations.
References
[1] The Institute of Medicine. Preventing Medication Errors: Quality Chasm Series. Washington: National
Academy of Sciences; 2006.
[2] Canada Health Infoway. National impacts of generation 2 drug information systems. Technical Report.
Deloitte; 2010.
[3] Mo, JPT & Mahmoudi, M. Optimisation and simulation of high speed production system. Journal of
Achievements in Materials and Manufacturing Engineering. 2008;31(2): 794-802.
[4] Bell D, Schueth A, Crosson J, Guinan J, Wu S, Pevnick J, Wang CJ, Neuman S, Patel M, Park H,
Tysinger B, Malakar CL, Schoeff D, Bradley M, and Newberry S. Pilot testing of electronic prescribing
standards. (Report). Funded by the Agency for Healthcare Research and Quality. 2007.
[5] Anderson J, Jay S, Anderson M, and Hunt T. Evaluating the capability of information technology to
prevent adverse drug events: a computer simulation approach. Journal of the American Medical
Informatics Association. 2002; 9(5): 479-490.
[6] Canada Health Infoway. HL7 v3 pan-Canadian messaging standards: Implementation guide volume 8 –
Pharmacy. 2009.
[7] Sweetser A. A comparison of system dynamics and discrete event simulation. In: Proceedings of 17th
International Conference of the System Dynamics Society and 5th Australian & New Zealand Systems
Conference, ed. Cavana RY, Vennix JAM, Rouette EAJA, Stevenson-Wright M and Candlish J,
Wellington, New Zealand: System Dynamics Society; 1999.
IOS Press, 2013
doi:10.3233/978-1-61499-203-5-388
A Model of Collaborative Agency and

Common Ground
Craig E. KUZIEMSKYa,1, Janet Alexandra CORNETTa
a
Telfer School of Management, Ottawa, Ontario, Canada
Abstract. As more healthcare delivery is provided via collaborative means there is

a need to understand how to design information and communication technologies
(ICTs) to support collaboration. Existing research has largely focused on
individual aspects of ICT usage and not how they can support the coordination of
collaborative activities. In order to understand how we can design ICTs to support
collaboration we need to understand how agents, technologies, information and
processes integrate while providing collaborative care delivery. Co-agency and
common ground have both provided insight about the integration of different
entities as part of collaboration practices. However there is still a lack of
understanding about how to coordinate the integration of agents, processes and
technologies to support collaboration. This paper combines co-agency and
common ground to develop a model of collaborative agency and specific
categories of common ground to facilitate its coordination.
Keywords. Common ground, collaborative agency, coordination
Introduction
Collaborative care delivery is challenging due to the fact it requires the integration of
multiple people, processes and information and communication technologies. Failure to
properly integrate these different entities results in communication issues, medical
errors, and poor information management [1].
Information and communication technologies (ICTs) can facilitate collaborative
care delivery by supporting different types of interactions [2]. To date there has been
limited research on ICT design to support collaboration with the more prominent focus
being on individual ICT usage [3]. However before we can designing ICTs to support
collaboration we need to fully understand the various interactions that exist as part of
collaboration.
Co-agency, described as humans, technology and processes working together in
the pursuit of jointly held goals [4], has been used to understand collaborative practices.
A shortcoming with existing studies is that they have not studied the multiple ways in
which collaborative agents, processes and technology integrate. It is these dynamic
integrative processes that we need to focus on because that is where problems most
often occur. Coiera’s interaction design theory states that we need to look beyond
1
Corresponding Author. Dr. Craig Kuziemsky, Telfer School of Management, University of Ottawa,
55 Laurier Avenue East,Ottawa, ON K1N 6N5, e-mail: [email protected]
C.E. Kuziemsky and J.A. Cornett / A Model of Collaborative Agency and Common Ground 389
individual agents or technologies to understand group interactions across multiple

agents and technologies [5]. A key part of group interactions is common ground, the
knowledge two or more agents need to share in order to collaborate. If agents do not
have common ground about roles or processes it can impact how a task is conducted
leading to adverse events or communication issues [6].
We suggest that co-agency and common ground can be combined into
collaborative agency to refer to how collaborative agents (i.e. human and
technological) and information and processes are integrated as part of collaborative
care delivery. Collaborative agency considers the multiple entities that are integrated
but also the process of integration.
This paper introduces the concept of collaborative agency and the role common
ground plays in its development. Specifically we develop a model of collaborative
agency that focuses on the integrative aspects of collaborative care delivery. We then
identify categories of common ground that impact the development of collaborative
agency.
1. Study Design
The authors studied collaborative care delivery in two different areas of medicine. First
was a study of palliative sedation on an inpatient palliative care unit. Second was a
study across three hospitals of a surgical information management system (SIMS) that
is used in all areas of the perioperative process. Both studies had specific objectives
related to studying aspects of collaboration. The findings from both studies were
combined and reanalyzed to provide insight on collaborative agency and common
ground
1.1. Data Sources
The palliative care study involved 76 hours of non-participant observation. The

perioperative study involved approximately 90 hours of non-participant observations.
Both studies also conducted follow up interviews to probe or gain insight into
observation details.
1.2. Methods
Qualitative content analysis [7] was used to re-analyze the data from both studies. The
analysis was not done with any preconceived themes other than wanting to identify
components of collaborative agency and common ground.
2. Results
Our results are presented in two sections. First we develop a model of collaborative
agency. Second we identify four types of common ground and describe how they
influence the development of collaborative agency.
390 C.E. Kuziemsky and J.A. Cornett / A Model of Collaborative Agency and Common Ground
2.1. Model of Collaborative Agency
Fig.1 shows our model of collaborative agency. There are two main aspects to the
model. First are the collaborative factors and second is the coordination of the factors
as part of collaborative care delivery.
Agents
Information
Collaborative Action
Event Processes and
workflows
Completed actions
facilitate subsequent
events If ineffective
Channels Try another
iteration
Figure 1. Model of collaborative agency (dotted oval is the collaborative cycle with four collaborative
factors within it)
Collaborative agency is initiated with an event. The event then facilitates the
integration of four collaborative factors: agents, information, processes and workflows,
and channels. The integration of the different factors leads to an action. There are two
aspects to the model that illustrates the complexity of collaborative agency. First is that
collaborative agency is a dynamic process with multiple potential iterations. Although
a collaborative event will use each of the four collaborative factors in the collaborative
cycle (dotted oval in fig.1) the factors can be integrated in many different ways. There
are also different variations of each factor. For example, channels that are used to
communicate can include paper, electronic or oral means, while agents can be intra or
inter disciplinary. Second, collaborative agency is not a one-time action but rather is
an ongoing cycle of collaborative events and different integrations of the collaborative
factors. A collaborative event may cycle back to form a new iteration of the
collaborative factors if the action was ineffective and did not achieve the desired
outcome. For example, if a communication action did not reach the desired recipient a
new channel or different agent may be integrated into the event. Successful
collaborative events will drive subsequent events as part of the continuum of care.
2.2. Common Ground and the Development of Collaborative Agency
Common ground represents the shared meaning that the collaborative factors need to
have in order for collaboration cycle to work effectively and facilitates the coordinative
aspect of collaborative agency. If there is weak common ground amongst any of the
collaborative factors it will adversely impact the development of collaborative agency.
C.E. Kuziemsky and J.A. Cornett / A Model of Collaborative Agency and Common Ground 391
We identified four specific types of common ground: technology, information,

process and personnel. The four types map to the four collaborative factors from
section 2.1 (technology common ground incorporates channels and personnel common
ground incorporates agents from section 2.1).
Technology Common Ground – If agents do not have common understanding
about how to use technology they may use it in ineffective ways or develop
workarounds. In the perioperative study an anesthetist was entering all her data in the
Surgical Information Management System (SIMS) in real time and commented that the
technology made it difficult to administer medications and monitor the patient.
However, the system was designed to allow a clinician to tag certain events as a
bookmark to enable them to go back and enter data during a quieter time in the surgery.
The real time data entry that impacted patient care was not necessary but the anesthetist
lacked common ground about how SIMS should be used.
Information Common Ground – Information is what is disseminated through
channels as part of collaboration and lack of common ground about information will
impact the agent’s ability to collaborate effectively. One of the biggest barriers to
information common ground was informal communication practices. In the palliative
care study informal communication practices, both text-based and oral, were common.
Informal communication may lead to inconsistent messages and may not be
disseminated to all agents. Both of those issues prevent the development of information
common ground.
Process Common Ground – If agents do not have common knowledge about
processes and how they should be conducted it can impact integration with other
collaborative factors. For example, in the perioperative study SIMS allows users to
‘bring forward’ a previous patient assessment. The anesthetist then simply has to
update the patient assessment rather than entering in a whole new assessment. During
observations we learned that it was common for anesthetists to not know about the
‘bring forward’ process and instead do a whole new assessment. Further they would
blame SIMS for being the cause of redundant data entry. One anesthetist who knew
about the ‘bring forward’ process commented ‘That’s a process issue and not a
technology issue’.
Personnel Common Ground – An essential part of coordinating collaboration is
common understanding about what roles or tasks collaborating agents can and cannot
do. Personnel common ground looks at the different roles that agents can play, the
abilities they have, and their level of comfort at doing a task. Part of personnel common
ground is differentiating an agent’s ability to do a task with their willingness to do it.
As one physician noted, “it’s not fair to ask [nurses] to nurse the patient if they’re not
comfortable with what you’re asking them to do”.
3. Discussion
This study combined co-agency and common ground to introduce the concept of
collaborative agency. We developed a model of collaborative agency and identified
four collaborative factors that comprise agency. We have also expanded existing work
on common ground by identifying four specific types of common ground and
describing their role in the development of collaborative agency. Although the
importance of co-agency and common ground has been previously described this is the
392 C.E. Kuziemsky and J.A. Cornett / A Model of Collaborative Agency and Common Ground
first study that has combined the two and identified specific collaborative factors and
types of common ground.
The collaborative factors and common ground types can be used to design and
evaluate ICTs to support collaboration. For example, while technology is often blamed
for workflow problems, the description of technology and process common ground in
section 2.2 revealed that people do not always use systems in an appropriate way.
Although SIMS was blamed for workflow and redundant data entry issues it was
actually a lack of technology or process common ground that caused the issues.
Conducting in-situ evaluation studies, better training on system usage, and longitudinal
evaluation studies would help the formation of process and technology common ground.
Collaborative agency and common ground can also help process redesign to
support collaborative care delivery. One of the biggest barriers to common ground
development was informal communication practices or processes. Informal processes
(i.e. workarounds) make it difficult for collaborative agents to coordinate actions and
may be an ineffective or even unsafe way of doing the process. Informal information
reduces standardization which can adversely impact information exchange. Protocols
that facilitate collaborative agency and support common ground development (i.e.
formalized processes and information) should be developed as part of collaborative
care delivery.
This study has illustrated the relationship between collaborative agency and
common ground and their roles in supporting collaborative care delivery. While
collaborative agency represents the integration of agents, processes, information and
technology it is common ground that ensures that the collaborative factors are actually
interoperable with each other.
Acknowledgements
This work was supported by a Discovery Grant from the Natural Sciences and
Engineering Research Council of Canada and a School of Management Research Fund
from the Telfer School, University of Ottawa.
References
[1] Ash JS, Berg M, Coiera E. Some Unintended Consequences of Information Technology in Health Care:
The Nature of Patient Care Information System-related Error. J Am Med Inform Assoc 2004;11:104–
112
[2] Reddy MC, Gorman P, Bardram J. Special issue on Supporting Collaboration in Healthcare Settings:
The Role of Informatics. Int J Med Inform, 2011;80(8):541–543
[3] Reddy MC, Shabot MM, Bradner E. Evaluating collaborative features of critical care systems: A
methodological study of information technology in surgical intensive care units. J Biomed Inform,
2008;41:479–487
[4] Thraen I, Byron B, Mullin S, Weir CR. Characterizing “information transfer” by using a Joint
Cognitive Systems model to improve continuity of care in the aged. Int J Med Inform, 2012;81:435–441
[5] Coiera E. Interaction design theory Int J Med Inform, 2003;69:205-222
[6] Kuziemsky CE, Varpio L. Describing the clinical communication space through a model of common
ground: ‘you don’t know what you don’t know’, AMIA 2010 Symposium Proceedings,2010; 407-411.
[7] Hseih HF, Shannon SE. Three approaches to qualitative content analysis. Qual Hlth Res
2005;15(9):1277-1288.
IOS Press, 2013
Subject Index
ABC framework 286 CPOE 151, 367
adverse drug event 162 critical thinking 68
agent-based model 383 cystic fibrosis 137
APEX 238 data collection 93
approaches 68 data integration 98
appropriateness 132 data management programs 251
autopoiesis 42 data mining 98, 111
baccalaureate nursing education 356 data quality assurance 37
benefits evaluation 15 data storage issues 227
bicycle accidents 93 data validation & verification 28
biomedical 189 database 238
biomedical informatics 151 decision making 107, 189
business case 28 decision quality 332
cancer registry 251 decision support system 132
cancer therapies 98 decision support tools 132
cardiology 63 dental 314
care coordination 21 dental informatics 314
CDSS 162 diabetes 57
challenges 9 diabetes mellitus 74
child 43 diagnostic imaging 132
chronic illness 308 digital stethoscope 63
clinical adoption 15 disadvantage 276
clinical decision support 98, discrete-event simulation model 383
116, 126 drug information system 168
clinical improvement 251 education 28, 49, 345
clinical informatics 251 efficiency 174
clinical practice guidelines 220 e-health 74, 220, 251, 271, 281
clinical research informatics 98 electronic health record 3, 37, 42,
clinical simulation 220 49, 232, 276
clinicianled development 3 electronic healthcare record
coding 28 (EHR) 28, 145, 151
collaboration 251 electronic medical record (EMR) 15,
collaborative agency 388 28, 49, 145, 214, 375
common data model 28 electronic prescribing 327, 383
common ground 388 electronic whiteboards 174
communities of practice 350 EPM model 332
competencies 345 foster 43
complex adaptive systems 209, 383 general practice 314
complex interventions 263, 375 general practice computerization 28
consumer health 302 GOMS-KLM 174
consumer informatics 244, 296, 302 Hajj 281
coordination 388 handoffs 21
394
handovers 21 interoperable standards 195

handwritten prescribing 383 iPad 238
healing 281 Islam 281
health 238 jQuery mobile 238
health behaviour change 286 KAMC 57
health care economy 42 kiosk 79
health care research 74 knowledge discovery 111
health informatics 49, 107, 145, knowledge translation 257
151, 263, 345 knowledge
health information 9 representation/organization 302
health information exchange 116 LOINC 195
health information system 151, 257 looked after 43
health literacy 179 machine learning 201
health network 116 managed care 116
health record 43 mature personal health record 9
health services research 263 medical device 337
health websites 244, 296 medical education 107
health-care associated infections 87 medical imaging 132
healthcare information medication 327
technology (HIT) 21, 145 medication adherence 116
health-care public reporting 87 medication errors 367
health-care quality 87 medication management 168
HealthShare 291 medicine 327
heart failure 63 methods 345
heuristic evaluation 179 M-health 227
HIT implementation 209 mobile phone apps 227
HL7 V3 CCD 195 mobile technologies 232
HL7 V3 CDA 195 national guard 57
human computer interaction 157, 232 natural language processing 201
human engineering 162, 337 neuroanatomy 107
ICT supported chronic disease nurse practitioner 111
self management 137 nurse practitioner integration 111
improvement 168 nurses 350
in-care 43 nursing 367
incremental project development 137 nursing curriculum 356
informatics 238 nursing informatics 345, 356
informatics in primary care 319 online forum 350
information accountability 145 online social network 286
information exchange 28 ontologies 201
information extraction 201 open source 3, 238
information governance 145 open standards 3
information retrieval 302 order entry system 132
information sharing 302 organization models 42
information systems 276 out of home 43
in-situ 157 oxygen 126
integration 42 participatory 68
international 251 patient communication 271
internet 314 patient education 271, 302, 314
interoperability 209 patient empowerment 244, 296
395
patient safety 132, 151, 337, 367 specialization 42

patient-centered web-based spirituality 281
portal 319 SpO2 126
PatientsLikeMe 291 STDC 332
pediatric oncology 251 synoptic operative report 195
performance measurement 87 system support 28
personal health record (PHR) 9, systems analysis 37
179, 308 systems safety 214
population based outcomes systems thinking 42
research 98 tablet 238
population health management 116 tailored health communications 319
practice patterns 111 technological pedagogical
prevention 286 content knowledge 49
primary health care 375 technologies as agents of change 232
process redesign 209 technology assessment 189
procurement 308 technology-induced error 151, 332
prototype 132 telecardiology 63
provincial health services telecare 57
authority (PHSA) 63 telehealth 63, 68, 79
quality improvement 126 TeMaD 57
radiation exposure 132 touch-based 238
Ramadan 281 training & support 28
relational care 319 trauma registry 93
research 28 trust 189
retinopathy of prematurity 126 usability 151, 157, 168, 308
review 43 usability evaluation 174
risk management 337 usability testing 145, 157, 179
safety 214 usefulness 220
Saudi Arabia 57, 244, 296 user-centered design 79, 162, 286
security & confidentiality 28 Vancouver Island 63
self-management 291 Vancouver Island health
semantic web 201 authority (VIHA) 63
simulation modeling 209 videoconferencing 63
simulations 151 virtual community 257, 271
SNOMED-CT 195 virtual physiological human 382
social media 291 web-based 238
software 337 website usability 87
software issues 227
IOS Press, 2013
Author Index
Aarts, J. 327 Cirelli, J. 126
Abrahamson, K. 350 Clayton, M. 314
Ahluwalia, R. 3 Cornett, J.A. 388
Ahmad, A. 367 Courtney, K.L. v, 79, 189
Alkadi, K. 57 Cummings, E. 137, 227
Al-Mutairi, M. 296 de Semir, M. 271
Alshaikh, A. 367 Del Fiol, G. 116
Alsughayar, A. 296 Dogar, N. 3
Alsulaiman, A. 195 Duncan, K. 98
Alsuweed, F. 367 Ebdon, N. 68
Amini, A. 87 Edwards, R. 116
Anderson, J.G. 350 Eisenstein, E.L. 116
Anstrom, K.J. 116 Elias, D. 302
Antillon, F.G. 251 Ellis, L. 137, 291
Arreola, M. 251 Fels, S. 286
Aylott, M. 63 Foster, J. 345
Bainbridge, M. 28 Fox, R. 350
Bains, A. 238 Frank, G. 63
Barnett, J. 232 French, J. 98, 232
Bassi, J. 49, 257 Frisch, N. 345, 356
Beach, S.R. 189 Fyfe, M. 93
Beggs, S.A. 137 Fyfe, M.L. 63
Bellwood, P. 93 Gavin, D. 98
Beuscart, R. 162 Gene-Badia, J. 271
Beuscart-Zephir, M.-C. 162, 337 Ghany, A. 209, 383
Birnbaum, D.W. 87 Gidora, H. 68
Black, B. 87 Glover, K. 68
Bonney, W. 195 Grajales III, F.J. 271
Borrego, L. 201 Grau, I. 271
Borromeo, C. 314 Graydon, B. 126
Borycki, E.M. 49, 68, 93, 111, 132, Griffith, J. 132
157, 227, 332, 345, 356 Hagens, S. 257
Bowen, M. 15, 168 Harmer, S. 68
Bras Da Costa, S. 337 Hauser, J. 137
Bricker, A. 302 Hebert, M.A. 74
Cameron-Tucker, H. 137 Henderson, J. 63
Chelsom, J. 3 Hess, S. 314
Cheney, J. 137 Ho, C. 98
Cheung, H. 98 Hochheiser, H. 314
Chia, S. 98 Hood, S. 232
Chio, D.T.S. 332 Horwood, T. 63
Christie, S. 195 Househ, M.S. 9, 244, 281, 296, 367
398
Howard, S.C. 251 Nahm, M. 302

Hudson, D. 107 Nøhr, C. 220
Hyman, D.A. 87 Partridge, C. 15
James, A. 126 Patel, A.N. 251
Jensen, S. 220 Paterson, G. 195
Jessup, M. 137 Pelayo, S. 337
Jiang, Y. 189 Price, M. 168, 257, 375
Johnson, F.S. 116 Quaresma, P. 201
Juhra, C. 93 Quick, P.D. 68
Kahnamelli, A. 232 Quintana, Y. 251
Kamal, N. 286 Randhawa, G. 15
Kannry, J. 157 Rasmussen, R. 174
Kawamoto, K. 116 Reid, D.W. 137
Keshavjee, K. 209, 383 Reid, S. 98
Khan, M. 232 Ribeiro, R.C. 251
Kim, J. 263 Roehrer, E. 137, 227
Kitson, N.A. 168 Rohrer, W. 9
Koppel, R. 21 Ronan, J.P. 319
Kuo, M.-H. 111, 238 Roudsari, A. 37, 57
Kushniruk, A.W. 49, 93, 132, 145, Saddington, H. 137
157, 174, 179, 308, 332, 345 Sahama, T. 145, 345
Kuwata, S. 145 Salkeld, S. 93
Kuziemsky, C.E. 209, 383, 388 Sangster-Gormley, E. 111
Lapointe, N.M.A. 116 Schleyer, T. 314
Laskin, J. 98 Schreiber, R. 111
Lau, F.Y. 15, 43, 168, 257, 375 Schulz, R. 189
Leaver, C. 257 Shabestari, O. 37, 232
Leroy, N. 337 Shaw, N.T. 28
Lesperance, M. 375 Showell, C. 276, 291
Lindsay, J. 238 Siewiorek, D. 79
Lobach, D.F. 116 Siso, A. 271
Loiseau, S. 337 Slager, S. 232
Loyola, M. 63 Smailagic, A. 79
Lunblad, R. 302 Stroetmann, K.A. 382
Mak, P. 98 Taylor, M. 68
Marcilly, R. 162 Telles, J.L. 21
Mason-Blakley, F. 214 Thibault-Halman, G. 195
Matthews, J.T. 79, 189 Thiel, R. 382
McGahan, C. 98 Thompson, C. 43
McGregor, C. 126 Tuden, D. 68
McLean, J.A. 238 Turner, P. 137, 276, 291
McMillan, J.M. 79 Vassanji, K. 209, 383
Melhem, S. 232 Viceconti, M. 382
Micallef, N.A. 137 Vingtoft, S. 220
Miller, W. 68 von Below, A. 93
Minshall, S. 151 Wainwright, C. 137
Monkman, H. 179, 308 Waldron, J. 232
Motoc, B. 42 Walters, H. 137
399
Weber, J. 214 Wu, J. 98, 232

Wieskötter, B. 93 Ying, T. 238
Willis, J.M. 116 Zozus Jr., R.T. 302

Enabling Health and Healthcare Through Ict

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Enabling Health and Healthcare Through Ict

Uploaded by

Copyright:

Available Formats

ENABLING HEALTH AND HEALTHCARE

ISSN 0926-9630 (print)

Amstterdam • Berrlin • Tokyo • Washington, DC

ISBN 978-1-61499-202-8 (print)

Distributor in the USA and Canada

PRINTED IN THE NETHERLANDS

ITCH 2013 Steering Committee

Multi Media Production

Local Organizing Committee

ITCH 2013 Scientific Review Committee

Electronic Health Records

Clinician-Led Development of Electronic Health Records Systems 3

Telecare, Telemedicine and Telelaboratory

TeMaD System: Telecare for Managing Diabetes in Saudi Arabia 57

Developing a Strategy for Studying Critical Thinking in a Nurse Telehealth

Public Health Informatics

Public Reporting of Hospital Infection Rates: Ranking the States on Credibility

Clinical Decision Support Systems

Neuroanatomical Basis for Recognition Primed Decision Making 107

Human Computer Interaction

Information Accountability and Usability: Are There Any Connections? 145

Patient Safety, Medical Errors & Quality Management

Clinician Variations in Data Trust and Use 189

A Complex Adaptive Systems Perspective of Health Information Technology

Mobile Technologies and Telehealth

POND4Kids: A Global Web-Based Database for Pediatric Hematology and

Forumclínic: The Shaping of Virtual Communities to Assist Patients with

Islamic E-Health: Definitions, Applications, and Challenges 281

Ontologies, Trust and Standards

The Social Act of Electronic Medication Prescribing 327

Developing National Level Informatics Competencies for Undergraduate

A Framework for Leveling Informatics Content Across Four Years of a

Healthcare Modeling and Simulation

Developing a Multivariate Electronic Medical Record Integration Model for

Subject Index 393

Clinician-Led Development of Electronic

Keywords. Electronic Health Records, Open Standards, Open Source, Clinician-

Figure 1. cityEHR Reference Platform of Open Source Software Components

2. Implementation Using cityEHR

3. Spreadsheets for clinical users

Figure 2. Information Model as an Ontology, Edited Using Protégé

Figure 3. Electronic Health Records System Generated from Information Model

Figure 4. OpenOffice Calc Spreadsheet Mirrors the OWL/XML cityEHR Architecture

Figure 5. Information Model in OWL/XML Transformed to any XML for Editing

Implications of the Mature Personal Health

Abstract. The purpose of this paper is to present a future scenario projecting a

Keywords. Personal health record, mature personal health record, health

1. Defining the Personal Health Record

2. Future Scenario for the MPHR

2.1.Challenges to the vision of the Mature PHR

2. 2. Legal and Regulatory Challenges

One obstacle to consumer acceptance may be heightened concern about maintaining

pervasive societal concern about potential assaults on privacy of information and

2.3. Institutional and Management Challenges

2.4. Societal and Cultural Factors

2.5. Technological Challenges

Applying the Clinical Adoption Framework

Keywords. Electronic medical record, clinical adoption, benefits evaluation

2.1. Evaluation Design

A team of four university-based researchers conducted the evaluation over a 6-week

2.2. Data Collection and Analysis

3.1. EMR Quality

3.3. Net Benefits in Coordination and Efficiency

#

}

} *

<

*

* < *

`

<

*

}

><

} *|||<

!