Producing a Systematic

Review
David Denyer and Davld Tranfield

INTRODUCTION key technique. However, different academic

fiel ds have idiosync ratic characteristics and
The aim of thi s chapter is to provide guidance as a result require the development ofbespoke
to scholars. practitioners , and policy makers approaches specifically tailored to serve their
who are engaged in producing, commìssion~ particular purposes. forms, and applications.
ing, or using reviews of research evidence Systematic review is a specific method-
in the field of management and organization ology that locates existing studies. selects
s tudies. In contrast with many other natura[ and evaluates contributions, analyses and
and social science fields, inexperienced synthesizes data, and reports the evidence
researchers. particularly doctoral students in such a way that allows reasonably clear
in management and organization studies, conclusions to be reached about what is and
often receive relatively little training in is not known. A systematic review should
conducting research reviews . Compared not be regarded as a litcraturc review in
to the wealth of texts on philosophical the traditional sense. but as a self-containcd
approachcs to social science research and research project in itselfthat explores a clearly
methods for empirica! investigation. there are specifìed question. usually dcrived from a
few instructional texts on literature reviewìng. policy or practice problem, using existing
This is surprising gìven the criticai role rhat studies. Additionally, systematic review also
literature reviews play in doctoral theses differs from other review methods because of
and journal publications, and the potential its distinct and exacting principi es. Por ex.am-
role that they could play in creating and ple, in a systematic review, thc researcher
building bodies of knowledge and infonning is required to set prespedtied rclevancc and
policy and practice (Tranfield et al., 2003 ). quality criteria for the selectìon/inctusion of
We argue that researchers in management studies and to make such criteria transparent
and organization studies have a signitìcant to readers. Extensive searchcs are conducted
opportunity to evaluate and learn from many t o incorporate both published and unpuhlished
other fields that ha ve developed an evidence- studies. In tenns of outcome, whcre studies
based approach using systematic review as a providc consistent rcsults, systematic reviews
672 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS

might be expected to provide solid and • The extent to which systematic review methods
dependable evidence that is robust and has can or should be proceduralized an d the extent to
potential for transfer across different c:ontexts. which bias can be reduced or mitigated.
O n the other hand, if the review identifies • The validity and reliabil ity of synthesized evidence
produced by systematic reviews, and how such
knowledge gaps or inwngruent findings, then
evidence is used subsequently to inform policy,
thìs s ignìfies a research need and raises
practice, and future research.
questions for future research.
Major improvements in review techniques
over the last fifteen years or so have helped Other critics ha ve suggested that evidence-
to raise the profile of systematic review as based manageme nt is overtly managerialist
an important research methodology. Applica- (Learmonth and Harding, 2006) and fail s
tions of systematic review in medicai science to take into account situated judgernent and
have led the way and are often considered to ethics (Morrell, 2008). They also express a
reflect the most advanced thinki ng. A 'stan- concern that evidence-based management and
dard' approach to systematic review was systematic review pri vileges certain types of
developed initially in the medicai fìeld by the research and other forms, such as criticai
Cochrane Collaboration (2008), and has been theory, are precluded.
followed later by other consortia dedicared to We apprec iate many of these concerns
commissionìng and disseminating systematic and challenges and take the view that it is
reviews. unwise to adopt, in an uncritical fa<;hion,
A Systematic review has been argued to that which has proved useful in other fields.
bring replicable, scientific, and transparent Consequently, out purpose in this Chapter
approach, which seeks to minimize bias (NHS is to explain how and in what ways the
Centre for Reviews and Disseminati on, 200 l ) notion of evidence-based management mìght
and requires reviewer s to summarize ali benefit the field through improved q uality
existing information about a phenomenon in a and rigour, by defining a bespoke and fit
thorough and unbiased manner. .\1ore widel y, for purpose fonnat for re view. Differences in
systematic reviews bave been argued also contex t are vita!, for example, in medicine.
to have value in collating and synthesizing there has been a gene rai consensus regarding
existing evidence across a wide range of what constitutes an a ppropriate methodology
settings (including the social sciences) and to be used for evaluating evidence from
empirica! methods (Petticrew, 2001 }. primary studies to be included (Evans, 2003).
On the otherhand the use of systematic In fields where the value of quantitative,
review, particularly ils traditional form, has qualitative, and theoretical contributio ns are
been critìcized when applied to the social equally respected (such as management and
sciences (Hammersley, 200 l ). U sually, these organìzation studies), reviewers are likely
criticisms have centred on uncritical mimetic to need more inclusive and comprehens ive
application encouraging the simple transfer quality criteria an d novel approaches to
of the medicai model of systematic review, research synthesis. Such inherent epistemo-
exemplified by the Cochrane Collaboration logica! divers ity necessitates an alternati ve to
(2008), into other fielcls. Critics have placed ' C ochrane-style ' systematic rcviews. T here-
four sers of arguments at the centre of this fore, we argue for the development of a
debate concerning: bespoke and fit for purposed methodology,
which can cope with the variety and richness
of research desig ns, purposes, and potential
• Competing ontological and epistemologica! posi-
tions related to the traditions of different end uses of management and organization
disciplines. studies reviews.
• Different perspectives on the systematic review Despite these differences our point of
methodology regarding the types of study to be departure is that systcmatic review methodol-
included in a review. ogy has proved demonstrably worthwhile in
 PRODUCING A SYSTEMATIC REVIEW 673

many natura! and social science fields, and be seen as a response to the increasing
that would be wise to consider the lessons demands to organize knowledge into a format
researchers in management and organiza- that is rigorous and reliable as well as
tion studies already learned. Consequently, make a difference to practice. Evidence-
this chapter first outlines the principles of based practice requires decision making and
systematic review developed elsewhere and action to be infom1ed by the best and robust
evaluatcs their relcvance for the fìeld of evidence produced by high-quality research
management and organìzation studies. We studies. In addition to the tradìtional approach
discuss both the origins of systematic review of conducting new empirica! studies, many
and its roots in evidence-based polìcy and natura! and socìal sdence fields ha ve tumed to
practice and also examine the methodological reviews of existing primary studies reported
challenges to developing an evidence base in the lìterature to address important and
in management and organi.zation studies. We pressing questi o ns. In this demanding context,
note that some of the key principles of sys- systematic review increasingly has become
tematic revìew that are manifest in fields such a standard method for locating, selecting,
as medicine, often remai n elusive and perhaps and appraising research and transferring the
undesirable for the field of management and synthesized findings not only to researchers
organization studies. Second, and on the basis and academics but also to practitioners
ofthis considered reftection, we offer a revised and policymakers in a digestible format to
and (to an extcnt) alternative frarnework and inform action.
set of principles for developing transparent, Evidence-based policy and practice (EBPP)
inclusive, explanatory and heuristic reviews constitutes a movement that can be traced to
for use in our fìeld. Finally. we explore the cari y t970s (Cochrane, 1972 ). EB PP i n
how the output from systcmatic reviews medicine can be defined as the 'conscientious,
might be used to inform policy, practice and explicit and judicious use of current best
furthcr rcscarch. In doing so, we argue that evidence in making decisiom;' about the
the methods of review should be selected care of individuai patients (Sackett et al.,
according to thc purpose and the nature 1996: 1). Since thc early 1980s, rcsearchers
and characteristics of the field of study in health care have been cngaged in a
(e.g. degree of fragmentation, plurality of programme of systematic reviews on the
methods, etc) and the nature of the evidem:e effectiveness of ìnterventions. TI1e Cochrane
availahle. We argue the case for combining Collaboration was fmmed in 1992 to gather.
the many facets of systematic review with evaluate, and disseminate research evidence
aspects of realist synthesis {Pawson, 2006). and developed a sophisticatcd and sub-
Harncssing the strength of both of thesc stantial set of gu idei ines for conductìng
approachcs offers an important methodology systematic reviews. Since then, rescarchers
for management and organization studies that have developcd and reflned the systcm-
has the potential to infonn policy. practice, atic review methodology with thc aim of
and future research. improving both the rigour ancl reliability of
the review process, and organ[zing knowl-
edge into a l.tseable forrnat. The results
The origins and principles of
of systematic reviews have become well
systematic review established as legitimate evidence on which
The idea of revicwing extsung research to base policy and practice. Website.s, such
studies for thc purposc of informing policy, as the Cochrane database (2008b), now
practice. and rescarch is not ncw. However, offer clinician.s a practical solution to the
over recent years, thcrc has heen a prolif- problem of staying up to date with the latest
cration of formai approachcs anci systematic research.
rne!hocls for locating, selecting, appraising, Given the relative success of evidence-
synthcsizing:, and reporting evidence. This can based approaches in medicine, the core
674 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS

principles, processes, and practices of sys- synthesize research accord1ng to an explici t

tematic review ha ve been adopted elsewhere. and reproducible methodology ' (Greenhalgh,
As Petti c re w (200 l: 99) argues: 1997). As in any scientific endeavour, the
methods to be us~d are required to be
.. .systematic review is an efficient technique for established beforehand (Light and P illemer,
hypothesis testing. far summarizing the results af 1984). Consquently, prior to a systematic
existing studies, and far assessing the consistency review being undertaken, a protocol is
among previous studies; these tasks are clearly not
produced to detail precisely how the review
unique to medicine.
will be conducted. The protocol is analogous
to and as crucial as a research design for
Following this line of thlnking, the Camp-
an empirica! study (Antman et aL, 1992;
bell Collaboration was formed in 1999 to
Cook et al., 1997). In medicine and other
establi sh the systematic review process in
fìelds, review protocols must be developed
the broader public policy arena (Campbell
and approved before the systematic review
Collaborati an, 2008). The Evidence for Policy
can commence. Reviewers are encouraged to
and Practice Information and Co-ordinating
publish their draft protocols (sometimes on
Ce ntre (EPPI-Centre) conducts systematic
these websites) allowing interested parties to
reviews in many fields, including education
provide feedback. While the intention should
and social care, and has developed an
be that a review will adhere to the published
approach for addressing a broad range of
protocol, it is acknowledged that the review
review questions and incorporating botb
protocol may need to be changed during the
quantitative and qualitati ve evidence.
course of the review. If modifìcations are made
In generai terms most fìelds using, sys-
to the protocol, they must be documented
tematic review prescribe that studies are
explicitly and explained (Higgins and Grcen,
expected to conform to a set of principles as
2006).
well as adopting specific methods to identify.
Systematic reviews bave a clearly defi ned
select, and cd tically appraise relevant primary
methods section wi th each step of the system-
research, before i t is appropriate to extract and
atic review rigorously reported. Justi fications
analyse data from the studies that are included
are gi ven for ali decisions taken by the
in the review (NHS Centre for Reviews and
reviewer. The ai m is to mitigate the preformed
Dissemination, 200 l ).
opinions that can bias his/her assessments of
In the following section, we outline the
both the relevance and validity of articles
four core principles traditionally <tpplied to a
in a review (Cooper ami Hcdges, 1994).
standard systematic review template. Reviews
Only studies that meet the inclusìon and
are expected to be: replicable, cxclusive,
exclusion criteria are indudcd in thc revicw.
aggregative, and algorithmic. We then con-
The reviewer also needs to discuss whethcr
trast these wìth a suggested set of principles
or not ali relcvant studies were identitied,
for systematic reviews in management and
whether or not all rclcvnnt data could be
organization studies.
obtaincd, and whcther or not the methods
used (for example, sean.: hi ng, study selection,
Replicable data extraction, and analysis and rcporting)
A fìrst key principle suggests that the generai could bave introduced hias. The aim is for
merhodological characteristics of systematic systematic reviews to:
review reflect an epistemology stipulating
... bring the same level of rigour to r~Niewing
that knowledge sbould be acquired through a research evidence as should be used in producing
'scientitìc', objective using an unbiased pro- that research evidence in the first piace (Davies and
cess. Systematic rcviews, particularly those Crombie, 200 1: 1).
associated with the Cochrane Collaboration,
attempt to avoid or to mitigate bias. As such, The usc of explicit, systematic methods in
'systematic reviews bave been developed to reviews has bcen argued botb to help produce
 PRODUCING A SYSTEMATIC REVIEW 675

more reliable results upon which to draw con- Table 39.1 The hierarchy of evidence in
clusions and make decisions (Amman et al., medicai sdence (Davies, Nutley and Smith,
1992). Cooper and Hedges (1994: ix) also 1999: 11)
argue that the 'intended result is a research H Systematic review and meta-analysìs of two or
more double blind randomized controlled trials.
synthesis that can be replicated by others'.
1-2 One or more large double-bl ind randomized
However, within a review of social science controlled trials.
literature, it is generally recognized that 11-1 One or more well-conducted cohort studi es.
professionaljudgment and interpretation play 11-2 One or more well-conducted case·control studìes.
an i mportant role and cannot be eliminated or 11-3 A dramatic uncontrolled experiment.
Ili Expert committee sitting in review; peer leader
replaced by proceduralization. Hamme rsley
opinion.
(200 l: 6) argues that: IV Personal experience .

. . . insisting that people follow a set of procedures

rigidly can be a hindrance. Literally, it may rule out
any reflection on issues that actually need attention, Typically, systematic reviews in medicine
encouraging a rather pragmatic orientation . And
this is especially likely w here the time available for
adopt a hierarchy of evidence (Table 39.1)
producing a review is very limited. As Polanyi points that ranks research designs according to
out, science, like many other activities, relies on their internai and external validity. Studies
skilled judgement, the capa city for which is built up are selected on the basis of relevance for
through experience; and skilled judgement means inclusion (ideally assessed by two reviewers).
taking account of particulars, and will often involve
In the medicai field. randomized controlled
apparently breaking the rules.
trials are regarded as the 'gold standard'
for judging 'what works' (Evans. 2003).
Exelusive For example, Oakley (2000: 318, quoting
A second key principle requires that if
Macdonald, 1996: 21) claims that anything
systematic reviews are to inform policy,
except randomized controlled trials 'is a
practice, and future research, they should
"disservice to vulnerable people"'.
synthesize the best evidence available (Slavin,
Whilst most evidence in medicine is made
1986). In systematic reviews, the validity of
available in quantitative form, a number of
a study is govemed hy the extent to which
systematic reviews include cvidence from
its design and conduct are likely to prevent
qualitative studies. Consequently, u set of
systematic errors, or bias (Moher et al., 1995 ).
quality standards for qualitative research
A review:
have been developed (see, for cxample,
Harden et al., 1999 ~ Oliver et al., 2005;
.. .should always focus on the best available
Popay et al., 1998 ) although the matter
evidence: that is, studies to be included in a
review should use methods that previde the least of what constitutes appropriate standards to
biased answer to the questìon asked (Counsell, apply to qualitative researc.:h stili remains
1997:380!. contested. More usually, evidencc t'rom
qualitative studies is rarely included in the
The Cochrane Reviewers' Handbook actual synthcsis process. and is more usually
(Higgins and Green. 2006) and the NHS descriptively summarized in thc systematic
Centre for Reviews and Dissemination review report. Whilst efforts have been
g uide lines (200 l ) both suggest that quality madc to incorporate qualitative evide nce in
relates to the extent to which the study medicai science, there remains a privileging of
minimizes bias and maximizes internai quantitative over qualitative resean:h (Dix:on-
and external validity. Relevant studies are Woods et al., 2005).
critically evaluated to assess their quality In the fie ld of management and organiz.ation
using chec klists that are included in the studies. there is a plurality of accepted
protocol along with details of how the methods and approaches. Whi lst this vmiety is
checklist was used in the study. often regarded a<; a wcakness, it also provicles
676 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS

diversity and richness whìch are a major despite subtle differences in effccts or lack
strengths considering that: of correspondence in the populations or
study contexts such as variation in form
Organizations are fuzzy, ambiguous. complex of implementation, measured outcomes, or
socially constructed systerns that cannot be well contextual factors that inftuence observed
understood from a single perspective (Denyer et al., effects (Rousseau et al., 2008). Yet, the
2008ì.
danger remains that unless primary studies
carefully report implementation processes
However, there is no do ubt that there ìs an d context, aggregati ve syntheses can mask
little agreement aver what represents high the mechanisms underlying effects (Pawson,
quality evidence across this field (Trantìeld 2002). Advocates of meta-analysis (Rauch
and Starkey, 1998) and certainl y there is an d Frese, 2006; Shadish and Sweeney, 1991 )
robust opposìtion to prioritizing one research argue that heterogeneity can be dealt with by
method over another. including mediators and moderator variables
into statistical techniques. A mediator is an
Aggregative intermediate variable that links an interven-
The third key principle suggests thatalongside tion with an outcorne. A moderator accounts
systematic review, statistica! meta-analysis for contextual differences, eithcr in terms
has become the preferred approach to research of the population o r the circumstances. A
synthesis in many domains. A meta-analysis host of demographic and contextual factors
extracts and aggregates data from separate can be specifìed and accountcd for in the
studies to increase the etfective sample size, analysis. The problem with this technique
in ordcr to calculate an overall effect size is that each of the likely mediators and
for an intervention. In order to perform moderators must be identifìed at the outset
a meta-analysis, it is important to identify of the review, and it is likely that in a
whether or not resu1ts from studies are con- managerial and organizational context the list
sistent one with another (i.e. homogeneous) of variables could be extensive. Further, each
or inconsistent (e.g. heterogeneous). Data ofthe studi es included in the review must have
from individuai studies are extracted · and comparable mediators and moderators, thus
tabulated in systematic review reports to severely limiting the number of studies on any
display thc impact of potential sources of specific topic that are amenable to aggregative
heterogeneity, e.g., study type, study quality, synthes is.
and sample siz:e. Additionally. despite advances in meta
lf the data extracted from these studies analytical techniques, many questions are
meet cert.ain requirements (the most impor- not amenable to aggregative approaches.
tant being a high level of homogeneity of In complex systems comprised of a large
effect measures across studies) then they variety of dynamically interacting variables
can be aggregated. Pooling data provides a such as can be found in management and
larger sample, increasing the possibility of organizational studies, the value of statistica)
detecting rea! effects that individuai smaller synthesis adoptcd as a single point solution is
studies are unable to ascertain. However, often questionable. Many systematic reviews,
aggregating studies can also cause problems even in medicine, do not contain a meta-
if the researcher fails to account for subtle analysis. Dixon-Woods et al. (2006) assert that
diffcrences in study design and study context, alternative approaches to synthesis, including
since it is possible to reveal minor bia<;es as qualitative techniques, can be incorporated
well as true effects. into systematic reviews.
Therefore, a key problem with aggregative
synthesis is that primary studies are rarely Algorithmic
entirely homogeneous. However, there are In health care, systematic reviews tend to
strong arguments that studics can be pooled focus o n questions relating t o the effecti veness
 PRODUCING A SYSTEMATIC REVIEW 677

of interventions. A large number of reviews particular circumstances in arder to obtain

provide a comparison of two or more optimum benefit.
alternative treatments to identify which is
the most effective. Systematic reviews in
medicai science tend to address specific Methodological challenges
questions structured according to three fea-
A review of the literature in the fiel d of man-
tures ~ the population, the intervention, and
agement and organization studies is typically
the o utcomes. A review may focus on very
the first step in the process of doing scientific
specifìc population groups by addressing
research, and is usually presented as either
factors such as the age and sex of the patients
a brief introduction to an empirica! study or
or the severity of their illness. Reviews
as an extensive detailed account of a body
roay also specify a particular setting of
of literature (see Academy of Management
interest such as whether people are living
Review or the lnternational Journal of Man-
in the community or are hospitalized (NHS
agement Reviews for examples). Reviewing
Centre for Reviews and Dissemination, 200 l ).
in management and organization studies is
The interventions are often the treatments
particularly challenging due to the fragmented
given to the populations of interest, which
nature of the fie ld (Whitley, 1984) an d al so itll
are usually a particular therapy, prevention,
transdisciplinarity (Tranfield et al., 2003 ). l t is
diagnosis, or screening (Higgins and Green,
a field in which many separate sub fields can
2006). The resulting outcomes are specific,
generate idiosyncratic questions, hypotheses,
measura ble clinica! and economie factors that
methodologies, and conclusions (Baligh et al. ,
are used for comparison. In some instances,
1996). Subfield academic communities easily
the study designs to be included in t he revìew
detach from identificatìon with the whole and
are also specified . An example of a well-
can often engage in their research unaware of
formatted review question is pro vided in the
work in associated areas:
Cochrane Handbook (Higgins and Green,
2006: 62):
The complicated state of MOS (Management
and Organization Studìes) research makes it
.. .whether a particular antiplatelet agent. such tough to know what we know, especially as
as aspirin, [inteNentionl is effective in decreasing specialization spawns research communities that
the risks of a particular thrombotic event, stroke, often don't and sometimes can't talk with each
[outcome) in elderly persons with a previous history other. Organ izatìonal researchers commonly adopt
of stroke. positions regarding management practices and
organizational decisions for which no accumula-
tion of evidence exists, or where at least some
Such algorithmic guidelines previde a of the evidence is at odds (Rousseau et al.,
procedure (a finite set of well-defined instruc- 2008: 477).
tions) for overcoming a condition and accom-
plishing a defì.ned outcomc. Algorithmic rules Tranfield et al. (2003) argue thal man-
require limited redesign for use in practice agement research is a nascent fìeld ~till
(van Aken, 2005). The problem with this developing in terms of agenda and focus.
approach is that it seeks to identify what In contrast with more mature fields, such
works without sufficient knowledgeof why an as medicine or engineering. management
i ntervention works or the partic ular conditions research does not enjoy consensus on research
of success (Pawson, 2006). To achieve richer methods and convergence on research ques-
understanding often requires practitioncrs to tions. Accordi ngly, researchers tend to ask and
integrate evidence with their knowledge of to address a steady ftow of questions rather
local circumstances and the preferences of than integrate and build coherent knowl-
the end user. Therefore, it is usually thought edge stocks or seek further understandi.ng
w ise formature users of systematic reviews to of particular phenomena. Vastly increasing
decide ho w appropriate rhe evidence is t o their volumes of management rescarch. coupled
678 THE SAGE HANDBOOK OF ORGAN!ZATIONAL RESEARCH METHODS

with the potential ofinformation technologies, faced by professionals suc h as clinicians,

has meant recently that synthesizing diverse engineers, lawyers, and managers in their
literatures and accumulating a knowledge work roles, rather than focusing on pure
base serving both the research and practìtioner knowledge problems. In so doing, they argue
communities, hac;; become an increasingly that systematic reviews in management and
topical challenge. organization studies can produce knowledge
In addition to informìng thc research com- of use to managers in providing insight. or
munity, a review of existing rescarch evidence designing solutions to field problems.
has the potential to guide practìtioners by
providing ideas, information, examples, and
recommendations for practìce. So why have Revised principles for management
practicing managers so far rarely used sys- and organization studies?
tematic review to improve their performance?
A nurnber of authors bave argued that the
First, practicing managers rarely have access
field of management and organization studi es
to academic journals or posses the skills
might benefit from the adoption of systematic
required to interpret and appraise research
review methods (Briner, 2000; Pittaway et al. ,
evidence. Second, many reviews have been
2004; Tranfield et al., 2003). However,
framed to answer theorctical questions for
Tranfield ec al. (2003) voicc a concem that the
academìcs and do not address the problems
adoption ofCochrane-style syscematic review
that managers face in their work roles. The
is questionable and potentially undesirable for
aim of academic reviews is to position an
use in the tield of management and organi-
empirica! study within an established body of
zation studies, arguing that a revised and fit
knowledge. According to Wallace and Wray
for-purpose systematic review methodology
(2006) such reviews relate to one or more
might be more appropriate. However, it has
explicit or, more typically, implicit review
remained unclear precisely which elements
questions that may be substantive (about
of the standard systematic rcview templatc
some aspcct of the social world), theoretical
should remain unaltered, which facets require
(about concepts, perspectives, theories, or
amendment, what should be refuted, and
models that relate to some aspect of the
w ha t needs t o be added for etfecti ve use
social world), or methodological (about the
in the field of management and organization
approach to conducting an empirical or
studies. Reviews in medicai scìence, we have
theoretical enquiry). A traditional review
noted, are based on fo ur core principles, but
seeks an understanding of the nature of the
ali of them bave limitations when applied
field, the major issues and debates about
to management and organization studies.
the topic, how the field has been structured
Consquently, we now offer, four alternative
and organized, the key theories, concepts and
principles for systematic reviews for use
ideas, and the ways in which the field has
in management and organization studies.
been researched to date (Hart, 1998). Such
We suggest reviews be tested for their
reviews usually tackle knowledge problems
transparency, inclusivity, cxplanatory, and
and focus on analysis and explanation, and
heuristic nature.
on problems and their causes (Denyer et al. ,
2008). Such research 'criticizes everyday
accounts an d practices .. . but does not seek Transparency
to transform them except in the generai Concerning our first key prìnciple, we contend
sense of demonstrating their incorrectness' that the reasons for documenting the review
(Whitley, 1984: 372). methods is not to achieve replication or erad-
Denyer et al. (2008) argue that, if the icati an of hias, as i n a Cochrane-style review,
product of a review is intended tu be used by but rather to ai d transparency (Tranfield et al.,
practitioners and policy makers, then it should 2003). We believe that there are three aspects
address field problems, that is the problems to transparency in a systematic review.
 PRODUClNG A SYSTEMATlC REVlEW 679

First, reviewers must be open and explicit A clear audit trai! is required linking the
about the processes and methods employed in claims made by the authors of the review with
the review. This enables readers to determine the existing evidence. According to Pawson .;.
:: '
precisel.y its scope and boundaries. Through- (2006: 79): :~ ' .

out the review, the steps undertaken need to

be specifìed, a pplied, recorded, and monitored ... other researchers, decision makers and other ~- ..

(Tranfield et al., 2003). For example, the stakeholders ca n look behind an existing review, to
; ,·
practice of making the review m ethods assure themselves of its rigoJr an d validity, rel iabilìty i
;, _
and verifiability of its f indings anè conclusions.
explicit before the review commences is a
constructive exercise. Producing a review .:

protocol enables the reviewcr to gai n feedback The third element to transparency necessi- i:..·
~:

on the proposed methods and highlight any tates the reviewer making clear the assump-
obvious errors or omissions so that they tions underpinning the review and engag- ·. i .

can be amended or rectified. lncluding a ing in a mindful questioning of a priori

methodology section in the systematic review beliefs regarding the scope and implica-
report enables readers to detennine precisely tions of relevant research. In conducting a
the scope and boundaries of the review. It review, particularly problem specification,
also provides an audit trai! and enables the study selection and synthesis, the revìewer
review to be updated and appraised in the necessarily falls back on their values,
future. A systematic review protocol does prejudices, and beliefs. Wallace and Wray
not mean that the predetermined methods (2006: 88) argue that the 'authors' explicit
are set in stone. It is important that the or implicit values about some aspect of
protocol does not restrict the review and the social world, their theorizing, research
it is quite norrnal far reviewers to alter methodology, and methods may affect their
the protocol durìng the course of conduct- focus and the nature ofthe knowledge claims
ing the review. A revìewer may, for example, that they make'.
find a body of work that they had not In partkular, the reviewer 's prior know-
foreseen or altematively for which they may ledge of the literature may influence the
wish to alter the selection criteria as their review. At worst, 'a reviewer may simply
understanding of the field develops. In such build a case in support of their personal .~ ':.

circumstances. the protocol can be used to beliefs, selectively citing studies along the
document and justify these changes. In the way' (Davies and Crombie, 2001: 2). As such,
final systematic revicw methodolog,y scction, it is important that reviewers make cxplicit
ìt is customary to produce an overview of the their value stance towards the a<;pect of the
main changes between the originai and fina! social world they are studying (Wallace and
protocols. Wray, 2006).
The second expression of transparency
concerns the presentation of the findings of the lnclusivity
review i n sue h a way that t here are clear links Our second key principle cmphasizes that
between the evidence found and the revicw- systematic reviewers in the field of omaniza-
ers' conclusions and recomme.ndatìons. As tion and management are likcly to en~ounter
noted by Slavin ( 1986 : 17): difficulties whcn appraising the quality of
information sourccs. Authors of articles, evcn
No review procedure can make errors impossible those published in some of the highest impact
or eliminate any ch ance that rev1ewers' biases will journals, sometimes fail to report on the
affect the conclusions drawn . However, applìca- methods of data collection and analysis in suf-
tions of best-evidence synthesis should at least fìcient dctail, rendering impossible attempts
make review procedures clear to the reader and
should previde t he reader with enough information
to a'i.sess study quality. Within management
about the primary research an which the review is and organization studies, there is also little
based to reach independent condusions. uniformity in methods of data collection
680 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS

and analysis. Studies rarely address identica! the subfield(s) involved, justify the reasons
questìons and samples vary in terms of the for inclusion/exclusion of studies, apply the
populations, sizes, study context, and the data criteria to all relevant studies, and communi-
reported. cate the warrants underpinning their claims
Boaz and Ashby (2003: 4) suggest that (Wallace and Wray, 2006).
rather than a hierarchy of evidence, the
selection of articles can be based on the Explanatory
criterion 'fit for purpose'; this, 'helps us to Our third key principle addresses the vex.ing
get away from the technocratic preoccupation question of synthesis. The evidence-based
with elegant research designs and allows for community in the UK (see for example
notions of appropriateness to be added into the Evìdence Network website: http://www.
the appraisal of research'. Pawson (2006) evidencenetwork.org/) has been working hard
also argues that the researcher should simply to develop severa! alternative approaches
ask whether or not the literature retrieved to meta-analysis as a mode of synthesis.
adds anything new to our understanding Particularly, progress has been made in
of the intervention. Pawson 's approach has narrative reviewing (see Popay et al., 2006;
no hierarchy of evidence. The worth of Greenhalgh et al., 2005) and the use of the
the study can only be determined through metaethnographic approach of Noblit and
the synthesis by evaluating each study's H are ( 1982; see Campbell et al., 2003 ).
contribution to theory building, prioritizing Evidence syntheses os ing these approaches
the vital evidence from primary studies as are tasked w ith the construction of larger
the originai researchers' interpretations and narrati ves and more generalizable theory
explanations, not just the results. Hence, (Estab rooks et al., 1994; Sandelowski et al.,
Pawson encourages the scope of the review to 1997). Interpretive and explanatory synthesis
include a wide range of studies, research types extracts descripti ve data and exemplars from
an d data forms to promote a full understanding individuai studies, building them into a
of the phenomenon of interest. mosaic or map (Hammersley, 2004). The
Salipantc et al. ( 1982) ha ve also arg ued reviewer is involved in juxtaposing the
that the inclus ion of a broad variety of e vidence, which can be quantitative, qu;ùita-
sources in a review can compensate for tive , or theoretical. from one study with that
researcher value judgments and uncontrolled of another.
validity threats. Heterogeneity also makes Unlike aggregative .synthesis which .seeks
possible the investigation of contextual fac- to avoid or mitigate bias. imcrpretive and
tors inftuencing rhe study's design and its explanatory syntheses are 'active' and ' cre-
findings. They argue that the more het- ative' methods (Pawson, 2006) that go beyond
erogeneous the distribution of uncontrolled a descriptive rcporting of thc cvidence. Thc
validity threats in a set of similar fìndings, the process is one of conceptual innovation and
grearer the validity of the findings from reinterpretation (Campbell et a l., 2003), while
the set. Following this logic, we suggest attempting to preserve the origina) stully's
that reviewers in management and organi- integrity or wholeness {Pawson, 2006). As
zation studies should beware of excluding such, the synthesis provides a rcasiblc
studies solely on the basis of adherence explanation of the study fìndings rathc:r
to the pursuit of absolute epistemological tha n a replicablc cxplanation ( Noblit and
standards. In particular, reviewers are best Hare, 1988). Thereforc. synthcs is involves the
advised to guard against using proxies for process of bringing the picces from indi vidual
research quality such as the quality rating texts together to make a whole that should be
of journals as a basis for exclus ion. We more than the su m of tbc pa11s.
do, however, agree that systematic reviewers Pawson (2006) has introduccd thc notion
in management and organization studies of realist synthesis by using existing primary
de vise quality checklist(s) appropriate to studies to build a nasccnt thcory of how
 PRODUCING A SYSTEMATIC REVIEW 681

an intervention may work. This will include than providing a detailed solution to a specific
'conjectures on the generative mechanisms problem. Heuristic rules will almost always
that change behaviour, ideas on the contexts require the application of informai judgment
that inftuence its operation, an d hunchcs about or experience to contextualise action. Denyer
the different outcome patterns that ensue' et aL (2008) suggest that systematic reviews
(Pawson, 2006: 73 ). The texts that are to of the existing published science base in
be included in the synthesis are 'regarded as management and organization studies can
case studies, whose purpose is to test, revise develop knowledge that managers can use
and retine the preliminary theory' (Pawson, to design solutions to problems in their
2006: 74). The review entaìls the systematic field. In management research, rather than
organization of the data into formats that presenting the truth, in the form of 'valid
allow sumrnary. This body of evidence is evidence', managers rnay be presented with
then probed, sifted, coded, and cross tabulated some 'clues/ideas'. 'tools', and 'methods'
in nurnerous ways. Each relevant pubhshed that may help to guide design for effective
artide is described and discussed in terms implementatìon. Decìding the degree to whìch
of its contribution to the emerging theory the findings presented in any particular review
(Pawson, 2002: 345): can inform practice is therefore a matter of
judgment for the practitioner.
. . . the rev1ewer's basic task 1s to sift through the
mixed fortunes of a programme attempting to
discover those contexts ((+) that have produced Applying the new principles
so!id and successful outcomes (O) from those
contexts (C) that induce failure (0-). The review Step 1: Question formulation
process is then repeated across a number of
As with any research, the primary decision
additional stud1es featunng the same underly-
lng mechanism (M), wrth the aim of gathering
in preparing a review is to establish its
together the vanous permutations of success focus (Light and Pillemar, 1984). This is
and failure the aim is to differentiate and done most effectively by asking clearly
accumulate evidence of positive and negative CMO framed questions (Cooper and Hedges, l 994 ).
configurations.
By clearly formulating the question, criteria
for primary study inclusion in thc review
Generalization is sought not in terms of the become clear:
associations among variables but in terms of
the rolc and i mpact of generati ve mechanisms
A good systematic review is based on a wel!-
that play out in different ways over time. formulated, answerable questìon. The question
guides the rev!ew by definìng which studies wil! be
Heuristic w.c!uded, what the search strategy to identify the
The output of a systematic review m man- relevant primary studies should be, and which data
need to be extracted from each study. Ask a poor
agement wiil allude to what works. and
question and you will get a poor rev1ew' (Counsell,
to why or how the relation occurs and 1997: 381 ).
in what circumstances, but is likdy to be
relatively abstract and is best regarded by the In order to clarify review questions. inform
practitioner as a 'design exemplar' (v an Aken, the rcview process an d improve the utilization
2005 ). Gi vcn the compìexity of organizational of findings from systematic revicws, it is
setting:s. such outputs are likely t o be heuristic. best to involve a broad rangc of stakeholders
A heuristic rule may help in sol vìng a problem, in the developrnent of review questions and
hut is not guaranteed to provide a detailed procedures (Tranficld et al., 2004). Petticrew
solution. Outputs of systematic revìew in (200 l: HX)} argues that,
management and organization studies are
likely to be rulcs, suggestions, guides, or pro- 'potential users of systematic rt:views, such as
totype protocols that rnay be useful in making consumers and policy makers, can be involved in
progress toward a solution of a problem rather the process. This can help to ensure that reviews
682 THE SAGE HANDBOOK OF ORGANfZATfONAL RESEARCH METHODS

are welf tocused, ask relevant questions, and are interesting example of how organizational
disseminated effectively to appropriate audiences'. forms are constrained by context:

Advisory groups can assist with defining Bureaucr aci es organi ze work routines in certa in
the broad topic area to be investigated and w ays and previde tightly specified employment
identifying the specific areas within that topic functions for t heir workforces, but whether any
that would be most useful to scrutinize in of th is happens depends on t he availab ility of
work, and ultimately, on the overall economie
depth. Involving experts is particularly impor-
heath of the nation . The efficacy of bureaucratic
tant when the literature is sparse. The review management is also contingent on the type of work
pane l bcst includes people with academic carried out. In sectors that th rive on innovat ion and
knowledge and practìcal expertise in the entrepreneurial activity, the applicability of fixed
subject area such as Iibrarians and information duties to fixed roles is likely to flop.

scientists and systematic review specialists ,

who may be consulted individually or as a In social science fields such as organization
group at key points throughout the review. and management studies, it is not enough
In the tìrst instance, the review question to know what works but also to asccrtain
may be free-form, written in naturallanguage. why or how the rel ation occurs and in
The free-form question can then be structured what circumstances. Pawson's (2006) rea!ist
into a reviewable question. In management approach requires the revìewer to determine
and organization studies, it is sometimes (or infer) context, mechanism , and outcomc
difficult to defìne and measure constructs of contìgurations through comparing and con-
interest. There is usually an emphasis on trasti ng interventions in different contexts.
the requirement to understand the impact Such an approach has the advantage of
of study context on the study results . In a including dìfferent types of information. such
complex social world, it is often difficult as case studies, so long as they provide
to predict with any surety a uniform link some insight into what works, w hy, w here,
between intervention and desirable outcome. and when. Denyer et al. (2008) dcvelop this
Therefore, research in complex soci al systems argument using the acronym CIMO (Context.
(such as organizations) must always take into Intervention, Mechanism. and Outcome)- an
account the crucial role of context. As noted acrony m that can be used to s pecify the four
by Pawson and Tilley ( 1997), interventions criticai parts of a well-bui lt systcmatic review
in such systems will be affected by at question (see Figure 39.1 ).
least four contextual layers - the individ- In management and organization studies,
uai , interpersonal relationships, institutional a practitioner question may be framcd as
setting, and wider infrastructural system. follows: how can project team perfonnance
Pawson and Tilley also highlight the role be optimized through leadership behaviours?
of generative mechanisms , which help to This may be structured into a se.ries of
explain the ephemeral relationship between reviewable questions s uch as: under what
thc intervention and o utcome. Generative conditions (C) does leadership style (1 )
mechanism s are the nuc le us of explanation in inftuence the performance of projec t teams?
realist synt hesis (P awson, 2006) and typically. What mechanisms operate in the influencc
for any process, there will be at least one of leadership s tyle ([) on project team
mechanism operating (see Reed, Chapter 25, performance (0)?
this volume). More likely, in complex social In reviewing these questions, thcrc are
situations, there will be many mechanisms multiple perspectivcs with regard t<> thc mean-
working concurrently (Denyer et al. , 2008). ing of key constructs. Team performance,
The generative mechani sm is a theory that for example, may be measured in temu;
helps to explain causai relatìons by 'describ- of projecl outcomes 1-\Uch as tìmc, cost.
ing powers inherent in the system' (Pawson, and quality. However. from team members'
2006: 2). Pawson (2006: 24) provides an perspective , performance may be interpreted
 PRODUCING A SYSTEMATIC REVIEW 683 ....

Component Questionato ask

Context (C) Who are the individuals of interest?

• Employees, customers, shareholders, etc.
• Sex, age, etc.
• Role, position in organization, etc.
Which interpersonal relationships are of interest?
• Teams, group dynamics, structure of social networks,
etc.
Which aspects of the institutional setting are of interest?
• Politics, power, technical system, complexity,
interdependencies, etc.
Which aspects of the wider infrastructural system are of interest?
• Degree of competition, stability, etc.

lnterventlons (l) What is the lntervention of lnterest?

• Leadership style, planning and contro! systems, training,
performance management, etc.

Mechanisms (M) What are the mechanisms of interest?

What is lt about interventions acting in a context that leads to the
outcome?
Why are mechanisms activated or not activated? : :·

Outcomes (O) What ar e the relevant outcomes?

What outcomes would be important to the indivlduals involved?
How will the outcomes be measured?
What ls the primary outcome and what are the secondary t .~
outcomes?
• Performance improvement, cost reduction, low error
rates, etc.

Figure 39.1 Constructing review questions using the CIMO 1ogic.

\:

in tenns of internai dynamics and their produce a scoping study of the field. Scoping
cxperiencc of being a team member. lt studies are exploratory reviews, since what
is, therefore, essential to defìne key tenns to include and exclude is not self-evident.
during the question formulation phase. Since The purpose of the scoping study is to make
defìnitions and concepts are value laden, clear the existing basis for the re/-iearcher's
differcnt perspectives and bcliefs need to work and specify wherc and how the proposed
be madc transparent. Alth()ugh the review systematic review fits into the current body
questions are specificd a priori, a ce1tain of knowledge. The scoping study should help
ftexibility and modification of questions may to define concepts and determine the revicw
be necessary as the reviewer gains a fuller qucstions to be addressed.
understanding of the problem.
Until a very precise question(s) for a Step 2: Locating studies
systematic review is specified, detailed work Systematic reviews aim to locate, select,
on a systematic review is best not started. and appraise as much as possible of the
It is impractical and unfair for novice research relevant to the particular review
researchers to begin a systematic review question(s). The methods used to find studi.e s
without a significant grounding in t.hcir field. (database searches, searches of specialist bib-
Tranfidd et al. (2003) suggest that prìor Iiographies, hand-searching of likely journals,
to a systematic revicw, novices and less and attempts to track down unpublished
experienced researchers are best advised to research) necd to be reported in some detail.
684 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS

Exhaustive literaturc searches of publi shed There is a common misconception that

and unpublishcd studies are conducted to systematic reviews only involve clectronic
ensure that the review results bave taken database searches. However, Grccnhalgh and
into account all the available evidence and Peacock (2005) have shown that, in system-
are ba:;ed on bcst quality contributions . The atic reviews addressing complex questions,
people of the search is to identify al! relevant database searching can oftcn account for
studies and can cover a range of difterent types as little as 30 percent of the total number
of information (academic articles, books, of relevant articles. Therefore, a systematic
practitioner materials, working papers, and searchcan be expected to use severa! methods.
the Internet). It sounds trite to say so, but the including; searching electronic databases,
searches need to be ahgned tightly with the hand searching known journals, recommen-
review questions. dations from experts, and cross-referencing.
The search generally commences with fn addition to academic papers, articles in
an investigation of citation databases using nonpeer reviewed journals, books, databases,
search strings, groupìng keywords, and apply- websites, conference papers, seminars, work-
ing search conventions. shops, technical reports and discussion papers
and other 'grey literature' are ali important.
• Simple operators include: The decision whether or not to search specifìc
data sources is dependent on the fìeld and
- Truncation characters- '* ', '?'; e.g. team?- on the evidence availablc. For example. in
searches for documents which contain the nascent and fast moving tìelds, much of thc
term team, teams, teamwork, teamworking, evidence might res ide in working papers. As
etc. with other aspects of thc revic w, explicit and
- Word association - 'w' or 'near'; e.g. transparent choices must be maùe by thc
high(w)reliability; high(near)reliability
revtewer.
- Exact phrase - ' ', ' '; e.g. 'high reliability'; 'high
The output ofthe scarch is a comprehensivc
reliability'
listing of articlcs and papers (core contri-
• Boolean Logic includes: butions) which helps to address thc review
questions. To aid information management,
- OR, e.g. team* OR group* searches for either citation manageme nt software packagcs, such
term in a document as Endnote or Rcfworks. can hc used to storc
- ANO, e.g. 'high reliability' ANO organi*ation information accurately.
searches far both terms in a document
- ANO NOT, e.g. team* ANO NOT virtual* Step 3: Study selection and evaluation
searches for documents which contain the Following the requircment for transparency
term team, teams, teamwork, teamworking of process, systemaric rcvicws usc a sct
but not virtual of explicit .~elcction critcriu to asscss the
relevance of each study found to see if it
• By combining Boolean operators with parentheses
actually does addrcss thc rcvicw qucstion.
compi ex searches ca n be constructed:
Detailed decisions are rccordcd spcdfyìng
precisely the busis on whkh infonnation
- 'high reliability' ANO {team* OR group*) AND
sources huve bcen indudcd and excluded.
NOT virtual
The ai m of being explkit about the sclcction
ciitcria is to makc the revi~~ wcr's dc<.~isions
lt is often wise to spend significant time available for scrutìny ami evalmttion. Selec-
on constructing the search strings, for this tion criteria also fac ilitatt; the updating of
investment can improve the efficiency of systematic revicws. Onc simplc mcthod for
the search. It [s important to note that constructing sclcction critcria is to undcrtake
different citation databases use dìfferent <1 small numbc r of some pilot sean.:hes in a
search conventìons. citation database and thcn rnakc a list of the
 PRODUCING A SYSTEMATIC REVIEW 685

reasons for the indusion and exclusion of Wallace and Wray (2006) and Solesbury
articles. [f the search is well focused then (2001):
within a short time it is usually possible to
produce an exhausti ve list of reasons for inclu- (1) What are the generai details of the study -
sion and exclusion. The reasons for inclu- author, title, journal, date, language?
sion and exclusion then form the selection (2) What are you seeking to understand or decide ; ·. •j

criteiia. by reading this? { .~-~ '

Generai quality checklists, such as those (3) What type of study is this (philosophical/
produced by the Criticai Appraisal Skills discursive/conceptual, literature review, sur-
Programme (2008), may be used to help eval- vey, case study. evaluation, experimentlquasi
uate studies. Different subfields of manage- experiment, etc.)?
(4) What are the authors trying to achieve in ': ~

ment have different perspectives on research

writing this? What are the broad aims of the
quality and so it is also important to take
study? What are the study research questions
this into consideration. For example, some and/or hypotheses?
journals produce comprehensive guidance on {5) How is the study informed by, or linked to, an
appraising research evìdence, such as the existing body of empirica! and/or theoretical
Journal of Occupational and Organizational research?
Psychology's g uidance on qualitative studies. (6) In which contexts {country, sector and setting,
By combining generai qualìty criteria and etc.) and whkh people (age, sex, ethnicity,
guidance from key joumals in the field, it occupation, role, etc.) or organizations was the
is possible to create a bespoke and practical study conducted?
quality appraisal tool. The purpose ofthe tool (7) What are the methodology, research design,
sample, and methods of data collection and
is not to exclude papers that are deemed to be
analysis?
.
'• .l
~

i
·..

low quality, butto evaluate and report on the ~

(8) What are the key findings? ,'

).··
study and its limitations. However, if studies (9) How relevant is this to what we are seeking to
are excluded o n the basis of quality, i t is crucial understand or decide?
for the reviewer to document and justify the (1O) How reliable/convincing is it - how well·
reasons f()r this exclusion. founded theoretically/empirically is this
(regardless of method)7
Step 4: Analysis and synthesis (11) How representative is this of the population/
After assembling the appropriate collection context that concerns us?
of relevant sources, the data analysis and (12) In condusion, what use ca.n l make of this7
synthesis commences. The aìm of the analysis ·.i -
is to break down individuai studies into The extraction form may also include the
constituent parts and describe how each details of the reviewer (particularly if rhere
relates to the other. On the other band, the are multiple extractors) and the date of the
aim of synthe~is is to make associations review. At the end of a systcmatic review,
between the parts identified in individuai a full tabulation of ali the included studies
studies. A synthesis needs to go beyond mere is displayed, providing a comprehens1ve
description by recasting the information into a summary representation of the field of study.
new or different arrangement and developing By cross tabulating the studies, key issues
knowledgc that is not apparent from reading can be identified. For example, in thc column
the indi viduai studies in isoiation. that lists the existing body of empirica) and/or
The fìrst step of analysis is w extract and theoretical research that studi.es refer to, it
store information ondata ex.traction forms for will be possible to see whether the field has
every study included in the review. The data a single theoretical foundation or contending 1 .

extraction forms generally address a series of or complementary theo ries.

interre lated questions, but the specific data Many systematic reviews employ two or
collected may vary from study to study. more inde pendent reviewers to extract data
The following questions were adapted from from studies. When the imerpretations and
686 THE SAGE HANDBOOK OF ORGANJZATIONA L RESEARCH METHODS

findings of reviewers are compared, it is In Cochrane-s tyle reviews, review

possible to mmlmtze errors, resolve any encouraged to think about whether or n·
differences, and produce a more robusr data is sufficient evidence to provide clear
set. The aìm of using data exrraction forms is lines for practice by asking three ques1
to previde an audi t trail from the claims made
in the review to the underlying evidence. (1) Will the pradice improve outcomes?
As noted in the foregoing section, reviews (2) Should the practice be abandoned in ligi
in management and organization studies are available evidence?
in generai more likely to be interpretive (3) Are there trade-offs between known
or explanatory rather than aggregative. Har- and known adverse effects?
monizing synthesis with the purpose of the
review therefore is criticai. In all cases, lf the review provides insuffkient e v
the resulting body of evidence is then to previde clear guidelines for practice
explored, cross-tabulated and analyzed while funher ques tions are raised:
engaging in rigorous reflection on any values,
beliefs, and perspectives thatmight impact the (1) ls the practice promising but requires
interpretation. In a quantitative synthesis, the evaluation?
reviewer might highlight both the regularities (2) Does a practice that has been shown
and discrepancies in the data, whereas, in have the effeds expected from it require
attention?
a qualitative synthesis, the reviewer could
(3) ls the re reasonable evidence that practic
explore analogous and different meanings
effective?
of respondents across the studies. Where
multiple forms of evidence are available, it
Reviewers in management and org
may be possible to attempt triangulation of
tion studi e.s may make similar claims.
the datalfindings.
ever, since recommendations from re
in our tield are likel y to be heuristic
Step 5: Reporting and using the resu/ts than algorithmic, it is essentialthat systc
A systematic review is structured in a
review reports should provide consid~
similar manner to a report of empirica)
detail from the originai s tudies so th<J
research. The introduction section provides
users of thc review can, if ncccssary, int1
a statement of the problem and the review
the results and j udge fnr thcmselvc
questions. The me thodology section pro-
strength of thc tìndings.
vides precise details of how the review
was conducted - the search strategy, the
selection criteria, and the analysis and
synthesis criteria. The findings and discus- CONCLUSION
sion section contains a summary of ali
the studies in terms of thc data extracted Despite common miscom:cptions in thc
from the studies such as the percentage and popular media, cvidencc -hascd pn
of studìes in the fìeld that are philosophi- has never sought to providc 'an~wcr
cal/discursive/conceptual, literature reviews, to re place judgment and cxpericnce,
surveys, case studies, evaluations, or exper- instcad attcmpls to inform dccision m:
iments/quas i experiments. The findings and and action (Sackctt et al .. l CJ96). The Cod
discussion section also specities precisely Handbook ( Higgi ns and Grccn, 2006: l~
what is known and unknown about the careful to point o ut that
questions addressed in the review. The
conclusion section provides a summary of ... the pnrnary purpo~e of t he review ~hould
present information. ratht"r than Clffer advto
the review, the limitations of the study,
d1scussion and condu<> ions ~.hou ld be to help p
recommendations for policy ami practice, and to under stancl the imphe<ltJOns of the ev1der
future research needs. rela tionship to pra( tkdl dec. 1s Jon~
 PRODUCING A SYSTEMATIC REVIEW 687

Academic evidence is only one input into REFERENCES

the decision making process even in more
developed fields such as medicai science, for Antman, E.M., Lau, J., Kupelnick, B., Mosteller, F.
'clinicians reason about individuai patients on and Chalmers, T.C. (1992) 'A comparison of results
the basis of analogy, experience, heuristics, of meta·analyses of randomized contro! trials and
and theory, as well as evidence' (Cook reco mmendations of clinica l experts. treatments for
et al., 1997: 380). Therefore, the tenns <evi- myocardial [nfarction', JAMA, 268(2): 240-8.
Baligh, H.H., Burton, R.M. and Obel, 8. (1996)
dence informed' or even 'ev1dence aware',
'Organizational consultant: creating a useable theory
rather than 'evidence based' may be more
for organizational design', Management Science,
appropriate in management and organization 42(12): 1648-62.
studies (Tranfield, et aL, 2003). Hammersley Boaz, A. and Ashby, D. (2003) Fit for Purpose?
(2004: 4) argues, that: Assessing Research Qua/jty for Evidence 8ased Policy
and Practice, Queen Mary: University of London,
. . .sometimes it will simp!y not be sensible to engage London .
in elaborate explication of goals, to consider ali Briner, R. (2000) 'Evìdence-based human resource
possible alternatives, to engage in a search far management', in L. Trinder and S. Reynolds
ìnformation about the relative effectiveness of (eds), Evidence~Based Practice: A Criticai Appraisal,
various strategies as agaìnst relying on judgements Blackwell Publishing, pp. 184.
about this, or to try to measure outcomes. Campbell Collaboration (2008) http://wvvw.campbell
The linear rational model tends to underplay
co!laboratlon.org/ [Accessed 22 june 2008).
the extent to which in many circumstances the
only option is tria! and errar, or even 'muddling
Campbell, R., Pound, P., Pope, C. an d Britten, N.
through'. (2003} 'Evaluatìng meta-ethnography: a synthesis of
qualitative research on lay experiences of diabetes
and diabetes care', Soda/ science and medicine,
However, any assumption that practitioners 56(4): 671.
are unlikely to use evidence that has been Cassell, C. Criteria for Evaluating Papers Using
produced by a systematic and rigorous process Quafitative Research Methods [Homepage of
solely because they do not have the necessary Journal of Occupational and Organizational
timc and appropriate skills, fails to question Psychology], [Online). Available: http://vvvvw.bps.org.
the desirability of sue h an outcome and draws uk/publications/journals/joop/qualitative-guidelìnes.
attention to the training and development cfm [February 2008].
needs of managers. lt is true th.at managers Cochrane, A. ( 1972) Effectiveness And Effidency:
Random Reflections on Health Services, (first edn.),
have at prcscnt rarely been trained to evaluate
London, Nuffield Provincia! Hospitals Trust.
research but it is patronizing in the extreme to
Cochrane Collaboration {2008a) http://www.cochrane.
suggest that their efforts çould not, or worst org/ [Accessed 22 June 2008\.
stili should not be infonned by the very best Cochrane Collaboration {2008b} http://w\f\IW.cochrane.
available evidencc. For academic evidence to org/reviews/ [Accessed 22 June 2008).
be used by managers it needs to be rendered Cook, D.J., Mulrow, C.D. and Haynes, R.B. (1997)
accessiblc, palatablc, relevant and useful. The 'Systematic reviews: synthesis. of best evidence for
linking of science base with practice remains clinica! decisions', Annals ot" Internai Medicine,
a key challenge for management and orga- 126(5): 376--80.
nization studics. Thercforc. in this chapter. Cooper, H.M. and Hedges, L.V. (1994) The Handboak
we have identifìed alternative principles and of Research Synthesis, Russell: Sage Foundation.
Counsell, C. ( 1997) 'Formulating questions and locating
outlined a methodology designed to furthcr
primary studies for incluslon in systematic reviews',
enhance and strengthen both management
Annals of Internai Medicine, 127(5): 380-7.
practice and management research. Thc Criticai Appraisal Skills Programme. (2008) Home
methodology offers the potential to couple IHomepage of Publìc Health Re$OUrce Unit (PHRU)i,
the best available academic evidence with IOnline]. Available: http://www.phru.nhs.uk/Pages/
the judgement and experience of practition- PHD/CASP.htm [February 2008).
ers in the true tradition of evìdcnce-based Davies, H.T.O., Nutley, S.M. and Smith, P.C. (1999)
practice. 'Editoria!: what works? the role of evidence in
688 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS

public sector policy an d practice', Public Money and Learmonth, M. and Harding, N.(2006) 'Evidence-based
Management, 19( 1): 3-5. management: the very idea', Public Administration,
Davi es, H.T. and Crombie, I.K. (2001) What is a System- 84(2): 245-66.
atic Review?, Hayward Medicai Communications. Light, R.J. and Pillemer, D.B. (1984) Summing Up: The
Denyer, D., Tranfield, D. ar1d van Aken, J.E. (2008) Science ofReviewing Research, Harvard, MA: Harvard
'Developing design propositions through research University Press.
synthesis', Organization Studies, 29(3): 393-415 . Macdonald, G. (1996) 'Ice therapy: why we need ran-
Dixon-Woods, M., Agarwall, S., Young, B., Jones, domised controlled trials', What Works. pp. 16-32.
D. and Sutton, A. (2006) Integrative Approaches Moher, D., Jadad, A.R., Nichol, G., Penman, M.,
to Qualìtative and Quantitative Evidence, London. Tugwell, P. and Walsh, S. (1995) 'Assessing the
Health Development Agency. quality of randomized controlled trials: an annotated
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B. bibliogra phy of scales and checklists', Contro/led
and Sutton, A. (2005) 'Synthesising qualitative and clinica/ trials, 16(1): 62-73.
quantitative evidence: a review of possib\e methods', Morrei!, K. (2008) 'The narrative of '' evidence based"
Journal of Health Services Research and Policy, management: a polemic', Journaf of Management
10(1): 45-53. Studies, 45(3): 613-35.
Estabrooks, C.A., Field, P.A. and Morse, J.M. (1994) NHS Centre for Reviews and Dissemination. (2001)
'Aggregating qualitative findings: an approach to Undertaking systematic reviews of research on
theory development', Qualitative Health Research, effectiveness: CRD's guidance for those carrying out
4(4): 503-11 . or commissioning reviews, NHS Centre for Reviews
Evans, D. (2003) 'Hierarchy of evidence: a framework and Dissemination.
for ranking evidence evaluating healthcare interven- Noblit, G. and Hare, R. (1988) Meta-Ethnography:
tions', Journal of Clinica! Nursing, 12(1 }: 77-84. Synthesizing Qualitative Studies, london: Sage.
Greenhalgh, T. and Peacock, R. (2005} 'Effectiveness Oakley, A. (2000) Experiments in Knowing: Gender and
and efficiency of search methods in systematic Method in the Socia/Sciences, Cambridge: Polity Press.
reviews of complex evidence: audit of primary Oliver, S., Harden, A, Rees, R., Shepherd, J., Brunton, G.,
sources', British Medicai Joumal, 331 : 1064-65. Garcia, J. and Oakley, A. (2005) 'An emerging
Greenhalgh, T. (1997) 'Papers that summarise other framework for including different types of evidence
papers (systematic reviews and meta-analyses}', BMJ in systematic revìews for public policy', Evaluation,
{Clinica/ research ed.), 315(71 09): 672-5. 11(4): 428.
Greenhalgh, T., Robert, G., Bate, P., Kyriakidou, O. and Pawson, R. (2002) 'Evidence based policy: the promise
MacFarlane, F. (2005) Diffusion of lnnovations in of 'Realist Synthesis', Evaluation, 8(3): 340-58.
Health Service Organizations: A Systematic Uterature Pawson, R. (2006) Evidence-Based Policy: A Realist
Review, BMJ Books. Perspective, London: Sage Publications.
Hammersley, M. (200 1) 'O n "systematic" reviews of Pawson, R. and Tilley, N. (1997) Realistic Evaluation,
research literatures: a "narrative" response to Evans London : Sage.
and Benefield', British Educational Research Joumal, Petticrew, M. (2001) 'Systematic reviews from astron·
27(5): 543-54. omy to zoology: myths and misconceptions', British
Hamm ersley, M. (2004) 'Some questions about Medica/Journal, 322(7278): 98- 101 .
evidence-based practice in education', in G. Thomas Pittaway, L, Robertson, M., Munir, K., Denyer, D.
and R. Pring (eds), Evidence-Based Practice in and Neely, A. (2004) 'Networking and innovation:
Education, Maidenhead: Open University Press, a systematic review of the evidence', lnternational
pp. 133-49. Joumal of Management Reviews. 5(3-4): 137--68.
Harden, A, Peersman, G., Oliver, S., Mauthner, M. Popay, J., Roberts, H., Sowden, A., Petticrew, M.,
and Oakley, A. (1999) 'A systematic review of Arai, l., Rodgers, M. and Brinen, N. (2006)
the effectiveness of health promotion interventions Guidance on the Conduce of Narrative Symhesis
in the workplace', Occupational medicine (Oxford, ;n Systematic Reviews, A Product frorn the ESRC
England), 49(8): 540-8. Methods Programme, http://www.conted.ox.ac.
Hart, C. (1998) Doing a Literature Review: Releasing the uk/cpd/hea lthscie ne es/course s/s hort ... courses/qs r/
Social Science Research Jmagination, london: Sage NSguidanceV 1-JNoyes.pdf.
Publications. Popay, J., Rogers, A. and Williams, G. (1998) 'Rationale
Higgins, J.P.T. and Green. S. (updated September and standards for the systernatic review of qualitative
2006) 'Cochrane Handbook for Systematic Reviews literature in health services research', Qualitative
of lnterventions', The Cochrane Library, 4.2. 6. health research. 8(3): 341-51.
 PRODUClNG A SYSTEMATIC REVIEW 689

Rauch, A. and Frese, M. (2006) 'Meta-analysis as a The Campbell Collaboratìon , Home. Available: http://
tool for developìng entrepreneurship research and www.campbellcollaboration.org/ [February 2008].
theory', Entrepreneurship: Frameworks and Empirica/ The Cochrane Collaboration , Homepage [Homepage
lnvestigations from Forthcoming Leaders of European of The Cochrane Collaboration]. [Online]. Available:
Research. http://www.cochrane.org/ [February 2008].
Rousseau, D.M., Manning, J_ and Denyer, D. (2008) The Evidence for Policy and Practice lnformation an d Co·
'Evidence in management an dorganìzational science: ordinating Centre (EPPI-Centre) , Home [Homepage
assembling the field's full weight of scientific of EPPI CentreL [Online]. Avaìlable: http://eppi.ioe.
knowledge through syntheses'/ Annals of the ac.uk/cms/ [February 2008].
Academy of Management. 2(1 ): 47 5-515. Tranfield, D., Denyer/ D., Marcos, J. and Burr, M.
Sackett, D.L., Rosenberg, W.M., Gray, J.A., Haynes, (2004) 'Co-producing management bowledge',
R.B. an d Richardson, W. S. (1996) 'Evidence based Management Decislon, 42(3/4): 375.
medicine: what it is and what it is1ù', BMJ (Clinica! Tranfield, D., Denyer, D. and Smart, P. (2003)
research ed.), 312(7023): 71-2. 'Towards a methodology for developìng evidence-
Salipante, P., Notz, W. and Bigelow, J. (1982) informed management knowledge by means of
'A matrix approach to literature reviews', Research systematic review', British Journal of Management,
in OrganizationaiBehavior, 4: 321-48. 14(3): 207.
Sandelowskì, M., Docherty, S., and Emden, C. ( 1997) Tranfield, D. and Starkey, K. (1998) 'The nature,
'Qualitative metasynthesis: issues and techniques', social organìzation and promotion of management
Research in Nursing and Health, 20(4): 365-71. research: Towards po~icy', British Joumal of Manage-
Shadish, W. R. and Sweeney, R.B. (1991) 'Mediators ment, 9(4): 341.
and moderators in meta·analysis: there's a reason va n Aken, J.E. (2005) 'Management research as a design
we do n't !et dodo birds te il us whìch psychotherapies science: artìculating the research products of mode
should have prizes', Journal of consulting and clinica/ 2 knowledge production in management', British
psychology, 59(6): 883-93. Journa! of Management, 16( 1): 19.
Slavin, R.E. (1986) 'Best-evidence synthesis: an alter- Wallace, M. and Wray, A. (2006) Critica/ Reading
native to meta·analytic and traditional reviews', and Writing far Postgraduates, London: Sage
EducationaJ Researcher, 15(9): 5-11. Publications.
Solesbury, W. (2001) Evidence Based Po!icy: Whence Whitley, R. (1984) 'The fragmented state of
it Came and where lt's Going. ESRC UK Centre for management studies: reasons and consequences',
Evidence Based Policy and Practice, UK. The Joumal of Managemr:nt Studies, 21(3): 331.