Professional Documents
Culture Documents
Denyer and Tranfield 2009 - Producing A Systematic Review
Denyer and Tranfield 2009 - Producing A Systematic Review
Review
David Denyer and Davld Tranfield
might be expected to provide solid and • The extent to which systematic review methods
dependable evidence that is robust and has can or should be proceduralized an d the extent to
potential for transfer across different c:ontexts. which bias can be reduced or mitigated.
O n the other hand, if the review identifies • The validity and reliabil ity of synthesized evidence
produced by systematic reviews, and how such
knowledge gaps or inwngruent findings, then
evidence is used subsequently to inform policy,
thìs s ignìfies a research need and raises
practice, and future research.
questions for future research.
Major improvements in review techniques
over the last fifteen years or so have helped Other critics ha ve suggested that evidence-
to raise the profile of systematic review as based manageme nt is overtly managerialist
an important research methodology. Applica- (Learmonth and Harding, 2006) and fail s
tions of systematic review in medicai science to take into account situated judgernent and
have led the way and are often considered to ethics (Morrell, 2008). They also express a
reflect the most advanced thinki ng. A 'stan- concern that evidence-based management and
dard' approach to systematic review was systematic review pri vileges certain types of
developed initially in the medicai fìeld by the research and other forms, such as criticai
Cochrane Collaboration (2008), and has been theory, are precluded.
followed later by other consortia dedicared to We apprec iate many of these concerns
commissionìng and disseminating systematic and challenges and take the view that it is
reviews. unwise to adopt, in an uncritical fa<;hion,
A Systematic review has been argued to that which has proved useful in other fields.
bring replicable, scientific, and transparent Consequently, out purpose in this Chapter
approach, which seeks to minimize bias (NHS is to explain how and in what ways the
Centre for Reviews and Disseminati on, 200 l ) notion of evidence-based management mìght
and requires reviewer s to summarize ali benefit the field through improved q uality
existing information about a phenomenon in a and rigour, by defining a bespoke and fit
thorough and unbiased manner. .\1ore widel y, for purpose fonnat for re view. Differences in
systematic reviews bave been argued also contex t are vita!, for example, in medicine.
to have value in collating and synthesizing there has been a gene rai consensus regarding
existing evidence across a wide range of what constitutes an a ppropriate methodology
settings (including the social sciences) and to be used for evaluating evidence from
empirica! methods (Petticrew, 2001 }. primary studies to be included (Evans, 2003).
On the otherhand the use of systematic In fields where the value of quantitative,
review, particularly ils traditional form, has qualitative, and theoretical contributio ns are
been critìcized when applied to the social equally respected (such as management and
sciences (Hammersley, 200 l ). U sually, these organìzation studies), reviewers are likely
criticisms have centred on uncritical mimetic to need more inclusive and comprehens ive
application encouraging the simple transfer quality criteria an d novel approaches to
of the medicai model of systematic review, research synthesis. Such inherent epistemo-
exemplified by the Cochrane Collaboration logica! divers ity necessitates an alternati ve to
(2008), into other fielcls. Critics have placed ' C ochrane-style ' systematic rcviews. T here-
four sers of arguments at the centre of this fore, we argue for the development of a
debate concerning: bespoke and fit for purposed methodology,
which can cope with the variety and richness
of research desig ns, purposes, and potential
• Competing ontological and epistemologica! posi-
tions related to the traditions of different end uses of management and organization
disciplines. studies reviews.
• Different perspectives on the systematic review Despite these differences our point of
methodology regarding the types of study to be departure is that systcmatic review methodol-
included in a review. ogy has proved demonstrably worthwhile in
PRODUCING A SYSTEMATIC REVIEW 673
many natura! and social science fields, and be seen as a response to the increasing
that would be wise to consider the lessons demands to organize knowledge into a format
researchers in management and organiza- that is rigorous and reliable as well as
tion studies already learned. Consequently, make a difference to practice. Evidence-
this chapter first outlines the principles of based practice requires decision making and
systematic review developed elsewhere and action to be infom1ed by the best and robust
evaluatcs their relcvance for the fìeld of evidence produced by high-quality research
management and organìzation studies. We studies. In addition to the tradìtional approach
discuss both the origins of systematic review of conducting new empirica! studies, many
and its roots in evidence-based polìcy and natura! and socìal sdence fields ha ve tumed to
practice and also examine the methodological reviews of existing primary studies reported
challenges to developing an evidence base in the lìterature to address important and
in management and organi.zation studies. We pressing questi o ns. In this demanding context,
note that some of the key principles of sys- systematic review increasingly has become
tematic revìew that are manifest in fields such a standard method for locating, selecting,
as medicine, often remai n elusive and perhaps and appraising research and transferring the
undesirable for the field of management and synthesized findings not only to researchers
organization studies. Second, and on the basis and academics but also to practitioners
ofthis considered reftection, we offer a revised and policymakers in a digestible format to
and (to an extcnt) alternative frarnework and inform action.
set of principles for developing transparent, Evidence-based policy and practice (EBPP)
inclusive, explanatory and heuristic reviews constitutes a movement that can be traced to
for use in our fìeld. Finally. we explore the cari y t970s (Cochrane, 1972 ). EB PP i n
how the output from systcmatic reviews medicine can be defined as the 'conscientious,
might be used to inform policy, practice and explicit and judicious use of current best
furthcr rcscarch. In doing so, we argue that evidence in making decisiom;' about the
the methods of review should be selected care of individuai patients (Sackett et al.,
according to thc purpose and the nature 1996: 1). Since thc early 1980s, rcsearchers
and characteristics of the field of study in health care have been cngaged in a
(e.g. degree of fragmentation, plurality of programme of systematic reviews on the
methods, etc) and the nature of the evidem:e effectiveness of ìnterventions. TI1e Cochrane
availahle. We argue the case for combining Collaboration was fmmed in 1992 to gather.
the many facets of systematic review with evaluate, and disseminate research evidence
aspects of realist synthesis {Pawson, 2006). and developed a sophisticatcd and sub-
Harncssing the strength of both of thesc stantial set of gu idei ines for conductìng
approachcs offers an important methodology systematic reviews. Since then, rescarchers
for management and organization studies that have developcd and reflned the systcm-
has the potential to infonn policy. practice, atic review methodology with thc aim of
and future research. improving both the rigour ancl reliability of
the review process, and organ[zing knowl-
edge into a l.tseable forrnat. The results
The origins and principles of
of systematic reviews have become well
systematic review established as legitimate evidence on which
The idea of revicwing extsung research to base policy and practice. Website.s, such
studies for thc purposc of informing policy, as the Cochrane database (2008b), now
practice. and rescarch is not ncw. However, offer clinician.s a practical solution to the
over recent years, thcrc has heen a prolif- problem of staying up to date with the latest
cration of formai approachcs anci systematic research.
rne!hocls for locating, selecting, appraising, Given the relative success of evidence-
synthcsizing:, and reporting evidence. This can based approaches in medicine, the core
674 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS
more reliable results upon which to draw con- Table 39.1 The hierarchy of evidence in
clusions and make decisions (Amman et al., medicai sdence (Davies, Nutley and Smith,
1992). Cooper and Hedges (1994: ix) also 1999: 11)
argue that the 'intended result is a research H Systematic review and meta-analysìs of two or
more double blind randomized controlled trials.
synthesis that can be replicated by others'.
1-2 One or more large double-bl ind randomized
However, within a review of social science controlled trials.
literature, it is generally recognized that 11-1 One or more well-conducted cohort studi es.
professionaljudgment and interpretation play 11-2 One or more well-conducted case·control studìes.
an i mportant role and cannot be eliminated or 11-3 A dramatic uncontrolled experiment.
Ili Expert committee sitting in review; peer leader
replaced by proceduralization. Hamme rsley
opinion.
(200 l: 6) argues that: IV Personal experience .
diversity and richness whìch are a major despite subtle differences in effccts or lack
strengths considering that: of correspondence in the populations or
study contexts such as variation in form
Organizations are fuzzy, ambiguous. complex of implementation, measured outcomes, or
socially constructed systerns that cannot be well contextual factors that inftuence observed
understood from a single perspective (Denyer et al., effects (Rousseau et al., 2008). Yet, the
2008ì.
danger remains that unless primary studies
carefully report implementation processes
However, there is no do ubt that there ìs an d context, aggregati ve syntheses can mask
little agreement aver what represents high the mechanisms underlying effects (Pawson,
quality evidence across this field (Trantìeld 2002). Advocates of meta-analysis (Rauch
and Starkey, 1998) and certainl y there is an d Frese, 2006; Shadish and Sweeney, 1991 )
robust opposìtion to prioritizing one research argue that heterogeneity can be dealt with by
method over another. including mediators and moderator variables
into statistical techniques. A mediator is an
Aggregative intermediate variable that links an interven-
The third key principle suggests thatalongside tion with an outcorne. A moderator accounts
systematic review, statistica! meta-analysis for contextual differences, eithcr in terms
has become the preferred approach to research of the population o r the circumstances. A
synthesis in many domains. A meta-analysis host of demographic and contextual factors
extracts and aggregates data from separate can be specifìed and accountcd for in the
studies to increase the etfective sample size, analysis. The problem with this technique
in ordcr to calculate an overall effect size is that each of the likely mediators and
for an intervention. In order to perform moderators must be identifìed at the outset
a meta-analysis, it is important to identify of the review, and it is likely that in a
whether or not resu1ts from studies are con- managerial and organizational context the list
sistent one with another (i.e. homogeneous) of variables could be extensive. Further, each
or inconsistent (e.g. heterogeneous). Data ofthe studi es included in the review must have
from individuai studies are extracted · and comparable mediators and moderators, thus
tabulated in systematic review reports to severely limiting the number of studies on any
display thc impact of potential sources of specific topic that are amenable to aggregative
heterogeneity, e.g., study type, study quality, synthes is.
and sample siz:e. Additionally. despite advances in meta
lf the data extracted from these studies analytical techniques, many questions are
meet cert.ain requirements (the most impor- not amenable to aggregative approaches.
tant being a high level of homogeneity of In complex systems comprised of a large
effect measures across studies) then they variety of dynamically interacting variables
can be aggregated. Pooling data provides a such as can be found in management and
larger sample, increasing the possibility of organizational studies, the value of statistica)
detecting rea! effects that individuai smaller synthesis adoptcd as a single point solution is
studies are unable to ascertain. However, often questionable. Many systematic reviews,
aggregating studies can also cause problems even in medicine, do not contain a meta-
if the researcher fails to account for subtle analysis. Dixon-Woods et al. (2006) assert that
diffcrences in study design and study context, alternative approaches to synthesis, including
since it is possible to reveal minor bia<;es as qualitative techniques, can be incorporated
well as true effects. into systematic reviews.
Therefore, a key problem with aggregative
synthesis is that primary studies are rarely Algorithmic
entirely homogeneous. However, there are In health care, systematic reviews tend to
strong arguments that studics can be pooled focus o n questions relating t o the effecti veness
PRODUCING A SYSTEMATIC REVIEW 677
First, reviewers must be open and explicit A clear audit trai! is required linking the
about the processes and methods employed in claims made by the authors of the review with
the review. This enables readers to determine the existing evidence. According to Pawson .;.
:: '
precisel.y its scope and boundaries. Through- (2006: 79): :~ ' .
(Tranfield et al., 2003). For example, the stakeholders ca n look behind an existing review, to
; ,·
practice of making the review m ethods assure themselves of its rigoJr an d validity, rel iabilìty i
;, _
and verifiability of its f indings anè conclusions.
explicit before the review commences is a
constructive exercise. Producing a review .:
protocol enables the reviewcr to gai n feedback The third element to transparency necessi- i:..·
~:
on the proposed methods and highlight any tates the reviewer making clear the assump-
obvious errors or omissions so that they tions underpinning the review and engag- ·. i .
circumstances. the protocol can be used to beliefs, selectively citing studies along the
document and justify these changes. In the way' (Davies and Crombie, 2001: 2). As such,
final systematic revicw methodolog,y scction, it is important that reviewers make cxplicit
ìt is customary to produce an overview of the their value stance towards the a<;pect of the
main changes between the originai and fina! social world they are studying (Wallace and
protocols. Wray, 2006).
The second expression of transparency
concerns the presentation of the findings of the lnclusivity
review i n sue h a way that t here are clear links Our second key principle cmphasizes that
between the evidence found and the revicw- systematic reviewers in the field of omaniza-
ers' conclusions and recomme.ndatìons. As tion and management are likcly to en~ounter
noted by Slavin ( 1986 : 17): difficulties whcn appraising the quality of
information sourccs. Authors of articles, evcn
No review procedure can make errors impossible those published in some of the highest impact
or eliminate any ch ance that rev1ewers' biases will journals, sometimes fail to report on the
affect the conclusions drawn . However, applìca- methods of data collection and analysis in suf-
tions of best-evidence synthesis should at least fìcient dctail, rendering impossible attempts
make review procedures clear to the reader and
should previde t he reader with enough information
to a'i.sess study quality. Within management
about the primary research an which the review is and organization studies, there is also little
based to reach independent condusions. uniformity in methods of data collection
680 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS
and analysis. Studies rarely address identica! the subfield(s) involved, justify the reasons
questìons and samples vary in terms of the for inclusion/exclusion of studies, apply the
populations, sizes, study context, and the data criteria to all relevant studies, and communi-
reported. cate the warrants underpinning their claims
Boaz and Ashby (2003: 4) suggest that (Wallace and Wray, 2006).
rather than a hierarchy of evidence, the
selection of articles can be based on the Explanatory
criterion 'fit for purpose'; this, 'helps us to Our third key principle addresses the vex.ing
get away from the technocratic preoccupation question of synthesis. The evidence-based
with elegant research designs and allows for community in the UK (see for example
notions of appropriateness to be added into the Evìdence Network website: http://www.
the appraisal of research'. Pawson (2006) evidencenetwork.org/) has been working hard
also argues that the researcher should simply to develop severa! alternative approaches
ask whether or not the literature retrieved to meta-analysis as a mode of synthesis.
adds anything new to our understanding Particularly, progress has been made in
of the intervention. Pawson 's approach has narrative reviewing (see Popay et al., 2006;
no hierarchy of evidence. The worth of Greenhalgh et al., 2005) and the use of the
the study can only be determined through metaethnographic approach of Noblit and
the synthesis by evaluating each study's H are ( 1982; see Campbell et al., 2003 ).
contribution to theory building, prioritizing Evidence syntheses os ing these approaches
the vital evidence from primary studies as are tasked w ith the construction of larger
the originai researchers' interpretations and narrati ves and more generalizable theory
explanations, not just the results. Hence, (Estab rooks et al., 1994; Sandelowski et al.,
Pawson encourages the scope of the review to 1997). Interpretive and explanatory synthesis
include a wide range of studies, research types extracts descripti ve data and exemplars from
an d data forms to promote a full understanding individuai studies, building them into a
of the phenomenon of interest. mosaic or map (Hammersley, 2004). The
Salipantc et al. ( 1982) ha ve also arg ued reviewer is involved in juxtaposing the
that the inclus ion of a broad variety of e vidence, which can be quantitative, qu;ùita-
sources in a review can compensate for tive , or theoretical. from one study with that
researcher value judgments and uncontrolled of another.
validity threats. Heterogeneity also makes Unlike aggregative .synthesis which .seeks
possible the investigation of contextual fac- to avoid or mitigate bias. imcrpretive and
tors inftuencing rhe study's design and its explanatory syntheses are 'active' and ' cre-
findings. They argue that the more het- ative' methods (Pawson, 2006) that go beyond
erogeneous the distribution of uncontrolled a descriptive rcporting of thc cvidence. Thc
validity threats in a set of similar fìndings, the process is one of conceptual innovation and
grearer the validity of the findings from reinterpretation (Campbell et a l., 2003), while
the set. Following this logic, we suggest attempting to preserve the origina) stully's
that reviewers in management and organi- integrity or wholeness {Pawson, 2006). As
zation studies should beware of excluding such, the synthesis provides a rcasiblc
studies solely on the basis of adherence explanation of the study fìndings rathc:r
to the pursuit of absolute epistemological tha n a replicablc cxplanation ( Noblit and
standards. In particular, reviewers are best Hare, 1988). Thereforc. synthcs is involves the
advised to guard against using proxies for process of bringing the picces from indi vidual
research quality such as the quality rating texts together to make a whole that should be
of journals as a basis for exclus ion. We more than the su m of tbc pa11s.
do, however, agree that systematic reviewers Pawson (2006) has introduccd thc notion
in management and organization studies of realist synthesis by using existing primary
de vise quality checklist(s) appropriate to studies to build a nasccnt thcory of how
PRODUCING A SYSTEMATIC REVIEW 681
an intervention may work. This will include than providing a detailed solution to a specific
'conjectures on the generative mechanisms problem. Heuristic rules will almost always
that change behaviour, ideas on the contexts require the application of informai judgment
that inftuence its operation, an d hunchcs about or experience to contextualise action. Denyer
the different outcome patterns that ensue' et aL (2008) suggest that systematic reviews
(Pawson, 2006: 73 ). The texts that are to of the existing published science base in
be included in the synthesis are 'regarded as management and organization studies can
case studies, whose purpose is to test, revise develop knowledge that managers can use
and retine the preliminary theory' (Pawson, to design solutions to problems in their
2006: 74). The review entaìls the systematic field. In management research, rather than
organization of the data into formats that presenting the truth, in the form of 'valid
allow sumrnary. This body of evidence is evidence', managers rnay be presented with
then probed, sifted, coded, and cross tabulated some 'clues/ideas'. 'tools', and 'methods'
in nurnerous ways. Each relevant pubhshed that may help to guide design for effective
artide is described and discussed in terms implementatìon. Decìding the degree to whìch
of its contribution to the emerging theory the findings presented in any particular review
(Pawson, 2002: 345): can inform practice is therefore a matter of
judgment for the practitioner.
. . . the rev1ewer's basic task 1s to sift through the
mixed fortunes of a programme attempting to
discover those contexts ((+) that have produced Applying the new principles
so!id and successful outcomes (O) from those
contexts (C) that induce failure (0-). The review Step 1: Question formulation
process is then repeated across a number of
As with any research, the primary decision
additional stud1es featunng the same underly-
lng mechanism (M), wrth the aim of gathering
in preparing a review is to establish its
together the vanous permutations of success focus (Light and Pillemar, 1984). This is
and failure the aim is to differentiate and done most effectively by asking clearly
accumulate evidence of positive and negative CMO framed questions (Cooper and Hedges, l 994 ).
configurations.
By clearly formulating the question, criteria
for primary study inclusion in thc review
Generalization is sought not in terms of the become clear:
associations among variables but in terms of
the rolc and i mpact of generati ve mechanisms
A good systematic review is based on a wel!-
that play out in different ways over time. formulated, answerable questìon. The question
guides the rev!ew by definìng which studies wil! be
Heuristic w.c!uded, what the search strategy to identify the
The output of a systematic review m man- relevant primary studies should be, and which data
need to be extracted from each study. Ask a poor
agement wiil allude to what works. and
question and you will get a poor rev1ew' (Counsell,
to why or how the relation occurs and 1997: 381 ).
in what circumstances, but is likdy to be
relatively abstract and is best regarded by the In order to clarify review questions. inform
practitioner as a 'design exemplar' (v an Aken, the rcview process an d improve the utilization
2005 ). Gi vcn the compìexity of organizational of findings from systematic revicws, it is
setting:s. such outputs are likely t o be heuristic. best to involve a broad rangc of stakeholders
A heuristic rule may help in sol vìng a problem, in the developrnent of review questions and
hut is not guaranteed to provide a detailed procedures (Tranficld et al., 2004). Petticrew
solution. Outputs of systematic revìew in (200 l: HX)} argues that,
management and organization studies are
likely to be rulcs, suggestions, guides, or pro- 'potential users of systematic rt:views, such as
totype protocols that rnay be useful in making consumers and policy makers, can be involved in
progress toward a solution of a problem rather the process. This can help to ensure that reviews
682 THE SAGE HANDBOOK OF ORGANfZATfONAL RESEARCH METHODS
are welf tocused, ask relevant questions, and are interesting example of how organizational
disseminated effectively to appropriate audiences'. forms are constrained by context:
Advisory groups can assist with defining Bureaucr aci es organi ze work routines in certa in
the broad topic area to be investigated and w ays and previde tightly specified employment
identifying the specific areas within that topic functions for t heir workforces, but whether any
that would be most useful to scrutinize in of th is happens depends on t he availab ility of
work, and ultimately, on the overall economie
depth. Involving experts is particularly impor-
heath of the nation . The efficacy of bureaucratic
tant when the literature is sparse. The review management is also contingent on the type of work
pane l bcst includes people with academic carried out. In sectors that th rive on innovat ion and
knowledge and practìcal expertise in the entrepreneurial activity, the applicability of fixed
subject area such as Iibrarians and information duties to fixed roles is likely to flop.
in tenns of internai dynamics and their produce a scoping study of the field. Scoping
cxperiencc of being a team member. lt studies are exploratory reviews, since what
is, therefore, essential to defìne key tenns to include and exclude is not self-evident.
during the question formulation phase. Since The purpose of the scoping study is to make
defìnitions and concepts are value laden, clear the existing basis for the re/-iearcher's
differcnt perspectives and bcliefs need to work and specify wherc and how the proposed
be madc transparent. Alth()ugh the review systematic review fits into the current body
questions are specificd a priori, a ce1tain of knowledge. The scoping study should help
ftexibility and modification of questions may to define concepts and determine the revicw
be necessary as the reviewer gains a fuller qucstions to be addressed.
understanding of the problem.
Until a very precise question(s) for a Step 2: Locating studies
systematic review is specified, detailed work Systematic reviews aim to locate, select,
on a systematic review is best not started. and appraise as much as possible of the
It is impractical and unfair for novice research relevant to the particular review
researchers to begin a systematic review question(s). The methods used to find studi.e s
without a significant grounding in t.hcir field. (database searches, searches of specialist bib-
Tranfidd et al. (2003) suggest that prìor Iiographies, hand-searching of likely journals,
to a systematic revicw, novices and less and attempts to track down unpublished
experienced researchers are best advised to research) necd to be reported in some detail.
684 THE SAGE HANDBOOK OF ORGANIZATIONAL RESEARCH METHODS
reasons for the indusion and exclusion of Wallace and Wray (2006) and Solesbury
articles. [f the search is well focused then (2001):
within a short time it is usually possible to
produce an exhausti ve list of reasons for inclu- (1) What are the generai details of the study -
sion and exclusion. The reasons for inclu- author, title, journal, date, language?
sion and exclusion then form the selection (2) What are you seeking to understand or decide ; ·. •j
i
·..
public sector policy an d practice', Public Money and Learmonth, M. and Harding, N.(2006) 'Evidence-based
Management, 19( 1): 3-5. management: the very idea', Public Administration,
Davi es, H.T. and Crombie, I.K. (2001) What is a System- 84(2): 245-66.
atic Review?, Hayward Medicai Communications. Light, R.J. and Pillemer, D.B. (1984) Summing Up: The
Denyer, D., Tranfield, D. ar1d van Aken, J.E. (2008) Science ofReviewing Research, Harvard, MA: Harvard
'Developing design propositions through research University Press.
synthesis', Organization Studies, 29(3): 393-415 . Macdonald, G. (1996) 'Ice therapy: why we need ran-
Dixon-Woods, M., Agarwall, S., Young, B., Jones, domised controlled trials', What Works. pp. 16-32.
D. and Sutton, A. (2006) Integrative Approaches Moher, D., Jadad, A.R., Nichol, G., Penman, M.,
to Qualìtative and Quantitative Evidence, London. Tugwell, P. and Walsh, S. (1995) 'Assessing the
Health Development Agency. quality of randomized controlled trials: an annotated
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B. bibliogra phy of scales and checklists', Contro/led
and Sutton, A. (2005) 'Synthesising qualitative and clinica/ trials, 16(1): 62-73.
quantitative evidence: a review of possib\e methods', Morrei!, K. (2008) 'The narrative of '' evidence based"
Journal of Health Services Research and Policy, management: a polemic', Journaf of Management
10(1): 45-53. Studies, 45(3): 613-35.
Estabrooks, C.A., Field, P.A. and Morse, J.M. (1994) NHS Centre for Reviews and Dissemination. (2001)
'Aggregating qualitative findings: an approach to Undertaking systematic reviews of research on
theory development', Qualitative Health Research, effectiveness: CRD's guidance for those carrying out
4(4): 503-11 . or commissioning reviews, NHS Centre for Reviews
Evans, D. (2003) 'Hierarchy of evidence: a framework and Dissemination.
for ranking evidence evaluating healthcare interven- Noblit, G. and Hare, R. (1988) Meta-Ethnography:
tions', Journal of Clinica! Nursing, 12(1 }: 77-84. Synthesizing Qualitative Studies, london: Sage.
Greenhalgh, T. and Peacock, R. (2005} 'Effectiveness Oakley, A. (2000) Experiments in Knowing: Gender and
and efficiency of search methods in systematic Method in the Socia/Sciences, Cambridge: Polity Press.
reviews of complex evidence: audit of primary Oliver, S., Harden, A, Rees, R., Shepherd, J., Brunton, G.,
sources', British Medicai Joumal, 331 : 1064-65. Garcia, J. and Oakley, A. (2005) 'An emerging
Greenhalgh, T. (1997) 'Papers that summarise other framework for including different types of evidence
papers (systematic reviews and meta-analyses}', BMJ in systematic revìews for public policy', Evaluation,
{Clinica/ research ed.), 315(71 09): 672-5. 11(4): 428.
Greenhalgh, T., Robert, G., Bate, P., Kyriakidou, O. and Pawson, R. (2002) 'Evidence based policy: the promise
MacFarlane, F. (2005) Diffusion of lnnovations in of 'Realist Synthesis', Evaluation, 8(3): 340-58.
Health Service Organizations: A Systematic Uterature Pawson, R. (2006) Evidence-Based Policy: A Realist
Review, BMJ Books. Perspective, London: Sage Publications.
Hammersley, M. (200 1) 'O n "systematic" reviews of Pawson, R. and Tilley, N. (1997) Realistic Evaluation,
research literatures: a "narrative" response to Evans London : Sage.
and Benefield', British Educational Research Joumal, Petticrew, M. (2001) 'Systematic reviews from astron·
27(5): 543-54. omy to zoology: myths and misconceptions', British
Hamm ersley, M. (2004) 'Some questions about Medica/Journal, 322(7278): 98- 101 .
evidence-based practice in education', in G. Thomas Pittaway, L, Robertson, M., Munir, K., Denyer, D.
and R. Pring (eds), Evidence-Based Practice in and Neely, A. (2004) 'Networking and innovation:
Education, Maidenhead: Open University Press, a systematic review of the evidence', lnternational
pp. 133-49. Joumal of Management Reviews. 5(3-4): 137--68.
Harden, A, Peersman, G., Oliver, S., Mauthner, M. Popay, J., Roberts, H., Sowden, A., Petticrew, M.,
and Oakley, A. (1999) 'A systematic review of Arai, l., Rodgers, M. and Brinen, N. (2006)
the effectiveness of health promotion interventions Guidance on the Conduce of Narrative Symhesis
in the workplace', Occupational medicine (Oxford, ;n Systematic Reviews, A Product frorn the ESRC
England), 49(8): 540-8. Methods Programme, http://www.conted.ox.ac.
Hart, C. (1998) Doing a Literature Review: Releasing the uk/cpd/hea lthscie ne es/course s/s hort ... courses/qs r/
Social Science Research Jmagination, london: Sage NSguidanceV 1-JNoyes.pdf.
Publications. Popay, J., Rogers, A. and Williams, G. (1998) 'Rationale
Higgins, J.P.T. and Green. S. (updated September and standards for the systernatic review of qualitative
2006) 'Cochrane Handbook for Systematic Reviews literature in health services research', Qualitative
of lnterventions', The Cochrane Library, 4.2. 6. health research. 8(3): 341-51.
PRODUClNG A SYSTEMATIC REVIEW 689
Rauch, A. and Frese, M. (2006) 'Meta-analysis as a The Campbell Collaboratìon , Home. Available: http://
tool for developìng entrepreneurship research and www.campbellcollaboration.org/ [February 2008].
theory', Entrepreneurship: Frameworks and Empirica/ The Cochrane Collaboration , Homepage [Homepage
lnvestigations from Forthcoming Leaders of European of The Cochrane Collaboration]. [Online]. Available:
Research. http://www.cochrane.org/ [February 2008].
Rousseau, D.M., Manning, J_ and Denyer, D. (2008) The Evidence for Policy and Practice lnformation an d Co·
'Evidence in management an dorganìzational science: ordinating Centre (EPPI-Centre) , Home [Homepage
assembling the field's full weight of scientific of EPPI CentreL [Online]. Avaìlable: http://eppi.ioe.
knowledge through syntheses'/ Annals of the ac.uk/cms/ [February 2008].
Academy of Management. 2(1 ): 47 5-515. Tranfield, D., Denyer/ D., Marcos, J. and Burr, M.
Sackett, D.L., Rosenberg, W.M., Gray, J.A., Haynes, (2004) 'Co-producing management bowledge',
R.B. an d Richardson, W. S. (1996) 'Evidence based Management Decislon, 42(3/4): 375.
medicine: what it is and what it is1ù', BMJ (Clinica! Tranfield, D., Denyer, D. and Smart, P. (2003)
research ed.), 312(7023): 71-2. 'Towards a methodology for developìng evidence-
Salipante, P., Notz, W. and Bigelow, J. (1982) informed management knowledge by means of
'A matrix approach to literature reviews', Research systematic review', British Journal of Management,
in OrganizationaiBehavior, 4: 321-48. 14(3): 207.
Sandelowskì, M., Docherty, S., and Emden, C. ( 1997) Tranfield, D. and Starkey, K. (1998) 'The nature,
'Qualitative metasynthesis: issues and techniques', social organìzation and promotion of management
Research in Nursing and Health, 20(4): 365-71. research: Towards po~icy', British Joumal of Manage-
Shadish, W. R. and Sweeney, R.B. (1991) 'Mediators ment, 9(4): 341.
and moderators in meta·analysis: there's a reason va n Aken, J.E. (2005) 'Management research as a design
we do n't !et dodo birds te il us whìch psychotherapies science: artìculating the research products of mode
should have prizes', Journal of consulting and clinica/ 2 knowledge production in management', British
psychology, 59(6): 883-93. Journa! of Management, 16( 1): 19.
Slavin, R.E. (1986) 'Best-evidence synthesis: an alter- Wallace, M. and Wray, A. (2006) Critica/ Reading
native to meta·analytic and traditional reviews', and Writing far Postgraduates, London: Sage
EducationaJ Researcher, 15(9): 5-11. Publications.
Solesbury, W. (2001) Evidence Based Po!icy: Whence Whitley, R. (1984) 'The fragmented state of
it Came and where lt's Going. ESRC UK Centre for management studies: reasons and consequences',
Evidence Based Policy and Practice, UK. The Joumal of Managemr:nt Studies, 21(3): 331.