Professional Documents
Culture Documents
Stanley and Doucouliagos (2012) - Meta-Regression
Stanley and Doucouliagos (2012) - Meta-Regression
1 E-Research
Transformation in scholarly practice
Edited by Nicholas W. Jankowski
3 Multi-Sited Ethnography
Problems and possibilities in the translocation of research methods
Edited by Simon Coleman and Pauline von Hellermann
1 Introduction 1
1.1 The Tower of Research 1
1.2 A historical sketch of meta-regression analysis 5
1.3 Practical examples 9
1.4 Plan of the book 10
Notes 154
References 168
Index 180
Figures
We are especially grateful for comments and suggestions from Margaret Giles,
Jost Heckemeyer, Julian Higgins, Stian Skår Ludvigsen, Debdullal Mallick, Jon
Nelson, Geoff Pugh, Randy Rosenberger, and Hossam Zeitoun. Our research
collaborators over the years have been instrumental in the development of our
ideas: Janto Haman, Steve Jarrell, Patrice Laroche, Martin Paldam, Andrew
Rose, Randy Rosenberger, and Mehmet Ulubasoglu. Furthermore, we need to
acknowledge the ideas, feedback and support that we received from numerous
scholars during various seminars and MAER Network colloquia. Needless to
say, any errors or omissions are solely our responsibility.
1 Introduction
This is but the start of their undertakings! There will be nothing too hard for
them to do. Come, let us go down and confuse their language on the spot so
that they can no longer understand one another.
(Genesis 11: 6–7)
Empirical results reported in economics journals are selected from a large set
of estimated models. Journals, through their editorial policies, engage in some
selection, which in turn stimulates extensive model searching and prescreen-
ing by prospective authors. Since this process is well known to professional
readers, the reported results are widely regarded to overstate the precision of
the estimates, and probably to distort them as well. As a consequence, statisti-
cal analyses are either greatly discounted or completely ignored.
(Leamer and Leonard, 1983: 306)
Each of these model specification choices affects the reported results, often by a
lot, and there is no reliable way to know which model specification is correct.3
Enter meta-regression analysis.
Meta-regression analysis can explicitly model the effects of observed model
specification variation and thereby directly estimate the associated misspecifica-
tion biases. Accommodating and correcting the biases associated with applied
econometrics is the central objective of MRA. Meta-regression analysis is a
systematic and comprehensive review of all existing, yet comparable, empirical
evidence. It allows the systematic reviewer to model and estimate any explana-
tory or biasing factor for which information or a proxy is available and thereby
filters out their influence on our scientific knowledge. This applies to selection
as well. Although MRA can accommodate the conventional sample selection
4 Introduction
biases that are often seen in empirical econometrics (Heckman, 1979; Stanley
and Jarrell, 1998), it can do much more.
Publication selection, as opposed to sample selection, arises if researchers,
editors, or reviewers use statistical significance as one model selection criterion.
Publication biases have been identified in the majority of economics areas of
research and often have important practical effects (Doucouliagos and Stanley,
2012).4 Because publication selection is caused by the process of conducting
empirical economic research itself, conventional econometrics is incapable of
correcting or estimating this effect. Hence, some “macro” perspective is required
that looks across an entire research field, and this is precisely what MRA provides.
Chapter 4 discusses how MRA can identify, estimate, and correct publication
selection bias, and subsequent chapters illustrate and explain how MRA can filter
out many other types of bias as well.
The purpose of this book is to introduce the tools of meta-analysis and meta-
regression analysis to business and economic researchers unfamiliar with their
use. Meta-regression analysis addresses the rising “Tower of Babel” that current
economics and business research has become. Evidence-based policy requires a
clear and objective assessment of the research record. Without a systematic and
objective way to summarize and understand current research, policy discussions
will be at the mercy of the subjective interpretation of our empirical knowledge.
Moreover, there is a real danger that vested interest or ideology will dominate the
discussion and thereby distort policy.
For example, it is clear that both of these forces dominated the anti-regulation
atmosphere in the USA that preceded the global 2008 financial meltdown. Alan
Greenspan, the former US Federal Reserve chairman, was a disciple of Ayn Rand
and a libertarian (Greenspan, 2007; Leonhardt, 2007). Greenspan has been
forthcoming about his free-market ideology. A case has been made that it was the
opposition to the regulation of derivatives by both Greenspan and the financial
industry that led to the worst recession in the USA since the Great Depression
(Public Broadcasting Service, 2009), and Greenspan admitted the error of his
ideology to the US Congress (Andrews, 2008).5
A more positive trend is that governmental agencies are funding dozens of
systematic reviews and meta-analyses of their programs and policies.6 In 2011,
the United Kingdom’s coalition government has renewed its pledge to protect
its international development aid from the spending cuts and to double its
international aid commitment. Needless to say, this puts the Cameron government
under considerable political pressure, not the least of which comes from their
party loyalists (Hennessy, 2011). In a climate of large cuts to domestic programs,
it is especially important to ensure that government policies and programs are
getting “value for money.” Here, too, MRA has an important role to play, because
it can offer an objective, comprehensive and rigorous summary and evaluation of
what is known, empirically, about a given intervention or policy.
In our view, we are at the dawning of a new era of empiricism in economics and
business. Even though the capacity of future empirical methods cannot be fully
known, there will remain conflict in what these methods reveal about specific
Introduction 5
business and economic effects. These phenomena are irreducibly contingent on
prevailing cultural and political institutions, and we live in dynamic societies.
Thus, an important role for meta-analysis is virtually assured.
If the past is any guide, future systematic reviews and meta-analyses will,
on occasion, find that strongly held economic theories are not supported by the
weight of empirical evidence. For example, minimum wage raises do not cause
lower employment in the US (Doucouliagos and Stanley, 2009) – see Chapters 4
and 5. In other cases, intentionally weak governmental policy (i.e. non-mandatory
regulation) will be found to have their intended effects – for example, chief
executive pay and corporate performance (Doucouliagos et al., 2012a).7
f = −2 ∑ ln P i (1.1)
i=1
Glass was interested in showing that psychotherapy had a beneficial effect. For a
couple of decades, the effectiveness of psychotherapy had been in great dispute
and both sides resorted to “vote-counting” hundreds of relevant papers (Hunt,
1997). Glass understood that merely counting the number of studies that found a
significant treatment effect was not a valid way to accumulate scientific evidence.8
Rather, Glass offered the “effect size,” g, as a means to compare the magnitude of
the empirical effects reported across studies:
__ __
Xe – Xc
g = ______
S (1.2)
where the numerator of this ratio is the average difference between the experimen-
tal and control groups on some relevant measure of effect, and S is the standard
deviation of this measure as seen in the control group. Glass’s g is a standardized
measure of effect that has no dimensionality; that is, no units of measurement.
As such, studies that employ different outcome measures (e.g. different scales of
mental health) can be directly combined and compared. Note also that Glass’s g
preserves the magnitude of this effect, not merely its direction or significance. If
the measured mental health of treated patients improves a lot, on average, relative
to the background variation in what happens to similar, untreated subjects, g will
be correspondingly large.
Introduction 7
Now, it is standard practice to report “effect size” in education and psychology
as a way to focus on the practical importance of an empirical finding, not only its
statistical significance.9 Cohen (1988) offers widely accepted guidelines for the
practical interpretation of effect size. When .2 < g < .5, there is a small effect. A
medium effect has .5 < g < .8, and a large effect is found when g exceeds .8.
Smith and Glass (1977) summarize hundreds of studies of psychotherapy and
show that it has a beneficial, if moderate, effect on patients’ mental health – on
average, g = .68. After Glass (1976), the use and further development of meta-
analysis slowly blossomed in psychology and medical research, two fields where
experiments often give differing results. Meta-analysis is now the accepted method
to summarize scientific knowledge; its results are often regarded as “definitive.”
See Hunt (1997) for a more comprehensive, yet delightfully readable, history of
the development and application of meta-analysis.
Empirical econometrics employs a variety of multiple regression techniques
to isolate the marginal effect of price, income, some intervention, or other eco-
nomic variable, holding a myriad of other factors constant. These partial and
marginal effects are what typically interests economists. As discussed above,
different statistical methods, models, and sets of independent variables are used
to estimate the marginal effect in question; thus, we always find much misspeci-
fication bias and large heterogeneity among reported econometric estimates. To
accommodate and filter out these biases and genuine heterogeneity, Stanley and
Jarrell (1989) proposed using essentially the same statistical tools which pro-
duce econometric estimates, to summarize and explain the observed variation in
these reported estimates. “Meta-regression analysis” was always conceived as
a “multivariate” means to summarize and explain multiple regression estimates
or transformations of these estimates.10 Economic meta-analysts believe that
observed econometric estimates are the product of complex multifaceted forces,
much like the observed economic phenomena, themselves.
Another advantage of MRA is that it uses essentially the same tools and
statistical methods as do the econometricians who produce empirical economic
estimates. Thus, econometricians have no rational basis upon which to object to
the heightened scientific scrutiny that MRAs offer. If there is some fundamental
weakness or limitation in MRA, then econometrics will likely suffer from very
similar problems.
Economists produce millions of empirical estimates each year, and they are
used by policy makers to design critical interventions (e.g. a stimulus package to
moderate a recession). Meta-regression analysis takes empirical economics seri-
ously and seeks to improve it. The point of departure of MRA is that empirical
economic estimates represent an important phenomenon worthy of deeper exami-
nation. Unsurprisingly, economists have been slow to accept this added level of
scrutiny. After all, what producer embraces an objective and critical assessment
of his products? Nonetheless, MRA has been widely accepted in recent years just
as quality assurance is conventional practice in manufacturing.
To provide a rough sketch of the trajectory of the discipline’s adoption of meta-
analysis, we searched EconLit for “meta-analysis” or “meta-regression” in either
8 Introduction
60
50
40
30
Studies
20
10
9
2
5
8
4
5
7
9
0
9
0
9
9
8
20
20
20
20
19
19
20
20
20
20
20
20
19
19
19
19
19
19
19
19
19
Year
the abstract or title among all papers that also concern “economics.” Figure 1.1
plots the growth of published meta-analyses in economics over its first 20 years,
while Figure 1.2 shows that a simple exponential growth model provides a rather
good fit (R2 = 0.88). The adoption of meta-analysis in economics is, on average,
growing at 18 percent per year. We know that these numbers of meta-analyses of
60
Observed
Growth
50
40
Studies
30
20
10
0
1985 1990 1995 2000 2005 2010
Year
The commonly held belief that research progress will be made if only we
“let the data speak” is sadly erroneous. ... it would be more accurate to say
that the data come to us encrypted, and to understand their meaning we must
first break the code.
(Hunter and Schmidt, 2004: xxxi)
Empirical studies and their estimates are scattered throughout a complex research
landscape. Some are published in prestigious peer-reviewed journals; others
exist only in unpublished working papers, dissertations or online. Still others are
known only to the researcher who produces them, never to be seen by any other
scholar. Relevant empirical estimates are generated from sophisticated structural
econometric models, reduced-form models, and also simple bivariate compari-
sons. Studies differ widely in terms of the control variables, data sources, and
estimation techniques employed. This multidimensional nature of research makes
deriving clear inferences and policy advice difficult. What we need is an efficient
set of tools to summarize, integrate, correct, and evaluate research findings.
Enter meta-analysis. Like all statistical techniques, data fuels meta-analysis.
However, “data” in the meta-analysis context are the complex products of the
research process. Typically, meta-data will be comprised of estimates of some
economic association (also known as “effect sizes”) linked to key dimensions
of the research process that produced these effects. Like any good empirical
study, meta-analysis commences with a theory, or a group of theories, that predict
associations. These theories are next investigated empirically in a research
literature. While it is this empirical literature that meta-analysts explore, it is the
underlying economic theory that shapes this empirical inquiry that is of ultimate
interest to economists. Hence, an indispensable component of a meta-analysis is a
thorough understanding of the underlying economic theories. This understanding
will shape the meta-analyst’s search for studies, coding of research, subsequent
statistical analysis, and ultimately the interpretation of the meta-analytic statistics
produced. Meta-analysis begins neither with statistics nor data, but rather with a
clear understanding of economic ideas.1 Theory provides the topographical map
of the terrain to be explored, while meta-analysis provides the tools for extracting
the precious ores, should they be present.
Identifying and coding meta-analysis data 13
This chapter focuses on the collection of the data that defines meta-analysis. In
particular, we discuss where to collect the data and what information to collect.
Chapter 3 explores alternative ways of summarizing research findings. More
complex statistical meta-analyses will be explored in subsequent chapters.
Non-English studies
Most authors will include only studies written in English. In other disciplines,
most notably in medicine, some effort is taken to include non-English studies.
We do not, in general, see this as a critical issue in economics. Most empirical
economics papers are actually written in English, so that any bias resulting from
omitting non-English studies should be of a second order. Moreover, it will be a
rare case where only the effect sizes and standard errors need to be identified in a
non-English study. Simply getting an English translation of the reported estimates
will be insufficient. It is critical that studies are understood clearly.
There are, of course, obvious exceptions. For example, if the aim was to
assess the effects of economic policy in say Latin America, it might be necessary
16 Identifying and coding meta-analysis data
to access non-English papers. Likewise, if the study’s aim is to analyze labor
demand elasticities in France, then studies published in French would need to
be included.
Obscure studies
The same principle applies to hard-to-get studies. Some studies are published
through obscure journals or working paper series that are very difficult to access.
The inability to obtain and hence include these studies is unlikely to affect the
findings of a meta-analysis. Indeed, it is our experience that the central findings
of meta-analyses are remarkably robust to marginal changes to the definitions of
the population of studies, the data, or the coded moderator variables. However,
influential estimates, in a statistical sense, do occasionally exist. When influential
estimates are found in the “gray literature” their inclusion is suspect. Even when
contained in our best academic journals, the influence of any one estimate should
be minimized by employing robust statistical techniques. These recommendations
may seem strange to medical researchers, where one large randomized clinical trial
may be more reliable than all of the remaining research combined. In econom-
ics, even the most rigorous and sophisticated research study may have omitted a
critical variable or somehow misspecified the estimation model and thereby report
biased estimates due to unavoidable data or methodological limitations.
Same estimates
Some studies have the same author(s), use the same data and report the same
estimates as previously published in an earlier paper. These should not be
included.11 Some studies are pure replications: they use the same data, the same
estimates, but are produced by different authors. Some meta-analysts choose
to exclude these, while others include them. If replication studies are included,
their replication status needs to be coded.
Essential
The most basic meta-analysis will involve a simple weighted average and/or a
simple linear regression model. This requires that data be collected on effect sizes,
their sample sizes and standard errors. It will also be necessary to code the name(s)
of the author(s), the title of the paper and publication outlet.18
Typical
Most meta-analysts collect data on the types of data used in the study, the
estimation technique used and the econometric model structure. Examples of
data differences include: cross-sectional, panel, or single country and whether
Identifying and coding meta-analysis data 21
establishment, firm or industry level data were used. Further data differences
involve: the country under investigation, the type of industry (e.g. manufactur-
ing or services), and the time period under investigation.19 Experience has shown
that the functional form (e.g. double log) and the exact model specification used
(the control variables included) are important dimensions of the original research
to code and to model. The estimator used can also be important. Does the study
use ordinary least squares? Does it control for endogeneity? If panel data are
used, does the study control for fixed effects? Where there are rival theories
regarding an effect size, it might be useful to include information on the theory
tested.20 It is also important to note omissions of relevant independent variables,
which can bias the original research findings, and the causal or error structures
(e.g. endogeneity) modeled in each primary study. Estimating and correcting
empirical economics for such potential biases was, in fact, the original intent
of MRA (Stanley and Jarrell, 1989). This topic is discussed in further detail in
Chapters 4 and 5.
Meta-regression analysis has the potential to correct the original econometric
research for a variety of misspecification, omitted-variable, and other biases. Thus,
it is very important to code studies and estimates for obvious omitted relevant
variables that might bias the original estimates. For example, omitting a measure
of capital in a production function is an obvious omission that might seriously bias
the remaining estimates. For the efficiency-wage effect on productivity, omitting
a measure of capital reduces the corrected estimate of the efficiency-wage effect
by nearly half of its typical value, making a practical and statistically significant
difference (Krassoi-Peach and Stanley, 2009).
Value-added
One of the great advantages of meta-analysis over both the original econometric
research and conventional narrative reviews is that it can add new and relevant infor-
mation unavailable to the original study to explain variation in research findings. As
mentioned above, MRA can be used to control and correct for omitted-variable bias
(see Chapter 4). But these omissions may have been unavoidable in the original
study due to data limitations. In many economics databases, information on known
relevant variables simply does not exist. Often study results will be influenced by
factors that are “study-invariant.” That is, factors that are constant for a given study
but vary across studies. Only the meta-analyst can model and estimate the effects of
these factors on econometric findings because a given “study-invariant” dimension,
by definition, does not vary across the data within a given study.
An example of how study-invariant data may be added is found in Jarrell and
Stanley (1990), where the unemployment rate is added to the MRA that explains
the union-wage premium. Due to the largely cross-sectional nature of early
union-wage studies (Lewis, 1986), the unemployment rate or any other cyclical
indicator does not vary in most of the data used by this research literature.
However, from the perspective of a meta-analysis, the unemployment rate is
easily added and found to explain successfully some of the difference among
22 Identifying and coding meta-analysis data
reported union-wage premiums (Jarrell and Stanley, 1990). A second example
adds study-invariant spatial data to the meta-analysis of the willingness to pay
for land preservation (Johnston and Duke, 2009). This information is unavailable
in the original studies because they involve surveys of willingness to pay for
preservation of a specific geographic site. Johnston and Duke (2009) find that
adding this study-invariant geographical data back into their MRA has practically
important effects on the estimates of willingness to pay. Furthermore, adding a
geographical dimension is especially important when meta-regression estimates
are used for benefit transfer to unstudied sites at new geographical locations
(meta-analysis for benefit transfer is discussed in Chapter 7).
It has become routine for meta-analysts to include the average year of the data
and/or the year that a study was published, and other study-invariant dimensions,
as a means to account for potential trends or path dependencies in research.21
Increasingly, it is becoming common to collect information on citations received
and journal impact factors.22 These are not collected from the study itself and
are again study-invariant. Further study-invariant measures may include more
“socio-economic” factors such as: the study’s authors, their gender (Stanley and
Jarrell, 1998), their funding sources (Doucouliagos and Paldam, 2010), and their
links with other researchers (Doucouliagos and Laroche, 2003).23 Meta-regression
analysis can be used to study the socio-economic process of economics research
itself (Stanley et al., 2008). The potential for meta-analysis to add value to existing
research is nearly limitless.
In the process of analyzing data, new directions and dimensions sometimes
emerge that the analyst would like to explore further and more objectively. This
data might not have been coded initially. Thus, it is not uncommon for the meta-
analysts to go back to the original studies. Furthermore, referees might require
added dimensions to be explored to ensure that the central findings of the meta-
analysis are robust, and this too might require additional data collection. So be
prepared to revisit your coding of the original research.
We provide concrete illustrations of the information needed to be coded when
we look at multiple meta-regression models in Chapter 5. All this coding, however,
can impose a heavy burden on degrees of freedom. We return to this challenge in
Chapter 7.
Yit = α0 + α1 Xit + ∑α Z
j
j jit + uit (2.1)
lnYit = α0 + α1 Xit + ∑α Z
j
j jit + uit (2.3)
Interest, in all cases, lies in either the estimates of α1 or on the marginal effect
of X on Y, which is a function of α1. The search reveals the group of studies that
report estimates of α1 or the marginal effect of X on Y. Effect sizes in economics
are typically computed from regression coefficients. For these to be included in
the meta-analysis, they need to possess two important properties. First, the effect
should be a partial one; that is, it should measure the effect of one variable on
another, holding other factors constant (the familiar ceteris paribus assumption of
economics).24 Second, the effect should be comparable within and between differ-
ent studies (Stanley and Jarrell, 1989; Becker and Wu, 2007).
( )
1+r
z = __12 ln ___
1−r (2.5)
This z-transformation also addresses the issue of the standard error of r not
being independent of the value of r. At least for the sake of robustness, the meta-
analysis can always be conducted using both the partial correlation and Fisher’s
z-transform. However, in past applications, we have found that these transforma-
tions make little practical difference to the central findings of a meta-analysis
(e.g. Doucouliagos and Laroche, 2003).
A second drawback of using the partial correlation is that it is not an economic
measure of effect. For example, partial correlations cannot be used in environmental
economics where the aim of the meta-analysis is benefit transfer. Hence, it might
be necessary to supplement the partial correlation with a measure of the economic
26 Identifying and coding meta-analysis data
effect. This will invariably mean that the partial correlation is used for a larger
number of estimates than the economic effect. For example, Doucouliagos
and Laroche (2009) employed the partial correlation to examine the effects of
unions on profits using the results of 45 studies. For 12 of these studies, they also
calculated the economic effect, the percentage reduction in profits evaluated at the
average degree of unionization.
2.3.4 Elasticities
Elasticity is the most widely used and most commonly known measure of an
empirical economic effect. Elasticity measures the percentage change in some
important economic phenomenon, say demand or Y, arising from a percentage
increase in some stimulus, say price or X. It is a natural effect size to analyze
in economics, because there is so much economic discussion about elasticities,
and an elasticity is often the crucial magnitude used to gauge the likely effect of
a given policy intervention. Many meta-analyses have used elasticities, includ-
ing Dalhuisen et al. (2003) on price and income elasticities of residential water
demand, Knell and Stix (2005) on the income elasticity of money, Melo et al.
(2009) on urban agglomeration economies, and Doucouliagos and Stanley (2009)
on minimum-wage employment elasticities. Elasticities are the common effect
size used in meta-analysis of research in marketing (e.g. Tellis, 1988; Bijmolt
et al., 2005; Albers et al., 2010).
There are two drawbacks with elasticities. First, the elasticities cannot always
be calculated. When the econometric model estimated is in double-log form, the
regression coefficients are direct estimates of elasticities. In other functional forms,
however, elasticity needs to be imputed from the statistics reported.33 This is a
problem if the authors do not report sample means of independent and dependent
variables. While sample means might be approximated/estimated using outside
sources, doing so might also introduce a new source of measurement error. In
contrast, partial correlations can be directly calculated from routinely reported
regression statistics. In some cases, this can make a practical difference. For
example, Doucouliagos et al. (2012a) explore the links between chief executive
pay and firm performance in the UK. The key theoretical variable of interest here
is the elasticity of executive pay with respect to performance. However, the authors
were able to collect only 187 elasticities from 44 studies, compared to 511 partial
correlations. Meta-analysis of the elasticities indicated no link between pay and
performance. In contrast, the larger dataset of partial correlations suggested a small
but statistically significant positive association (a partial correlation of +0.08).34
A second hurdle is deriving standard errors. Standard errors are needed in
meta-analysis to calculate optimal weights when constructing meta-averages
(see Chapter 3), and they are needed to correct a literature for selection bias
(see Chapter 4). In a log-log form, this is simple: the regression coefficients are
elasticities and their standard errors can be used directly.35 In other cases, however,
the elasticities have to be calculated, usually evaluated at sample means. The
standard errors of the regression coefficients in these cases are not the standard
Identifying and coding meta-analysis data 27
errors of the elasticity. The solution to this is to use either the delta method or the
Fieller method to estimate the standard error.36 This means that the number of
estimates that can be included will tend to be smaller than if the partial correlation
is used.37
In the linear form__regression
__ (2.1),
__ the
__ elasticity can be evaluated at the mean by
calculating η1 = α1 X / Y , where X and Y are the average values of the explanatory
and dependent variables, respectively.38 Using the delta method, the variance for
this is given by
__ __ __
2 2
X X
var η1 = __
__ var α + α2 __
2 1 1
__ var Y
2 (2.6)
Y Y
(see Valentine, 1979, for details). If sample means and variances are not reported,
then they might be estimated by using information from outside the study. If this
is not possible, then the delta and Fieller methods cannot be applied.
In practice, it is possible to bypass this issue altogether. First, because
standard errors are needed to weight the relative importance of each estimate,
it is possible to use a different set of weights. For example, instead of using
precision, which requires standard errors, it is possible to use sample size or
its square root.39 Second, it might in some cases be possible to approximate the
standard error for the elasticity by using the standard error of the regression
coefficient, though this might introduce a measurement error. Third, and most
important of all, in our view, the imprecision introduced from not using the
appropriate standard errors via, say, the delta method, is a second-order concern
compared to misspecification and publication selection biases. Most studies
that report regression coefficients from which elasticities must be calculated,
typically do not report the standard error of the elasticity. If they did, then the
issue of deriving the standard errors would no longer be relevant because they
would be reported. However, the vast majority of studies do report the statistical
significance of the regression coefficient. Hence, the process of selecting which
estimates will be reported works through the statistical significance of the
reported regression coefficient, rather than through the statistical significance
of the associated, but unreported, elasticity. Given that we wish to model the
research process and correct any distortions that might arise from it, we see it as
more important to use the t-statistics of the regression coefficients, rather than
the t-statistics of the elasticity. If publication selection is taking place, it is the
t-values of the reported regression coefficients that are being selected. From
this t-statistic, ti, it is very easy to compute a standard error for the elasticity
(η), SEη = η/ti.
It is worth noting that elasticity concepts vary. For example, while most
studies report long-run elasticities, many report short-run elasticities. In this case,
researchers may want to use only long-run elasticities or choose to convert all
estimates into long-run elasticities.40 Alternatively, this difference can be modeled
in the MRA by including a dummy variable that identifies short-run elasticities (see
Chapter 5).41 The latter approach has the advantage of quantifying the magnitude
of the change in responsiveness between the short run and the long run.
28 Identifying and coding meta-analysis data
There is also the issue of how to treat different concepts of own-price
elasticity. Some elasticity estimates include the income effect (Marshallian or
uncompensated demand) while others represent the pure price effect (Hicksian
or compensated estimates). Some studies will report the conditional price
elasticity (demand is conditional upon a subset of the consumer’s budget and
not the entire budget), while others will report unconditional estimates. Smith
and Pattanayak (2002: 285) discuss three approaches to dealing with this issue:
(a) pool all estimates and control for differences in the multiple MRA; (b) adjust
estimates to “a common economic concept”; and (c) drop estimates that do not
use consistent measures. Smith and Pattanayak advocate options (b) and (c).
Separate meta-analyses can be conducted for different measures, and they can
be jointly estimated using a structural system of MRA equations (Smith and
Pattanayak, 2002) or seemingly unrelated meta-regressions (see Chapter 7).
2.3.5 Semi-elasticities
The semi-elasticity measures the percentage change in Y when X changes by
one unit. This is a useful measure when the dependent variable is expressed in
logs or the equivalent and the explanatory variable is not. Examples include the
trade effect of forming a currency union (Rose and Stanley, 2005), the gender
wage gap (Stanley and Jarrell, 1998), and the effects of corporate taxation on
FDI (Feld and Heckemeyer, 2011). The semi-elasticity has the advantage of
not requiring additional information to calculate elasticities. Also, the standard
errors for the semi-elasticity are in this case derived directly from the regres-
sion output (the standard error for the regression coefficient). It is important to
note, however, that only those estimates that use the same scale for the explana-
tory variable can be combined, otherwise the semi-elasticities are not directly
comparable.
2.3.6 t-statistics
Following Stanley and Jarrell (1989), many meta-analysts have used the reported
t-statistic.42 Like the partial correlation, this has the advantage of being compa-
rable across estimates and studies, and it can be calculated for all estimates that
report a significance level.
However, there are three disadvantages with using the t-statistic. First, like the
partial correlation, the t-statistic is a statistical measure rather than an economic
one. Second, it is not as easy to interpret t-statistics by themselves. Third, it is
also necessary to control for its predictable statistical power. Conventional MRA
begins with an effect size that is then converted into a t-statistic to correct for
heteroskedasticity by weighted least square – see equation (4.2) in Chapter 4.
A similar transformation must also be made to the right-hand side (RHS) of the
regression by dividing all of the moderator variables by the estimate’s standard
error (SE) and including 1/SE as an additional independent variable (Stanley and
Jarrell, 1989; Stanley, 2008) – again see equation (4.2). When specified correctly,
Identifying and coding meta-analysis data 29
this meta-regression model can also identify and correct for publication bias
(Chapter 4). The coefficient on 1/SE in this weighted least-squares MRA can be
interpreted as the estimated effect size. That is, the effect size is measured not by
t-values but rather by the corresponding partial correlation or elasticity. However,
if the t-statistic is used as the dependent variable, without a corresponding
transformation of the RHS variables, the interpretation of the meta-regression is
not what one might expect. Rather, it concerns only the process of publication
selection and not the heterogeneity among true empirical effects.43 This topic is
discussed further in Chapters 4 and 5.
• Study dependence. When studies report more than one estimate, estimates are
not strictly independent of each other.
• Author dependence. If authors publish more than one study, estimates between
these studies may not be independent of each other.
• Spatial dependence. When researchers receive direct feedback from each
other or are influenced by prior findings, this might cause data dependence.
In other areas of research (e.g. medical research and psychology) using experi-
mental trials, one estimate is likely to be independent of the next. In economics,
meta-analysis is often criticized on the grounds that the data are not independent.
Yet in some cases, they are. For example, when looking at the effects of unions
on productivity, studies often sample entirely different establishments. Similarly,
experimental economics uses entirely different samples of (typically) student
subjects. However, in many other areas of economics research, data dependence/
independence is likely to be more complex. Data dependence is particularly a
problem to the meta-analysis of macroeconomic data. Most applied econometric
studies draw from the same data sources (e.g. World Bank Development Indicators,
Penn World Tables, US Bureau of Economic Analysis, etc.), which implies that
the statistical results from the same data should be somewhat related.
So, where studies use the same data, is meta-analysis meaningful? If the studies
cannot logically be combined in a meta-analysis, then they cannot be combined in
a traditional qualitative literature review either. This would then mean that it would
not be possible to draw inferences from macroeconomic studies. However, unlike
conventional narrative reviews, the meta-analyst has two objective strategies at
her disposal.
First, it is routine to code for several of the dimensions of the original research,
including those potentially related to this dependence. For example, while all
studies might be drawn from the World Bank Development Indicators, they may
not include the same set of countries and time periods, and some may make
independent misspecification errors through the researchers’ idiosyncratic choices
of exact model specifications and methods. MRA routinely controls for the country
composition of the samples, the time periods used and other potentially dependent
dimensions of the research results.
Second, this problem of dependence is likely no worse in meta-analysis than in
macroeconomic research in general. Stanley and Jarrell (1989) argue that the issue of
data dependence is likely to be less problematic for MRA than for the typical econo-
metric applications where autocorrelation, path dependency, and non-stationarity
are ubiquitous and strong. Such potential dependence is routinely handled in applied
Identifying and coding meta-analysis data 37
econometrics by more sophisticated regression models or techniques. The same set
of tools, methods, and models available in conventional econometric studies are
also available to the meta-analyst. Thus, if dependence is thought to be a further
concern, we can use more sophisticated regression techniques to handle it. It has
become standard practice among meta-analysts to employ multilevel (or unbalanced
panel) MRA models to account for potential dependence explicitly and cluster-
robust standard errors to correct for its potential effects. These topics are discussed
in greater detail in Chapters 4, 5, and 6.
2.6 Summary
The coding of research is by far the hardest and most time-consuming step in
a meta-analysis. However, it is important because it provides the raw material
for meta-analysis. When coding, the meta-analyst needs to be as inclusive, com-
prehensive, yet insightful, objective, and as rigorous as is practically feasible.
Undertaking a meta-analysis is no shortcut to the “truth” or to an easy publication.
But it is likely to be the only path to a genuine understanding of contemporary
research in economics, business, social science and medicine.
A good meta-analysis is both an insightful, but short, narrative review and a
comprehensive and rigorous econometric analysis of the full research record.
When in doubt, we encourage meta-analysts to err on the side of being:
In the previous chapter we discussed how to search and code research for a
meta-analysis. This chapter focuses on the description of these data, present-
ing alternative ways of summarizing research findings. It is important to note at
the outset that describing a literature is not the central aim of meta-analysis in
economics. Meta-analysis offers so much more than this. The main contribution
of meta-analysis is to make inferences about the state of economic and business
knowledge and to correct a literature for misspecification and selection biases
that typically plague empirical studies (see Chapters 4 and 5). This more analytic
and comprehensive meta-analysis enables meta-analysts to test rival theories and
to provide accurate and corrected estimates of policy-relevant parameters.
While not the central focus of meta-analysis, it is nevertheless extremely useful
to commence a meta-study with a close look at the data. We will illustrate all
aspects of meta-analysis, including descriptive summaries, using data from four
published meta-analyses: the effects of unions on productivity (Doucouliagos and
Laroche, 2003), residential water price elasticities (Dalhuisen et al., 2003), the
value of a statistical life (Bellavance et al., 2009) and minimum wage elasticities
(Doucouliagos and Stanley, 2009). Table 3.1 summarizes some of the key features
of the data collection for each of these studies, and the nature of the meta-analysis.
The interested reader should refer to these studies for further details.
Field Search engines Type of data included Effect size Number of studies Countries Aim of study Explores
used (estimates) studied publication bias?
Unions and Numerous Published in English Partial 73 (73) Various Testing Simple test
productivity and French correlation theories
Water demand Numerous Published and Elasticity 50 (110) Various Parameter No
unpublished estimate
Value of a Numerous Published only Dollar value 37 (39) Various Parameter No
statistical life estimate
Minimum wage Numerous Published and Elasticity 64 (1,474) USA Testing Extensively
unpublished theories
40 Summarizing meta-analysis data
We have found the funnel graph to be the most useful of these (see Stanley and
Doucouliagos, 2010). The funnel graph does a nice job of displaying publication
selection bias, which is discussed in detail in the next chapter. Funnel plots and
frequency distributions appear to be the most common graphs used in economics
meta-analysis. Forest plots can be used to illustrate the distribution of estimates
by plotting each estimate and its associated confidence interval. They show both
the pooled mean as well as the variation (Lewis and Clarke, 2001). Examples in
economics include Capelle-Blancard and Couderc (2007) and Havránek (2010).
90
80
70
60
1/Se
50
40
30
20
10
0
-.8 -.6 -.4 -.2 0 .2 .4 .6
r
One use of vote counts is that they present a rough summary of the distribution
of findings and the extent of apparent disagreement within a field. For our four
datasets, it appears that unions and productivity has the most disagreement. For
water price elasticities, it appears that there is agreement regarding a negative
effect. Readers and reviewers routinely carry out such vote counts by selecting
studies with conflicting estimates to illustrate that there is disagreement within a
given literature.
While fairly straightforward, vote counting has several disadvantages, all of
which stem from the deliberate loss of information; vote counting essentially means
taking a distribution of estimates and collapsing this into two to four categories.
The resulting loss of information is unnecessary and often misleading.
Vote counting can create problems where there were none. First, as Hunter and
Schmidt (2004) note, vote counting can be misleading because the vote counts
ignore the effects of sampling error. Sampling error creates variation in results
that makes them look like they disagree. When 95 percent confidence intervals are
constructed around the estimates, an entirely different conclusion might emerge.
A second problem with vote counting is that it does not provide an estimate
of an economic magnitude, such as an elasticity, that can be used for policy
making. Statistical significance is an important first step, but insufficient by itself.
Providing reliable estimates of economic parameters is critical for both policy and
economic understanding.
A third problem with vote counting is that by taking away the focus from
elasticities, it also takes away the focus from publication bias. We show in the
next chapter that the union-productivity literature is relatively free of selection
bias, whereas the literature on water price elasticities, the value of a statistical
life, and minimum-wage effects are highly contaminated by it. By focusing on
vote counts, the meta-analyst will entirely miss a key structural weakness in the
data and the need to correct the data before valid inferences and sensible policy
recommendations can be drawn.
A fourth problem is that we frequently require a clear understanding of the
source of variation between studies – not just whether estimates differ in terms of
the level of statistical significance. There are several cases of meta-analysis that
begin with vote counting and then proceed to use the vote counts as the dependent
variable in the MRA. As noted in Chapter 2, such meta-probit (or meta-logit)
models can be highly problematic.
Summarizing meta-analysis data 45
Lastly, vote counts have been shown to possess a statistically perverse property.
Due to the low power of many statistical tests, Hedges and Olkin (1985) show that
the probability that a vote count comes to the wrong conclusion actually increases
as research accumulates. It is an understatement to suggest that this is less than the
statistical ideal. Our recommendation is that vote counts be used very sparingly
or not at all. Although vote counts could be used as an alternative way to illustrate
the distribution of the meta-analysis data, they lose much information and thereby
are less useful than a simple funnel graph. Compare Table 3.2 to the relevant
funnel graphs in Chapter 4.
3.2.2 p-values
Arguably, meta-analysis began in the early twentieth century when R.A. Fisher
and Karl Pearson independently developed procedures to summarize the overall
effect of multiple independent tests (Fisher, 1932; Pearson, 1904). Some meta-
analysts still use Fisher’s method of combining p-values to see whether a research
literature, when seen as a whole, shows a statistically significant effect. Under the
null hypothesis that there are no genuine effects, p-values (Pi) will be uniformly
distributed:
L
f = −2 ∑ ln P = χ
i
d 2
(2L) (3.1)
i=1
is statistically the preferred choice, where the wi are the weights used, and ηi is the
measure of the estimated elasticity. The optimal weights have been shown to be the
inverse of the estimates’ variances (Hedges and Olkin, 1985; Cooper and Hedges,
1994). However, optimal weights might not always be practical. Hence, other weights
might at times be used. For example, the sample size can be used where standard
errors and, hence, variances are not available;7 journal impact factors can be used as
a measure of the quality of the journal in which the study was published; citations
can be used as a measure of the importance the profession has placed on the study;
and for survey-based studies, it might be possible to use the survey response rate.
Some authors use weights that take into account the number of effect sizes included
from each study, in order to ensure that no one study exerts an undue influence on the
results. While the inverse of the variance is optimal from a statistical point of view,
these other weights can be used as a sensitivity analysis or as a matter of practical
necessity.8 Nevertheless, preference should be given to the use of optimal weights.
The estimated standard error of ηw can be calculated as the square root of
the reciprocal of the sum of the optimal weights. This can be used to construct
confidence intervals and to test the statistical significance of the weighted mean.9
Averages can also be calculated for subsets of the data. For example, for
sensitivity purposes, we might want to report the weighted mean for certain
regions (or firms), certain time periods, for specific measures of effect, or for
those estimates published in what are deemed to be leading journals.
effecti = β0 + ui (3.3)
Unweighted
Field FEE 95% CI (FEE) REE 95% CI (REE)
average
Meta-analysts should note that statistical significance of the overall effect size is
a necessary, but not a sufficient condition for empirical relevance or for policy
importance. True empirical importance further requires that the overall estimated
(and corrected) effect be large enough to have a notable economic impact.
4 Publication bias and
its discontents
When the majority of reported findings are selected for statistical significance,
empirical phenomena can be manufactured. For example, the efficacy of a par-
ticular pharmacological treatment or the adverse employment effect of raising the
minimum wage is seen by many researchers as established fact, yet these effects
may be nothing more than the outcome of publication selection (Krakovsky, 2004;
Doucouliagos and Stanley, 2009).
“It is a fact of life that people polish their goods to make them as shiny as
possible to attract customers” (Doucouliagos and Paldam, 2009: 445). In reviewing
the effectiveness of development aid, Doucouliagos and Paldam (2009) identify a
“reluctance” on the part of researchers to go against the prevailing view. “To find a
negative effect of aid is to question this ‘do-good’ enterprise; hence the ‘reluctance’”
which they argue arises out researchers’ priors “to be seen as ‘good’, and their
activity to have a ‘good’ purpose” (Doucouliagos and Paldam, 2009: 445). Most
researchers and reviewers wish to make a positive contribution to the laudable
enterprise of development aid. Publication bias need not arise from any nefarious
motive. Rather, it is often the unintended consequence of good intentions or sound
scientific practices. That is, publication selection is likely to be unavoidable – all
the more reason to be aware of it and to correct its adverse effects.
The real problem of publication selection is not its existence, but the large
biases that it can impart upon any summary of empirical economic knowledge,
when uncorrected. For example, the average reported value of a statistical life
(VSL) is likely to be biased by a factor of 5 or more (Doucouliagos et al., 2012b),
and the adverse employment effect of minimum wage is exaggerated manyfold
(Doucouliagos and Stanley, 2009). Doucouliagos and Stanley (2012) document
how publication selection may represent a serious problem (“substantial” or
“severe” publication selection) in nearly two-thirds of the empirical areas of
economics.
Publication selection has also been found to be widespread within other
sciences: the natural sciences (Sterling et al., 1995), political science (Gerber
et al., 2001; Gerber and Malhorta, 2008), and medical research (Hopewell et al.,
2009). After the widely publicized discoveries that Paxil and Vioxx have known,
but unreported, life-threatening side effects, the best medical journals changed
their publication policies to require the prior registration of all clinical trials
(Krakovsky, 2004). When ignored, publication selection can distort any literature
review, whether it is a conventional narrative review or a meta-analysis (Laird
and Mosteller, 1988; Stanley, 2001). Systematic reviews of medical treatments
now routinely use funnel graphs to discuss potential publication bias (see the
Cochrane Reviews).
Publication bias and its discontents 53
As we discuss in the next chapter, heterogeneity may also cloud and distort
research. However, the effects of heterogeneity are usually less one-sided, less
biased, than publication selection. It is the selection of random misspecification
biases, heterogeneity, and sampling error that generates publication bias.
90
80
70
60
1/Se
50
40
30
20
10
0
-.8 -.6 -.4 -.2 0 .2 .4 .6
r
80
50
40
60
30
40
Precision (1/se)
20
Precision (1/se)
20
10
0
0
-.4 -.2 0 .2 .4 .6 -.4 -.2 0 .2 .4 .6
Partial Correlations - Non-US Studies Partial Correlations
200
60
150
40
100
Precision (1/se)
Precision (1/se)
20
50
0
0
-1 -.5 0 .5 1 -.2 -.1 0 .1 .2 .3
Partial Correlations Cost Difference
225
200
175
150
125
1/Se
100
75
50
25
-25
-2.5 -2 -1.5 -1 -.5 0 .5 1
PE
The most accurate, or precise, estimates are at the top of a funnel graph. These
estimates will be least affected by publication selection because their high precision
make them less likely to be statistically insignificant. But how can we identify the
“top” of a funnel graph? Simulations show that averaging 10 percent of the most
precise reported estimates goes a long way towards correcting publication bias
(Stanley et al., 2010). For water elasticities, the average of the top 10 percent is
−0.105, −0.106 for the top four, and the most precise price elasticity is −0.122.
Publication selection bias distorts price responsiveness by a factor of three to four.
The manager of the local Water Board who doubles the water rate, seeking a 38
percent conservation of water, will be quite disappointed to find that this has little
effect on water use.
Our third focused example involves the value of a statistical life. These values
make no claim to quantify the multifaceted joys, meanings and tragedies of the
human condition. Rather, researchers observe people’s behavior as they engage in
voluntary risky behavior, such as choosing occupations, purchasing extra safety
devices, and buying insurance. Such behaviors allow economists to impute the
value that people are placing on their own lives. Needless to say, such a VSL
can be controversial but is also essential for the planning of many governmental
programs such as reducing toxins in our environment, improving the safety of
transportation, or the construction of public infrastructure. So useful, in fact, that
there have been 14 meta-analyses on the subject.2
Publication bias and its discontents 57
Figure 4.4 displays 39 VSL estimates, in millions of US dollars (2000 base year),
calculated from the coefficients of a variable that represents the probability of death
in a hedonic wage equation. Clearly, these values are highly skewed to the right,
indicative of publication bias.3 Note that there are no negative VSLs. It appears that
researchers use a positive VSL or, equivalently, a positive coefficient on the probabil-
ity of death as a model selection criterion. Negative values are just not economically
plausible – recall number 2 of Card and Krueger’s sources of publication bias.
10
8
Precision (1/Se)
4 2
0 6
-10 0 10 20 30 40 50 60
Value of a Statistical Life
4000
80
3000
60
2000
1/SE
40
Precision (1/se)
1000
20
0
-.1 0 .1 .2 .3 0
Beta Convergence -1 -.5 0 .5 1 1.5 2 2.5 3 3.5
Estimates of gamma
80
50
40
60
30
40
20
Precision (1/se)
Precision (1/se)
20
10
0
0
-3 -2 -1 0 1 -2 -1.5 -1 -.5 0 .5
Elasticity Elasticity
300
250
200
1/SE
150
100
50
0
-1.25 -1 -.75 -.5 -.25 0 .25 .5 .75 1 1.25
elasticity
With publication selection, researchers who have small samples and low preci-
sion will be forced to search more intensely across model specifications, data, and
econometric techniques until they find larger estimates. Otherwise, their results
will not be statistically significant. In contrast, researchers with larger studies need
not search so hard from the practically infinite model specifications to find sta-
tistical significance and will thereby be satisfied with smaller estimated effects.7
When publication selection is present, the reported effect is positively correlated
with its standard error, ceteris paribus; otherwise, estimates and their standard
errors will be independent, as required by the conventional t-test and guaranteed
by random sampling theory.
Such considerations suggest that the magnitude of the reported estimate will
depend on its standard error, or
ti = β1 + β0(1/SEi) + vi (4.2)
where ti is the t-statistic of each individual estimated empirical effect, 1/SEi is its
precision, and vi = εi /SEi, which should make its variance approximately constant.
When we begin with the variance in equation (4.1), WLS becomes
Intercept: β̂1 0.65 (1.72)* −2.86 (−7.27) 3.20 (6.67) −1.60 (−4.49)
1/SEi: β̂0 −0.0179 (−1.06) −0.0817 (−5.34) 0.808 (3.56) −0.0094 (−1.09)
n 73 110 39 1,474
*
t-values reported in parentheses are from heteroskedastic-robust standard errors.
Publication bias and its discontents 63
coefficient on precision in equation (4.2), this test is called the precision-effect test
(PET). Notice further that only the price elasticity of water demand and the value of
a statistical life have statistically significant β̂0s (reject H0: β0 = 0; t = {−5.34; 3.56};
p <.001); see Table 4.1. Even though these two research literatures are the most
asymmetric and skewed thereby imparting the largest relative amount of publication
bias ( β̂1 = {−2.86; 3.20}),11 we can still see through this fog of preferential selection
to identify a genuinely negative price effect on residential water consumption and
an authentic positive VSL. That is, we have reason to believe that raising the price
of water will, in fact, reduce water consumption, but not by very much, and that
workers do need to be compensated in the form of higher wages to take, voluntarily,
higher risks. For our other examples, we find no statistical evidence of any adverse
minimum-wage effect (accept H0: β0 = 0; t = −1.09; p >.05) nor evidence of any
productivity effect from union membership (accept H0: β0 = 0; t = −1.06; p >.05).
Thus, only water price elasticities and the VSL pass the PET.
Once we allow for publication selection bias, what are the overall empirical
effects in these important areas of economics research? Often, it is the magnitude
of the empirical effect, say an elasticity, that embodies many the important
economic questions. The estimate of β0 is such a corrected estimate of empirical
effect, but this estimator has its problems. When there is no effect, it is biased
upward, in magnitude, and when there is an effect, its bias is downward (Stanley,
2008). Consequently, we are better off just assuming that the effect is zero if a
research area fails to pass the PET (i.e. accept H0: β0 = 0).
In our examples, VSL and water elasticities pass the PET. The precision coeffi-
cients are only $0.808 million and −0.082, which are 8.5 percent and 22 percent
of the unweighted average VSL and price elasticity, respectively. In other words,
78 percent of the average reported price responsiveness is publication bias, and
−0.082 is very inelastic demand. To achieve, say, a 50 percent reduction in resi-
dential water consumption, our corrected elasticity implies that prices would
64 Publication bias and its discontents
need to be raised by 610 percent, which is a lot more than the 136 percent price
increase implied by the average elasticity. The point is that publication selec-
tion can make a huge practical difference to even our most widely accepted
economic phenomena. This difference is even larger for the VSL.
As well as being statistically insignificant, the other two areas of economics
have practically insignificant precision coefficients as well. Take, for example,
the corrected estimate of the minimum-wage elasticity (−0.009). This implies that
a doubling of the minimum wage would cause less than 1 percent of employed
teenagers to lose their jobs.12 Even if this effect were statistically significant, it is
negligible from any practical policy perspective.13
4.3.3 Limitations
Thus far, we have discussed how publication selection can be identified and
how the presence of a genuine effect, robust to publication selection bias, can be
tested using meta-regression analysis. However, these tests do have their weak-
nesses. The FAT, which identifies the presence of publication selection, is known
to have low power (Egger et al., 1997; Stanley, 2008). The PET, which identi-
fies whether there is actually an empirical effect beyond the bias of publication
selection, is usually powerful enough, but it can have inflated type I errors and
mistakenly detect effects that are not there (Stanley, 2008). These inflated type I
errors occur when there is much excess unexplained heterogeneity in the meta-
regression model.
Simulations show that the PEESE provides a better estimate of the underlying
“true” effect when there is an effect (Stanley and Doucouliagos, 2007, 2011;
Moreno et al., 2009a). However, this is not true when there is no empirical
effect and only publication selection. When the true effect or population param-
eter is zero (i.e. α1 = 0),16 we can show that the linear MRA model (4.1) is
correctly specified, and its WLS estimate of the precision coefficient in MRA
model (4.2) is less biased (see Section 6.3). Nonetheless, when there is no
effect (i.e. α1 = 0), but publication selection, both MRA corrected estimators are
biased upward. To be conservative, we recommend that the PEESE corrected
estimate of β̂0 from (4.3) be used only if we have reason to believe that there is
a non-zero effect (i.e. rejecting H0: β0 = 0 using (4.2) and thereby passing the
Publication bias and its discontents 67
PET).17 See the flowchart (Figure 4.7) at the end of this chapter for a summary
and visual representation of how these meta-regression methods are interrelated
and should be employed.18
In spite of our own advice, we report the PEESE estimates for all four of our
example areas of economics research (see Table 4.2). Recall that only two of these
areas of research have robust evidence of a genuine empirical effect. The PEESE
estimate of water price elasticity is −0.115, which as expected is somewhat larger,
in magnitude, than the precision coefficient, −0.082, for MRA model (4.2). Still,
the PEESE correction for publication bias lowers the simple average elasticity
by 70 percent, and there is little practical difference between these two corrected
estimates. Our visual estimate of the top of the funnel graph for water elasticities
(−0.1) is well within the confidence intervals of both corrected estimates. Using
the PEESE confidence interval (−0.086, −0.145) seems quite appropriate for this
application.
Turning to VSL, the PEESE estimates the value of a statistical life to be
$1.67 million, which is also consistent with a visual inspection of the top of
the funnel graph (Figure 4.4). This corrected estimate is merely 18 percent of the
unweighted average VSL. Needless to say, reducing the VSL more than fivefold is
likely to have practical consequences for many areas of public policy. As a result,
some programs that seek to reduce environmental or safety hazards will no longer
be cost-effective.
In the case of union-productivity effects, the PEESE is virtually zero; thus,
there is no conflict with our previous evaluations. The only potential exception is
the minimum-wage literature, which fail to show evidence of a genuine adverse
employment effect using robust standard errors. As expected, the PEESE is
somewhat larger (−0.036) than the PET coefficient (−0.0094). Admittedly, this
corrected effect, −0.036, is somewhat closer to being practically significant. On
the other hand, this estimate implies that the $0.70 rise in the US federal minimum
wage over the last three years (2007–09) caused less than a 0.5 percent decline
in teen employment for each of these years. Note also that this corrected estimate
of the employment elasticity of minimum wage is only one-fifth the average
reported estimate. No matter how we measure it, there is a lot of publication bias
in the minimum-wage literature. Once one accounts for likely misspecification
biases, our best evidence indicates that there is no practically meaningful adverse
minimum-wage effect (Doucouliagos and Stanley, 2009).
Table 4.2 PEESE estimates of corrected effect – MRA (4.3) (dependent variable t)
SEi: β̂1 2.14 (1.00)* −0.917 (−0.22) 0.325 (2.81) −0.857 (−4.58)
1/SEi: β̂0 −0.0034 (−0.24) −0.115 (−7.76) 1.665 (5.50) −0.036 (−10.11)
n 73 110 39 1,474
*
t-values reported in parentheses are from heteroskedastic-robust standard errors.
68 Publication bias and its discontents
4.3.5 Dependence
Thus far, we have discussed simple OLS MRA models of publication selection.19
It is our view that simple methods are often more robust and resilient to random
data problems and misspecification biases than more sophisticated maximum
likelihood methods. Nonetheless, when we have specific empirical evidence of a
threat to validity, it is necessary to use those methods that explicitly address this
threat, if for no other reason than to ensure that the simple OLS MRA’s central
findings are, in fact, robust. In this section, we consider the effects of dependence
among the reported estimates. In the next chapter, we discuss explicitly the issues
of heterogeneity and how to model systematic and random heterogeneity. Suffice
it to say that although heterogeneity needs to be explored, it does not necessarily
invalidate the results of these simple MRA models (see Section 6.4.1).
A common assumption that underpins all regression analysis is that the data
are independent or, more technically, that the errors terms are independently and
identically distributed. Violations of this assumption are common in conventional
econometric applications (e.g. autocorrelation). Meta-analysts have long
acknowledged the potential dependence among reported research estimates and
have sought methods to accommodate dependence (Stanley and Jarrell, 1989;
Stanley, 2001; Florax, 2002). Although there are several potential sources of such
dependence, this issue is especially acute when multiple estimates from the same
study are coded. There is always a possibility that the estimates reported in a given
study share some common effect (perhaps due to the researchers’ idiosyncratic
choices of data or methods) missed by the meta-analyst (perhaps even unreported
in the study) and thus omitted from the MRA.
When multiple estimates from the same study are collected, they can be averaged
across each study, eliminating the issue of dependence (Stanley, 2001). Doing so,
however, reduces the degrees of freedom available to the MRA and its statistical
power. Furthermore, some of the multiple estimates may be essential in statisti-
cally identifying the effect of a specific important research dimension. To account
Publication bias and its discontents 69
explicitly for potential within-study dependence, unbalanced panel models can be
used (Rosenberger and Loomis, 2000b; Bateman and Jones, 2003).
The unbalanced panel version of MRA (4.1) becomes
for the ith estimate in the sth study. vs represents an unobserved study effect,
which traditionally is assumed to be either “random” or “fixed.”20 The “fixed-
effect” term can be estimated by replacing vs with δD (where D is a matrix of
study dummy variables).21
The unbalanced panel version of WLS-MRA (4.2) is
In effect, the meta-regression model (4.6) assumes that study effects operate
largely through an unobserved differential propensity to select for statistical sig-
nificance. To see this, multiply (4.6) by SEis. The result is entirely the same as
(4.5) with the exception that study effects are now interacted with the standard
errors, μs · SEis. It remains a question for future research whether MRA (4.6) or the
WLS version of (4.5) better reflects typical economics and business research.
We prefer to use panel models in the context of the WLS panel model
(equation (4.6)) or to weight MRA (4.5) by precision squared (1/SEis2 ) using a
WLS statistical package (“analytic” weights in STATA).22 Beyond correcting
for heteroscedasticity, precision serves as an indicator of quality. Weighting by
precision in either of these two ways limits the influence of widely dispersed and
sometimes implausible effect estimates that lie at the bottom of the funnel graph.
In our previous examples, only the minimum wage meta-analysis has the full
multidimensional data structure required for panel analysis. Here, we found 64
studies containing 1,474 minimum-wage estimates and their standard errors.
Studies contain anywhere from 1 to 96 estimates and average 23 per study.
Columns 1 and 2 of Table 4.3 reports both the “random”- and “fixed-effects” panel
(or multilevel) FAT-PET-MRA results for the minimum-wage literature using
(4.6). In this application, there are no practical differences between the random-
effects (REML) and fixed-effects (FEML) multilevel MRAs. The corrected
elasticity estimates are virtually the same, approximately −0.01. Although now
statistically significant, as we discussed previously, such a small elasticity remains,
nonetheless, practically negligible.
REML critically assumes that the unobserved study effects are independent of
the included independent variables, SE or 1/SE. Researchers who select findings
for their statistical significance will likely experiment with econometric model
specifications and techniques to achieve their goal. These efforts are expected to
be correlated with SE, because larger standard errors require larger selected study
effects to achieve statistical significance. Thus, this critical assumption will be
routinely violated in economic MRA applications. As a consequence, we prefer to
use “fixed-effects” panel MRA models. The researcher who wishes to test which
Table 4.3 Panel and cluster MRA of publication selection among minimum wage employment effects (dependent variable t)
Intercept: β̂1 −1.71 (−5.62)* −1.59 (−20.5)* −1.60 (−4.49)* −1.59 (−11.8)* −1.15 (−11.4)* −1.25 (−2.61)*
1/SEi: β̂0 −0.0099 (−4.17) −0.0097 (−4.06) −0.0094 (−1.09) −0.0097 (−1.64) −0.0072 (−1.05) −0.0151 (−1.08)
k 64 64 64 64 64 64
n 1,474 1,474 1,474 1,474 1,474 64
*
t-values are reported in parentheses.
a
REML conventionally stands for “restricted maximum likelihood.” Here, it can also stand for “random-effects multilevel” because restricted maximum likelihood
may be used to estimate the random-effects multilevel MRA model (4.6).
Publication bias and its discontents 71
of these multilevel models is appropriate for her meta-analysis can conduct a
Hausman test to decide. In a multiple MRA context, Chapter 5, it is even more
likely that the unobserved study effects will be correlated with some independent
variables.
The issue of dependence concerns efficiency, rather than bias. Not correctly
accommodating the proper error structure in one’s MRA can cause the MRA
standard errors and t-values to be calculated incorrectly. In such cases, the usual
worry is that simple methods will be too generous in calculating these statistics
and might give a false appearance of statistical significance. This is not the case
in the minimum-wage literature because both REML and FEML find the MRA
coefficients to be statistically significant. Furthermore, efficiency is a second-
order concern compared with bias. As we have seen above, publication bias is
a much greater threat to understanding a given economic phenomenon, often
biasing average reported effect manyfold. Efficiency issues pale in comparison to
the often overwhelming effect of publication and misspecification biases.
For a more conservative assessment of the MRA coefficients, the meta-analyst
can use cluster-robust standard errors. Treating each study as a cluster and
thereby allowing potential dependence among the reported estimates within each
study to calculate the standard errors is another sensible way to handle potential
dependence. Column 3 of Table 4.3 presents the MRA results for minimum-wage
employment effects using cluster-robust standard errors. Non-robust, conventional
OLS standard errors make β̂0 statistically significant (t = −3.55; p < 0.01; not
reported in the tables). As expected, cluster-robust standard errors provide a
more conservative assessment than OLS standard errors (t = −1.09; p > 0.05;
reported in Table 4.3). Whether the standard errors are conventional or cluster-
robust, the estimated MRA coefficients will be identical; thus, either approach
furnishes an identical, practically insignificant, corrected estimate of the effect of
minimum wage on employment. To err on the conservative side, meta-analysts
should routinely use cluster-robust standard errors whenever multiple estimates
are coded per study.
A potential criticism of fixed-effects panel methods is that they tend to give
smaller standard errors and are thus likely to exaggerate the significance of
estimated MRA coefficients. But this too is easy remedied by calculating cluster-
robust standard errors within a fixed-effects panel model context. Calculating
cluster-robust standard errors is a menu option in STATA, and we report these
fixed-effects cluster-robust findings in column 4 of Table 4.3. Note how the
signal of publication selection remains very strong, but the existence of a genuine
minimum wage effect becomes questionable. When the fixed-effects WLS
version of MRA (4.5) is used (column 5), the exact estimated MRA coefficients
are somewhat different, but the overall results are essentially the same as cluster-
robust and FE-robust (columns 3 and 4).
Lastly, this potential dependence may be accommodated by running an MRA on
the average study estimate and their standard errors. That is, we can easily calculate
the average of all estimates, t-values, and standard errors in each study and run our
MRA across studies. Doing so will typically result in many fewer observations
72 Publication bias and its discontents
and thereby a loss of efficiency (64 vs. 1,474 for the minimum-wage literature),
but it has the additional benefit of weighting each research study equally. When
each estimate is coded and analyzed, studies reporting a large number of estimates
(e.g. 96 vs. 1) can have an undue influence on the statistical assessment of the entire
research literature. As a simple approach to potential within-study dependence and
to avoid any undue weighting of a literature’s findings, Stanley (2001) advocates
meta-analyzing study averages. Differences in how one weights each research
study can reverse the overall assessment of a research literature, for example class
size effects on study achievement (Krueger, 2003; Mishel and Rothstein, 2002).
We still advocate this approach because it offers a more conservative and often
more realistic assessment of the MRA’s statistical significance. Nonetheless, we
also suggest using multiple estimates with multi-level models and cluster-robust
standard errors to ensure the robustness of central MRA findings.23
When calculating averages for each study, researchers need to consider whether
they should be using a simple average or a weighted average. Hunter and Schmidt
(2004) recommend strongly that a weighted average should be constructed for
each study, using sample size as weights. Precision can also be used, or some
other set of weights can be used depending on the circumstances.
Our recommended menu of MRA methods may seem a little eclectic or even
bewildering to the uninitiated. We view the running of multiple MRA models
and approaches as a sensitivity analysis, a way to ensure the robustness of our
findings. In our experience, the central findings (whether there is a genuine effect
and its approximate magnitude) are robust to MRA method. Our approach is to
look for common patterns across several plausible models. For those who seek
the “correct” model, we are agnostic. The correct model will depend on nuances
in the structure of research in the specific area under investigation. Of course, one
can use a battery of econometric specification tests to see which MRA model is
appropriate. However, these specification tests are subject to type I errors and tend
to have low power. Because multiple specification tests are required, there will be
a high probability of making some error. Nonetheless, several of these tests are
discussed in the next chapter. Our approach is a pragmatic compromise. Aside
from the irreducible ambiguity of econometric specification testing, referees and
editors will ask the meta-analyst to perform such robustness checks because this
is the accepted practice in conventional econometrics.
where 1 < j < k, and Φ−1(a1) is the inverse cumulative normal (Hedges and Vevea,
1996: 304). The ajs are arbitrary cut points, such as 0.10, 0.05, 0.01 and 0.001,
which are chosen a priori. The weights of these arbitrary cut points, aj, can be
estimated from the data. After fully parameterizing this selection model, Hedges
(1992) derives the joint likelihood and uses a multivariate Newton–Raphson
method to find its maximum.
Hedges’ maximum likelihood, publication selection estimator (MLPSE) has
been applied to several areas of economic research but with mixed success; see
Ashenfelter et al. (1999), Florax (2002), Abreu et al. (2005), Nijkamp and Poot
(2005), and Huang et al. (2009). The MLPSE has been problematic in many of
these applications. For example, Florax (2002) finds that the MLPSE does not
converge for estimates of the price elasticity of water demand, which is one of the
examples we have been using. Worse still, Florax (2002) unearths the “awkward”
implication that the probability of publishing a statistically insignificant elasticity
is greater than that of publishing a statistically significant one. Likewise, Abreu
et al. (2005) obtain implausible weights for publishing estimates of economic
convergence. They find that studies with insignificant p-values between 0.05 and
0.10 are more likely to be published than statistically significant ones. We believe
that such patterns of selected p-values provide evidence that Hedges’ publication
selection model is misspecified. The story is similar for Huang et al. (2009). They
find that highly significant social trust and social participation effects ( p < 0.01)
are less likely to be published than marginally significant ones (0.01 < p < 0.05),
76 Publication bias and its discontents
and the corrected effect of social trust is virtually the same as the unadjusted mean
even though there is strong evidence of publication selection for a positive social
trust effect (Huang et al., 2009: 460).
We suspect that Hedges’ assumed selection model misspecifies publication
selection in economics. We are dubious that there are multiple steps of p-values
that should be given different weights in economics. Selection is likely to be more
complicated than any function of p-values alone. Rather, selection for publication
in economic journals will depend on whether or not an effect is statistically
significant but also on other unrelated features concerning the perceived quality
of the analysis. For example, methodological innovation is often prized over mere
statistical significance. Or selection may be related to the author’s reputation or
the novelty of the data she uses. In any case, only a portion of any economics
research literature is likely to be selected for statistical significance.25 In the
simulations used in our research, we assume varying fractions (from 0 to 75
percent) of a literature are selected for statistical significance while the remaining
portions are assumed to be selected for unrelated reasons (Stanley, 2008; Stanley
and Doucouliagos, 2007). Multiple FAT-PET-MRAs can explicitly model a more
complex publication selection process, allowing it to be affected by any observed
study characteristic. These more complex MRA models of publication selection
and heterogeneity are the subject of the next chapter and are discussed in detail
there. Maximum likelihood methods are highly sensitive to small changes in data
and model specification. Thus, in practice, the results obtained using Hedges’
MLPSE are less likely to be reliable.
γ1 will be zero if there is no effect, and γ1 = 1/2 when there is a non-zero effect.
This trace of statistical power has been used as a test for a genuine effect beyond
publication selection and has been called “meta-significance testing” (MST: Card
and Krueger, 1995a; Stanley, 2001, 2005a).
Like all tests, MST has its limitations. When there is publication selection the
relationship between the t-statistic and its sample size washes out. But worse,
MST often has type I error inflation (Stanley, 2008). In this context, the type I
error is the mistake of finding that there is a genuine empirical effect when there
is really none. Because scientific inference is designed to be conservative, the
type I error is the worse error to make. This problem arises from the use of the
absolute value before taking the logarithm of the t-value. This causes both very
positive and very negative values to be large and positive. When there is excess
heterogeneity, ubiquitous in economic applications, such large t-values (positive
or negative) will be found more often in large samples.
Box 4.12 What might prevent the t ratio from rising with sample size?
Card and Krueger (1995a: 239) erroneously use the absence of the relation
between the logarithm of a study’s reported t-value and the logarithm of its sample
size (or degrees of freedom) as evidence of publication bias in minimum wage
research – recall equation (4.9). However, there is a second, equally valid, cause
of an insignificant estimated γ1 in equation (4.9). Perhaps, there is simply no
actual empirical effect. In the absence of an empirical effect, t-values will not rise
with their sample size, regardless of whether or not there is publication selection
(Stanley, 2005a). Unfortunately, this oversight has been repeated several times
by economists. MST should not be used as a test for the presence of publication
selection. The simple answer to Card and Krueger’s rhetorical question: “What
might prevent the t ratio from rising with sample size?” (Card and Krueger,
1995a: 239) is that the minimum wage has no employment effect (Stanley and
Doucouliagos, 2009).
78 Publication bias and its discontents
Due to MST’s type I error inflation, we reluctantly do not recommend its use.27
Nonetheless, this idea of searching for the trace of statistical power is central
to differentiating genuine empirical phenomena from publication selection bias.
Fortunately, this same fundamental statistical relationship, statistical power, is
also embedded in the FAT-PET-MRA model (4.2). Note how MRA model (4.2)
is a regression of t-values on precision. Precision is literally part of a t-statistic
(recall equation (4.8)); thus, it must be a better measure of statistical power than
the square root of the sample size. Also, MRA model (4.2) does not need to use
absolute values and is thereby much less vulnerable to type I error inflation. Lastly,
the FAT-PET-MRA has more power than MST (Stanley, 2008). Thus, MRA model
(4.2) dominates MST.
ti = β1 + β0 (1/SEi) + vi (4.2)
where ti is the reported estimate’s t-value, and SEi is its standard error. This meta-
regression model (4.2) contains tests for publication bias (funnel-asymmetry test,
H0: β1 = 0), and for the presence of a genuine effect beyond publication selection
(precision-effect test, H0: β0 = 0).
The precision-effect estimate with standard error (PEESE),
provides a better estimate of the actual empirical effect corrected for publication
bias, when there is one. Although there are other methods to deal with publica-
tion selection, FAT-PET-PEESE are the best available meta-analytic methods for
economic and business applications.
All economic and business research literatures should be assumed to contain
publication bias, regardless of one’s assessment of the funnel graph. In our
experience over dozens of areas of economic and business research, the clear
majority possess substantial publication selection bias. Although the funnel-
asymmetry test is a valid test for the presence of publication selection under
typical conditions, it is widely known to have low power. Thus, the failure to
find explicit evidence of publication selection is no guarantee that its bias is not
a serious problem.
Regardless of the outcome of the FAT (node 1 in the schema shown in Figure 4.7),
we recommend using the PET to test for the existence of a genuine effect beyond
potential contamination from publication bias (node 2). Lastly, PEESE should
Publication bias and its discontents 79
Economic phenomena are complex and nuanced. Even where there is a simple
underlying phenomenon, unforeseen economic and political events easily over-
whelm the most stable and fundamental economic relation. For example, the
added uncertainty caused by the 2008 global financial crisis broke down the nor-
mally stable relation between income and consumption expenditures and between
interest rates and the demand for new loans, or at least those versions of these
relationships that fail to incorporate the full effects of unobserved uncertainty.
Likewise, economics research is complex and nuanced. Even when applied
econometrics is estimating a clear, meaningful and stable parameter (e.g. a price
elasticity), the econometric technique employed (e.g. structural equations), the
type of data (e.g. panel) or whether relevant explanatory variables are omitted
(e.g. income) often has a dominating impact on the estimated coefficient. Past
meta-analyses routinely find wide differences among the reported estimates of
purportedly the same economic parameter. Great disparities among research
finding are ubiquitous. For example, the reported minimum-wage elasticity of
employment ranges from −19 to nearly +5, with a standard deviation of 1.1.
Note that this variation, however measured, overwhelms the reported average
elasticity, −0.19. Similarly, the price elasticity of residential water demand has
an implausibly large reported range (from −2 to +0.8), implying that demand is
anywhere from quite price elastic to highly inelastic or even upward-sloping!
For these price elasticities, the average (−0.38) is also dominated by the reported
variation (standard deviation 0.41).
In previous chapters, we have focused on describing and summarizing research
findings for any given area of research. However, the most common finding among
the hundreds of meta-analyses conducted on economic subjects is that there is
excess heterogeneity. That is, the observed variation in any area of economics
research is always much greater than what one should expect from random
sampling error alone. Such excess variation begs explanation. Furthermore,
the failure to account for such excess variation can invalidate any simple meta-
analysis. Like conventional econometric analysis, the omission of a relevant
explanatory variable might possibly cause bias in the simple meta-analysis. In
this chapter, we discuss how to accommodate and explain this excess research
variation using “multivariate” (or “multiple”) meta-regression analysis.
Explaining economics research 81
5.1 Heterogeneity
The “problem” of heterogeneity arises from the fact that the expected value of a
reported estimate will often depend on many other factors: country or region, time
period, presence (or absence) of other relevant variables in the original economet-
ric model, dependent variable measure, functional form used and the econometric
technique employed, among others.
If unaccounted, heterogeneity can bias any simple MRA estimate.
Econometricians are well aware of omitted-variable bias. When a relevant
independent variable that is correlated with an included independent variable
is omitted from a regression (conventional or meta), the estimated regression
coefficients will be biased. Systematic heterogeneity, not explicitly accommo-
dated, may bias simple MRA estimates.
The most obvious approach to addressing excess heterogeneity is to explicitly
model it by coding any research dimension thought to have a potential effect on
the reported effects, including the standard deviation or its inverse, precision.
It is standard – and highly recommended – practice to model systematic
heterogeneity using a multivariate or multiple MRA.1 Multiple MRA is the
central topic for this chapter. A further advantage of meta-regression analytic
explanation of research heterogeneity is that it is based on theory. This theory
is statistical theory and need not be assumed by meta-analysts because the
researchers of the primary empirical literature themselves must have made the
necessary assumptions if their results are to be taken at face value. We discuss
the theory of meta-regression analysis and the more technical aspects of the
MRA in Chapter 6, while extensions to MRA, including multiple MRA equation
systems, are discussed in Chapter 7. In this chapter, we first briefly evaluate the
random effects approach to dealing with heterogeneity and then discuss how to
use MRA to explain heterogeneity.
e = Zβ + ε (5.2)
P = 1[Kδ + u ≥ 0] (5.3)
where λ(c) is the inverse Mills ratio, c = a − α1/σi, α1 is the “true” regression coef-
ficient or empirical effect, a is the critical value of the standard normal distribution,
and σ is the standard error (see Section 6.3 for a more detailed explanation and rigor-
ous derivation of this and related relations). Replacing the inverse Mills ratio term in
(5.4) with β1SEi gives our previously reported FAT-PET-MRA:
Explaining economics research 85
effecti = β0 + β1SEi + εi (4.1)
Recall that β1SEi represents systematic selection for statistical significance and
provides a linear approximation of this truncation relation. The telltale signal of
publication selection is a systematic relation of reported effects with their stand-
ard errors as revealed by meta-regression analysis (Card and Krueger, 1995a;
Stanley, 2005a, 2008).
Both selection and authentic empirical effect are likely to be more complex
than the simple models introduced in Chapter 4. Both terms in the simple FAT-
PET-MRA (4.1) can be expanded to allow for greater complexity. The true effect,
α1, may be replaced by β0 + ∑ βkZki to allow for heterogeneity and/or large-sample
misspecification biases. Again, Chapter 6 discusses these issues in greater detail.
Simple publication bias, β1SEi, may also be given a multivariate form, β1SEi +
∑ δj SEi Kji . Exploding MRA model (4.1) to allow both types of effects gives
effecti = β0 + ∑β Z k ki + β1SEi + ∑ δ SE K j i ji + εi (5.5)
As a result, there will be no single “true effect”, and publication selection will no
longer be represented by a single term, but rather all the terms β1SEi + ∑ δj SEi Kji .
SEi Kji represents any factor that might affect the researchers’ decision to report
a given estimate. No doubt such factors will include the perceived quality of
the econometrics used, such as whether the model includes obvious important
variables (e.g. income in a demand relation), whether suspected econometric
problems (e.g. non-stationarity) are properly accommodated, and whether
appropriate econometric methods are employed. These K-variables may include
any observable dimension of “quality.” Recall our discussion of research quality
in Chapter 2. The Z- and K-variables in MRA model (5.5) can be employed to
accommodate the effects of research quality on both the magnitude of the actual
empirical effects, Z, and the propensity to report an estimate, K.
Of course, as discussed in Chapter 4, MRA model (5.5) will also have heteroske-
dasticity. Either a WLS statistical routine should be used with precision squared,
1/SE2i , as the “analytic” weights, or OLS can be applied on the MRA model that
results from dividing equation (5.5) by the estimated standard errors, SEi.,
t i = β1 + ∑δ Kj ji + β0 /SEi + ∑ β Z /SE + u
k ki i i (5.6)
Moderator variables
In Section 2.2, we discussed the typical dimensions of research that are routinely
coded in meta-analyses. Chapter 2 identifies the standard error (precision) or the
sample size as essential moderator variables to weight the reported estimates of
effect and to correct for publication bias. Typical moderator variables include
omitted relevant variables, alternative measures of key variables, econometric
model and methods employed, and the data source used, among others. These
typical moderator variables become “essential” if the MRA is to avoid potential
misspecification biases due to omission of important explanatory variables.
Meta-regression analysis was originally conceived to quantify objectively
the magnitude of likely misspecification biases in econometric applications
(Stanley and Jarrell, 1989). Central among such biases is omitted-variable
bias. Because economics and business research is typically observational,
using pre-existing databases, researchers are often forced to exclude important
explanatory variables from their models. Consequently, there is the real risk
of misspecified studies. Fortunately, MRA can quantify the extent of likely
misspecification bias.
For example, in a meta-analysis of the gender wage gap, whether the worker’s
wage equation accounted for the workers’ age, experience, industry, and private/
governmental job status were all found to affect the reported estimated wage
gaps (Stanley and Jarrell, 1998). For minimum-wage studies, Doucouliagos and
Stanley (2009) coded for whether the original model included workers’ education
(School ), the unemployment rate (Un), a time trend (Time), year-specific
effects (Yeareffect), and regional effects (Regioneffect) – see Table 5.3 for the
full list of variables.9 Bellavance et al. (2009) find that whether compensation
and endogenous risk are taken into account in the hedonic wage equation has
important effects on the estimated value of a statistical life, while the omission
of injuries does not seem to have a noticeable effect. See Table 5.4 for a list of
moderator variables used in the meta-analysis of the VSL. Further discussion of
these coded moderators is given below.
Alternative measures
Often the most important explanatory variables in a meta-analysis concern alterna-
tive ways that key variables are measured by the researcher. In the estimation of
gender wage discrimination, the most important research dimension is how workers’
Table 5.3 Moderator variables for minimum-wage research
Table 5.4 Moderator variables for hedonic estimates of the value of a statistical life
Data sources
Virtually all meta-analyses in economics have coded for the obvious potential
heterogeneity caused by nuances in different sources of data. Surprisingly, dif-
ferent data sources do not always cause a noticeable difference to the reported
estimate of effect, for example among estimated gender wage gaps (Stanley and
Jarrell, 1998). However, differences in data and particularly subpopulation are
found to be important in minimum-wage and VSL research, see below. Because it
is conventional scientific practice, business and economic researchers are almost
always careful to state and describe the source of their data. Thus, meta-analysts
would be remiss if they did not code these data differences. One should be espe-
cially careful to denote the subpopulation, country or region, time period and
industry from which the data are drawn, because these dimensions often contain
unobservable, but important, differences in socio-economics or institutional set-
tings not fully incorporated into the econometric model employed.
Value added
Unlike conventional narrative reviews or conventional applied econometric
research, meta-analysis can add new and relevant information, unavailable
Explaining economics research 89
to the original study. It is now conventional practice among meta-analysts to
include several such moderator variables. Examples are legion. Take the value
of a statistical life. Bellavance et al. (2009) recognize that the average income
level of the workers studied (AveIncome, Table 5.4) may have an important
effect on their willingness to accept higher risks of death. More affluent workers
may be expected to hold out for greater compensation to increase their occu-
pational risks. The average income of workers between different studies can be
considerable because these studies involve different samples of workers from
different jobs, countries, regions, or time periods. Such variation in average
income cannot be controlled by the individual econometric study, because it
does not vary within a study but only varies across studies. If a meta-analysis
did nothing more for our understanding of the VSL than account objectively and
systematically for this one important dimension, it would make an important
contribution. Other study-invariant factors coded by Bellavance et al. (2009)
are the year a study was published (Year) and the average probability of death
(Death). For the minimum-wage effect on employment, the average year of
the data (Aveyear) is coded because it has been alleged that there have been
structural changes to the relationship between minimum wage and employment
(Doucouliagos and Stanley, 2009).
Because economic and business research is a socio-economic enterprise, meta-
regression has the potential to account for these factors as well. For example,
Stanley and Jarrell (1998) found that the gender of the researcher is correlated with
the reported estimate of the gender wage gap.10 Funding source and professional
associations have also been found to be important (Doucouliagos and Laroche,
2003; Doucouliagos and Paldam, 2008). Regardless of the judgment of meta-
analysts, meta-analysis is empirical; thus, the research record itself will decide
which research dimensions are relevant. As we have discussed earlier (Chapter 2),
it is important to err on the side of inclusion and code any research dimension that
is suspected to have an important effect of the reported results.
-10 0 10 20 30 40 50 60
Value of a Statistical Life
Figure 5.1 The value of a statistical life (in millions of 2000 US dollars)
Source: Bellavance et al. (2009).
Explaining economics research 91
When a mere 10 moderator variables are coded and are allowed to be either Z- or
K-variables, there will be more than a million possible models. Only coding
10 moderators represent a rather modest meta-analysis in economics.
The G-to-S approach begins with all explanatory variables included in the
equation. Then the least statistically significant variable is removed, one at time,
until only statistically significant variables remain. Yes, this too is less than ideal,
but some choice of moderator variables is necessary. “The strength of general
to specific modeling is that model construction proceeds from a very general
model in a more structured, ordered (and statistically valid) fashion, and in this
way avoids the worst of data mining” (Charemza and Deadman, 1997: 78). The
only other sensible approach is to report only the MRA model that includes
all coded moderator variables. Any other model has to be regarded as having
“negative” degrees of freedom (hence essentially worthless), because the number
of meta-regressions considered and the selection mechanism could be anything.
Unfortunately, this general, all-inclusive, MRA model also has great limitations.
Assuming that there are sufficient degrees of freedom to run the all-inclusive MRA,
the fog of high multicollinearity and low statistical power virtually guarantees the
obscuring of much of the existing pattern of research.
Returning to the VSL example, three variables (LnIncome, Year, and Death)
that are constant for each study are selected to be only Z-variables (i.e. research
dimensions that explain the variation among reported estimates) because it is unlikely
that researchers were selecting across these dimensions to obtain significantly
positive VSL estimates. Meta-analysts may restrict some of the MRA coefficients
in equation (5.6) to be zero when they have reason to do so. Here, we have so few
observations of VSL available (39) that we believe that it is prudent to reduce the
number of Z-/K-variables. Adding dozens of interaction terms (K-variables) will
almost certainly create high multicollinearity and thus obscure individual conditional
effects. We suggest that meta-analysts constrain some of the δj coefficients in MRA
(5.6) to be zero whenever there is a “theoretical” reason to do so.
The remaining moderator variables are entered as both Z- and K-variables
(i.e. those that allow for a differential propensity to report a given VSL estimate).
Beginning from a multiple MRA that contained 20 moderator variables and the
WLS-MRA version of equation (5.5), the G-to-S approach identifies four variables
as statistically significant (see Table 5.5). Note that Table 5.5 is reported in the
form of MRA model (5.5) and is estimated by using a WLS statistical package
with 1/SE2i as the analytic weights. The OLS version of MRA model (5.6) will
give the same results.13
Although many variables were allowed to be K-variables, G-to-S modeling
identifies SE alone to be related to publication selection. The MRA coefficient
for SE, 3.07, is in millions of US dollars and suggests that if the standard error
of the VSL estimate increases by $1 million, reported VSL will increase by
$3.07 million, ceteris paribus. This is a huge effect, practically and statistically,
inflating the average estimate of VSL by $9.15 million and explaining nearly half
of the observed variation among reported VSL estimates.14 Selection dominates
the reported hedonic wage estimates for the value of a statistical life.
92 Explaining economics research
Table 5.5 General-to-specific multiple MRA of the value of a statistical life
(dependent variable = VSL in millions of 2000 US dollars)
Table 5.5 finds three Z-variables (LnIncome, Year and Comp) related to the
observed heterogeneity among these reported VSLs, and their coefficients are
quite reasonable. We would expect that life is a normal good and that workers
value their lives more highly as their income increases. This is confirmed by our
meta-regression ( p < 0.01). The positive coefficient on LnIncome corroborates
this expectation, and its value implies that if average income were to increase
by 1 in natural logs, or 2.72 times, workers behave as if their lives are worth
$1.86 million more. We also find a trend among these estimates that increases
VSL by $190,000 per year. Note that both of these variables (Year and LnIncome)
constitute “value added” by the meta-regression. That is, these moderator variables
are study-invariant; thus, their influence on VSL cannot usually be investigated by
conventional econometric analysis.15
The MRA coefficient on Comp is also quite sensible. Studies that control for
worker’s compensation insurance (Comp = 1) tend to report $1.88 million lower
VSL estimates. Thus, we corroborate the common understanding in this research
literature that the failure to account for workers’ compensation insurance can
cause a significant exaggeration to the VSL.16
Lastly, the estimated intercept may seem somewhat absurd. How can the value
of a life be nearly negative $16 million? This meaningless number represents
an extrapolation of the MRA estimated equation to an average income of $1
(recall that the natural log of 1 is 0), well outside the observed range of average
income ($3,000 to $49,000). Needless to say, intercepts of economic relations
are often meaningless or misleading because they refer to very implausible
and irrelevant circumstances. If the intercept were recalibrated to the average
observed LnIncome (10.2), it would become 4.3, implying that VSL would be
$3.3 million at the average log income for the year 2000 when the hedonic wage
equation did not account for worker compensation.
Explaining economics research 93
Because economics and business research is conditional, determining a
representative estimate of effect requires some judgment on the part of the
meta-analyst and some notion about what constitutes “best practice” research
(Doucouliagos and Stanley, 2009). Recall that when we ignore the multidimensional
nature of VSL research, our corrected estimate, of VSL is $1.67 million with a 95
percent confidence interval of (1.05, 2.27) – see PEESE Table 4.2. However, this
simple MRA does not allow for the effects of the other moderator variables found
important above. But what values should we substitute into this multiple MRA for
these moderator variables? In conventional econometric applications, researchers
will often use the sample means of the independent variables to avoid making
such judgments. However, this makes no sense in the context of meta-analysis.
Using the sample means for all of our moderator variables will just give us back
the average reported VSL. But we already know that this average contains much
publication bias and perhaps misspecification biases as well. Surely meta-analysis
can do better.
First, we need to remove identified publication selection bias by setting SE = 0.
Recall that, as SE → 0, a study approaches perfection with no estimation error and
no publication bias. Selecting a year is easy and simply depends on the year for
which we wish to estimate or to use as an arbitrary benchmark. We choose 2000
because it is a nice round number. Less obvious professional judgment is required
to choose the appropriate values of Comp and LnIncome. As discussed above, it is
widely argued in this research literature that omitting worker compensation biases
results upward. Thus, following best practice in this literature, we substitute 1.0 for
Comp. Alternatively, using the sample mean, 0.21, would bias the estimated VSL
upward by about $395,000. Lastly, what is the most appropriate value of worker
income? The answer depends on the specific group of workers that one wishes
to use as a reference group. For our current purposes, we use the sample average
worker log income, because we seek only to provide a generally representative
VSL estimate corrected for identified biases.
Next, we substitute these values into the WLS multiple MRA model reported
in column 1 of Table 5.5. Doing so “predicts” VSL to be $1.36 million, with
95 percent confidence interval ($34,000, $2,693,000).17 This prediction is quite
close to the simple PEESE estimate that did not explicitly control for these other
dimensions of research. Even the downwardly biased PET coefficient (Table 4.1)
is well within this confidence interval, and vice versa. In Section 6.4.1 we argue
that the simple MRA models of publication selection introduced in Chapter 4 may
reflect the total publication bias despite potential omitted-variable bias, and their
estimates often adequately summarize a research literature. This interpretation is
consistent with the multiple MRA of the value of a statistical life.
It is customary in applied econometrics to run a few “robustness checks,” and
meta-analysts will likely be forced to follow this practice if they expect their
research to be published in top economic journals. Especially worrisome is that
one study (Sandy and Elliott, 1996) reports a VSL more than twice that of any
other, approximately $54 million. Because this study also has the largest standard
error, these MRA models will give it a small weight and a larger publication bias
94 Explaining economics research
correction – recall our discussion in Chapter 4. Nonetheless, it would be prudent to
use robust regression methods that minimize the influence of any one potentially
influential outlier. Column 2 of Table 5.5 reports the robust regression version of
our WLS-MRA and give virtually the same results, except that the effect of higher
income is notably larger.18 The MRA robust regression coefficients predict VSL
to be $1.15 million in 2000 for a worker with the sample of average income when
worker compensation has been included.
As a further check, we also report the multiple PEESE-MRA (recall model (5.7))
in column 3 of Table 5.5. Using the variance, β1SE 2, to approximate publication
selection bias gives very similar statistical results. However, a corrected VSL
based on this PEESE-MRA is somewhat higher, $2.74 million ($1.30 million,
$4.18 million), but like our other estimates is much smaller than the simple
average reported in this research literature ($9.48 million).
When there are multiple estimates reported per study, other MRA models are
needed to control for the potential dependence with studies and to ensure the
robustness of simpler MRA structures. Because Bellavance et al. (2009) did not
collect multiple estimates from each study, there is no need to accommodate
potential dependence among these VSL estimates. However, the minimum-wage
research literature routinely reports many estimates per study, and we discuss
the use of appropriate multiple MRA models of within-study dependence in the
next section. Before we move on to the minimum-wage literature, we should
note that there has been much consistency among the simple MRA models of
publication bias for VSL and their multiple MRA counterparts. The central
findings are that VSL is greatly reduced when identified publication selection
is filtered from the reported estimates, rises with income, and is reduced further
by nearly $2 million when worker compensation is included. See Doucouliagos
et al. (2012b) for a further discussion of this meta-analysis of VSL estimates and
its policy implications.
250
200
1/SE
150
100
50
0
-1.25 -1 -.75 -.5 -.25 0 .25 .5 .75 1 1.25
elasticity
Retail, and Food) for estimates relating to agriculture, retail, and food (mainly
restaurants). Typical subpopulation concerned the differences between males and
females (Male), whites and non-whites (Non-White), and teenagers and young
adults (Adults). Some estimates relate to a specific region, and the variable Region
is included to control for any differences between region-specific and US-wide
elasticities. Although most estimates relate to the contemporaneous employment
effects of a minimum-wage rise, many estimate the lagged effect of minimum
wage rises (Lag).
There is some debate in the literature about the need to control for cyclical
effects (Un) and school enrollment (School ). These are included to investigate
whether omitting them creates any noticeable “bias” or differences. The vast
majority of estimates relate to employment, but some relate to hours worked
(Hours); such differences in the measure of employment might affect the
estimated elasticity. Time allows for any effect of including a time trend in the
specification of the employment equation. A large group of estimates (696)
come from studies that use panel data (Panel ), while 210 use cross-sectional
data (Cross). Two related variables are Yeareffect and Regioneffect, which
control for the inclusion of period and cross-section (region/state) fixed effects,
respectively.
The majority of minimum-wage elasticity estimates have been reported in
published academic journals (Published), while others come from working papers
and have yet to be published. Two final controls relate to further differences in the
measurement of the minimum wage – the use of the Kaitz index (Kaitz) and the
use of a dummy variable (Dummy) for the presence of minimum wage.
96 Explaining economics research
As before, we used a G-to-S approach, and the resulting multiple MRA is
reported in Table 5.6. With 1,474 estimates of the employment elasticity of the
minimum wage, there are ample degrees of freedom for this G-to-S modeling
approach; thus, we allowed all of these moderator variables to be both Z- and
K-variables. Furthermore, we did not have a priori reasons to constrain any of the
possible effects to be zero. The G-to-S process resulted in 14 variables remaining
statistically significant, 12 Z-variables and two K-variables, and explaining
41 percent of the reported variation among minimum-wage elasticities.19
Of special note are the significant time trend, Aveyear, which suggests that
adverse employment effects, if any, are smaller each year, and the large MRA
coefficient on Panel. The latter suggests that studies using panel data find employ-
ment elasticities to be approximately 0.18 more negative, or adverse, than those
using time series data. However, this must be seen in the full context of the mul-
tiple MRA reported in Table 5.6. Note that the intercept, 0.12, suggests that a 10
percent increase in the minimum wage actually increases employment by 1.2 per-
cent, assuming of course all the other MRA moderator variables are zero. When
a study uses panel data and nothing else is coded “1”, the WLS-MRA estimates
the minimum-wage elasticity to be −0.062, which is much less than the average
reported elasticity, −0.19. But this assumes that all of the other moderator varia-
bles are zero, and this is not consistent with a high-quality research study. To have
only Panel = 1 implies that study did not include any year effects (0.069), did not
use the Katz index (0.052) or a log-log specification (0.064), among many other
things. One could easily argue that the better studies include year effect, uses the
Katz index and a double-log specification. When included with a panel study,
these research dimensions predict a positive employment effect from minimum
wage raises (0.123).20
Publication selection
For the minimum-wage literature, in a multiple MRA context, neither publication
bias nor authentic effect is represented by any single MRA coefficient. These
effects are themselves multivariate. In particular, the MRA coefficient on SE is no
longer a measure of the magnitude of the average publication bias by itself. Rather,
it is the combination of this MRA coefficient and all the K-variables (Un·SE and
Double·SE). We can easily test the joint hypothesis that all of the associated MRA
coefficients are zero. Doing so gives clear evidence that publication selection
remains in this multiple MRA (F(3, 1459) = 84.2; p < 0.0001).21
Estimated MRA coefficients from these K-variables can be used to calculate the
average estimated publication bias for a given research literature, which is −0.218
for the minimum-wage literature (calculated in terms of employment elasticity).
This is quite comparable to the value, −0.256, obtained from the simple MRA
(Table 4.1).22 Subtracting either of these estimated publication biases from the
reported minimum-wage elasticities converts the average minimum-wage
elasticity (−0.190) to a small positive value. Regardless of its sign, this positive
value is so small that it is of no practical import.
Table 5.6 Multiple MRA of minimum-wage research: WLS of model (5.5)
Moderator variables Column 1: WLS Column 2: Cluster-robust Column 3: REML Column 4: FEML Column 5: Robust
Heterogeneity (Z-variables)
Intercept (β1 ) 0.12 (10.45)* 0.12 (4.39)* 0.11 (7.03)* 0.10 (6.04)* 0.084 (8.78)*
Panel −0.18(−18.60) −0.18 (−4.72) −0.15 (−12.38) −0.15 (−10.48) −0.19 (−23.59)
Double 0.064 (8.90) 0.064 (3.20) 0.044 (5.98) 0.041 (5.42) 0.033 (5.45)
Region 0.040 (3.28) 0.040 (0.92) 0.087 (6.36) 0.090 (6.34) −0.065 (−6.43)
Adult 0.024 (4.27) 0.024 (2.68) 0.021 (3.75) 0.021 (3.72) 0.019 (4.04)
Lag 0.026 (4.46) 0.026 (1.60) 0.012 (2.08) 0.010 (1.59) 0.010 (2.13)
AveYear 0.004 (11.86) 0.004 (4.34) 0.003 (7.44) 0.003 (6.38) 0.003 (9.20)
Un −0.042(−6.47) −0.042 (−3.04) −0.041 (−6.15) −0.042 (−5.79) −0.020 (−3.82)
Kaitz 0.052 (8.76) 0.052 (3.06) 0.034 (4.51) 0.032 (3.88) 0.025 (5.05)
Yeareffect 0.069 (8.61) 0.069 (1.98) 0.068 (7.84) 0.067 (7.44) 0.106 (15.80)
Published −0.041(−7.85) −0.041 (−2.69) −0.039 (−5.63) −0.037 (−4.89) −0.028 (−6.48)
Time −0.022(−3.95) −0.022 (−2.08) −0.020 (−3.10) −0.017 (−2.46) −0.013 (−2.85)
Publication selection (K-variables)
SE −0.36 (−0.26) −0.36 (−0.11) −1.21(−3.86) −1.37 (−5.94) 0.11 (0.96)
Double·SE −1.48 (−8.52) −1.48 (−3.23) −1.09 (−4.33) −1.07 (−3.90) −1.01 (−6.97)
Un·SE −0.84 (−4.53) −0.84 (−1.87) 0.84 (2.61) 1.16 (3.08) −0.98 (−6.36)
No. of obs. (n) 1,474 1,474 1,474 1,474 1,474
No. of studies (k) 64 64 64 64 64
*
t-values are reported in parentheses.
Source: Doucouliagos and Stanley (2009).
98 Explaining economics research
Heterogeneity of minimum-wage elasticities
Like publication bias, genuine heterogeneity is multivariate. Rather than any
single overall minimum-wage effect on employment, there are many. The moder-
ator variables Panel, Double, Region, Adults, Lag, AveYear, Un, Kaitz, Yeareffect,
Published and Time all have a noticeable impact upon reported minimum-wage
effects beyond publication selection. Testing whether all of these MRA coefficient
are jointly zero, along with the intercept, β1, finds easy rejection (F(11, 1459) = 50.8,
p < 0.0001). There are genuine, systematic patterns among reported minimum-
wage research findings.
tis = β1 + ∑ δK
j jis + β0 /SEis + ∑β Z
k kis /SEkis + vs + uis (5.9)
for the ith estimate in the sth study. vs represents an unobserved study effect and
can alternatively be replaced by a “fixed-effects” term, δD (where D is a matrix of
study dummy variables). Equation (5.9) may also be regarded as a generalization
of the multilevel MRA model discussed in Chapter 4 (equation (4.5)), but one
that allows for heterogeneity and a more complex structure of publication selec-
tion. An alternative approach is to model unobserved study effects explicitly using
WLS in the context of MRA (5.5); recall Chapter 4.
Table 5.6 reports both the random-effects multilevel (REML) MRA (column
3) and the fixed-effects multilevel (FEML) MRA (column 4), for minimum-
wage research. REML is more accurately described as a mixed-effects model
because it contains both the fixed effects of all Z- and K-variables along with
random normal unobserved study effects. Although Table 5.6 is reported in the
form of MRA model (5.5), the WLS-MRA, equation (5.6) or (5.9), is used in all
cases. Here, too, the findings are largely the same as the previously discussed
multiple WLS-MRA findings. Again, the overall results of large publication bias
and of considerable systematic heterogeneity remain; however, their specific
structure changes somewhat. All of the Z-variables, which map the structure of
heterogeneity, remain statistically significant with the same signs, though their
coefficients change a little. The only notable difference is that the K-variable,
Un·SE, changes signs but remains statistically significant. Although we are not
particularly interested in any individual effect of a moderator variable, this change
poses a curious puzzle worth investigating further.27
The WLS-MRA finds that those minimum-wage studies that add the
unemployment rate to the employment equation are more likely to select for adverse
employment effects, ceteris paribus. This tendency is reversed when study-level
effects are allowed. The explanation for this reversal comes from the existence of
large outliers reported by a few studies. One study contains a dozen elasticities
between nearly −5 and −10; a second study reports four elasticities approximately
+2 and larger. Such large elasticities, whether positive or negative, are simply not
plausible. When unobserved study effects are allowed, these outliers dominate
the estimation of the study effects. Recall that the average reported elasticity is
−0.19. One of the beauties of the WLS-FAT-PET-MRA is that estimates with
large standard errors are given little weight, while precise estimates are given a
much larger weight. All of the studies with implausibly large elasticities also have
corresponding large standard errors; thus, the WLS-MRA automatically discounts
them. FAT-PET-PEESE-MRAs are remarkably resilient to such outliers.
Nonetheless, to ensure our findings are robust to the influence of a few outliers,
column 5 of Table 5.6 reports the associated robust multiple MRA. Robust
regressions are designed to be resilient to outliers and leverage points. Note that
robust regression MRA finds a significantly negative Un·SE coefficient, consistent
with the WLS-MRA. But most importantly, the overall findings of significant
publication bias and heterogeneity are confirmed.
102 Explaining economics research
5.5 Will the real meta-regression analysis model please stand up?
Thus far, we have been very optimistic about the prospects of employing meta-
analysis to identify the major patterns in economic and business research. We
have seen how meta-regression analysis can identify and correct for publica-
tion selection bias, and identify and estimate genuine empirical effects and
misspecification biases in both theory and practice. However, we are less san-
guine that MRA can accurately identify the full complex structure of economics
or business research. Because there is an indefinite number of potential MRA
models, even mature areas of research will have insufficient degrees of free-
dom to investigate them all fully. Furthermore, the research literature is likely
to contain idiosyncratic research choices that influence the reported results.
If some of these choices go unreported and are coincidentally correlated with
moderator variables, a given moderator’s MRA coefficient may represent some
other unobserved research dimension. Thus, it is unlikely that the values of
each reported empirical estimate can be fully understood by estimated MRA
coefficients. Needless to say, even if we estimate the “true” MRA model, there
would remain random unexplained research variation. But these limitations are
not unique to meta-analysis; they also apply at least as fully to applied econo-
metrics. Business and economic research is limited by the observational data
available, which are likely to be influenced by unobserved, but important, vari-
ables, and the number of potential economic models is routinely larger than the
number of observations (Sala-i-Martin, 1997).
Nonetheless, we remain confident that meta-regression analysis can adequately
identify a few of the important characteristics of a research literature whether or
not there is a practically significant effect. Recall that in the four meta-analysis
examples used, two (the value of a statistical life and the price elasticity of the
demand for water) found a genuine empirical effect in spite of strong publication
selection bias. With union-productivity correlations, there is little sign of
publication selection or genuine effect, and we find no adverse employment effect
from the minimum wage after allowing for publication selection. All of these
general findings are robust and confirmed by multiple MRA. Furthermore, much
of the shape of research is revealed by MRA. In the hedonic wage estimation of
VSL, average income, the presence of worker compensation for injuries and a time
trend were robustly identified as important. All of these factors are consistent with
theory. For minimum-wage research, many factors such as: the use of panel data,
year effects, the Kaitz index, a double-log model, data from young adults, lagged
effects and whether the study was published all had robust and consistent effects
on the reported minimum-wage elasticity. Obtaining such a clear and objective
understanding of any area of research is an important achievement.
But then, what is the best MRA model of economic research and how can
we decide? In our view this is the wrong question to ask. The right question is:
What research dimensions are robust to MRA model specification? However,
econometric training will cause many researchers to search for the “best” model.
Fortunately, there are objective criteria for MRA selection.
Explaining economics research 103
To illustrate the issues at stake, we return to our richest example of meta-analysis,
minimum-wage employment effects. Recall that this example has a multilevel
structure where many estimates are typically reported by each of the 64 studies.
Due to concerns of dependencies among estimates within a study, best econometric
practice would suggest that we use either cluster-robust or a panel model. But
which one?
First, let us consider whether there are significant study-level effects. For this
purpose, one can use the Breusch–Pagan Lagrange multiplier (LM) test. STATA
calculates this to be 1,272 for the minimum-wage literature, which is significant
at any level.28 Clearly, there are study-level effects. But are they “fixed” (FEML)
or “random” (REML)?
To answer this second question, the generic Hausman specification test can be
used (Hausman, 1978). As with all applications of the Hausman test, it is able
to differentiate between an estimator that is consistent under both alternative
specifications (FEML for this application) and a second estimator that is consistent
but also efficient only under one specification (REML). Recall that all “random-
effects” models assume that the random effects are uncorrelated with the independent
variables. However, as we discussed previously in this chapter, publication selection
is likely to make selected random effects correlated with the standard error, which
is one explanatory variable needed to be investigated in any MRA application. This
correlation is easily seen in Monte Carlo simulations and can, in effect, be tested
by the Hausman test. Returning to the minimum-wage example, the Hausman test
gives χ2(13) = 36.62 ( p < 0.001) and clearly rejects the random-effects MRA in favor
of the fixed-effects version. As expected, a random-effects multilevel MRA is
misspecified. The interested reader should consult Feld and Heckemeyer (2011),
especially Figure 2, for a more comprehensive diagram and discussion of model
specification testing for MRA. Feld and Heckemeyer (2011) also give an excellent
comprehensive discussion of the econometric issues at stake in MRA modeling.
What might surprise some applied researchers, especially medical researchers,
is that the fixed-effects model is a generalization of the random-effects models
because it allows the study-level effects to be correlated with the independent
variables (Mundlak, 1978). Thus, our general advice is to use “fixed-effects” and
not “random-effects” multilevel MRAs.29 In the case of minimum-wage research,
there is no practical consequence to a preference for fixed-effects MRA. The
MRA results for FEML and REML are identical with respect to direction and
significance of all moderator variables. As we discussed above, the FEML and
REML differ from the WLS and robust MRAs in finding that the inclusion of the
unemployment rate is correlated with selection against an adverse employment
effect, but this effect is not robust. In all other ways, FEML is practically identical
with both the WLS-MRA and the REML-MRA. Most importantly, our central
findings are robust to plausible variations in MRA model specifications.
We recommend that meta-analysts focus on those results that are consistent
across the multiple WLS, FEML, and cluster-robust MRAs along with the
simple FAT-PET-PEESE-MRAs. In our experience, all of these MRA models
give consistent results with respect to the existence of publication selection and
104 Explaining economics research
a genuine effect beyond publication bias. As far as explaining the large variation
observed among reported research results, the consistently significant moderator
variables identified across multiple WLS, FEML and cluster-robust MRAs should
be regarded as important revealed research dimensions. Lastly, a successful meta-
analysis will find consistent overall results between the simple FAT-PET-PEESE
models and the multiple MRA models regarding the presence of publication
selection, the existence of a practically significant empirical effect (or not), and
the approximate magnitude of the corrected effect.
ti = β1 + ∑ δK
j ji + β0 /SEi + ∑ β Z /SE + u
k ki i i (5.6)
where ti is the reported t-value of the ith reported effect, and SEi is the standard
error of this effect.
To ensure the robustness of the relevant explanatory research dimensions,
a number of alternative MRA model specifications and methods need to be
explored and reported (see Figure 5.3). When multiple estimates are typically
reported per study, MRA methods that explicitly accommodate potential within-
study dependence must be investigated. The best models are cluster-robust and
the fixed-effects panel or multilevel (FEML) MRA. We believe random-effects
(REML) models will be routinely invalid in meta-analysis due to likely correlation
between the unobserved study effects and the moderator variables. But this topic
requires further research. If researchers wish to be sure of the validity of their
chosen MRA model and thereby to convince reviewers and editors, a Hausman
specification test can be used to differentiate between “fixed” and “random”
effects, and a Breusch–Pagan Lagrange multiplier test can determine whether a
multilevel model is needed in the first place.
Explaining economics research 105
ti = β1 + ∑ δ j K ji + β 0 / SEi + ∑ β k Z ki / SEi + ui
Thus far, we have introduced and applied the conventional battery of statistical
methods and approaches to meta-regression analysis. The purpose of this chapter is
to delve a bit deeper into the statistical foundation of MRA. This book is intended to
be a practical guide for researchers who wish to conduct meta-analyses in econom-
ics and business; thus, we have avoided non-essential mathematics and associated
technicalities. In this chapter, we present: a theory of MRA directly derived from
econometric and statistical theory; a more mathematical and detailed representation
of the typical MRA models needed for economic and business applications; a more
careful derivation of our MRA models of publication selection; and further technical
details about the use of panel (or multilevel) methods in meta-analysis. In the proc-
ess, we will show how MRA results are entirely unaffected by issues of observed
and unobservable study quality when properly modeled. Applied researchers may
wish to skim the cream of this chapter.
e = Mβ + ε (6.2)
where:
σ21 0 0
Ω=
0 σ22 0 0
0 0 σ2L
Econometric theory and meta-regression analysis 111
and σ2i is the variance of the ith estimated effect, ei, and its sampling error, εi
(Davidson and MacKinnon, 2004; Green, 1990).
Equation (6.3) is a generalized least squares (GLS) estimator. WLS is a special
case of GLS where the variance–covariance matrix, Ω, has this specific diagonal
structure noted above.9 When the parameters in Ω are known, GLS is the best linear
unbiased estimator (Green, 1990). More relevant to empirical work is the fact that
this approach still has very desirable properties when consistent estimates of σ2i are
used in their place. With consistent estimates of σ2i , this feasible GLS version of
(6.3) will itself be consistent, asymptotically efficient, and asymptotically normal
(Wooldridge, 2002: 160–2).10
Here too, meta-analysts are in a better position than conventional econometricians.
By coding the statistical results of an entire empirical literature, they have ready
access to the informative content of ni observations from each of the L estimates
reported in the literature. That is, rather than using estimated squared residuals from
(6.2) and some skedastic function as a rough estimate of an individual variance
(Davidson and MacKinnon, 2004), each study in the research literature provides a
direct estimate of the needed variance, SE2i , from the ni observations used in that
study. By the assumptions made in each research study, the square of the standard
error of the reported estimate will be a consistent estimate of σ2i and often unbiased
as well. Our WLS estimation strategy (6.3) is easily implemented by most statisti-
cal packages using analytic weights = 1/SE2i in a WLS routine.11
With L estimates of σ2i , SE2i , we can also divide MRA model (6.2) by SEi to get
the entirely equivalent WLS-MRA in the form:
where ti is the t-value of reported effect i (Davidson and MacKinnon, 2004: 261;
Wooldridge, 2002). Estimating MRA (6.4) by OLS is equivalent to the feasible
GLS that uses SE2i to estimate σ2i in (6.3).
This form of the WLS-MRA, equation (6.4), makes the role of precision (1/SEi)
quite clear. However, some applied researchers have difficulty interpreting the
MRA coefficients from (6.4) correctly. Thus, in applications, we recommend
estimating WLS-MRA model (6.2), which calculates (6.3), and specifying 1/SE2i
as the analytic weights in standard statistical packages. Recall our previous
discussions of MRA models (4.1) vs. (4.2) and (5.5) vs. (5.6).
Hedges and Olkin (1985: 174) and Konstantopoulos and Hedges (2004: 293)
have argued that the standard errors of the WLS statistical packages are wrong for
meta-analysis and need to be divided by the square root of the mean square error
(MSE). This point has been repeated by many other meta-analysts. We do not agree.
What is at issue is whether σ2 in the variance–covariance matrix, σ2(MtΩ−1M)−1,
must be constrained to be equal to 1 or not. If we assume that there is no between-
study heterogeneity and that σ2i fully reflects the uncertainty of each individual
estimated effect, then the WLS variance–covariance matrix does reduce to
(MtΩ−1M)−1 and σ2 = 1, as Hedges and Olkin (1985) and Konstantopoulos and
Hedges (2004) suggest. Their point is technically correct, when the “fixed-effects”
112 Econometric theory and meta-regression analysis
model is used and we assume that there is no between-study heterogeneity (τ2 = 0
in conventional meta-analysis notation). However, in economics research, we
have not seen a case where there is no excess heterogeneity and therefore see no
need to constrain σ2 = 1. Allowing the research record to determine the best value
of σ2 permits the WLS standard errors to accommodate this overall heterogeneity.
Forcing σ2 = 1, as implied by Hedges and Olkin’s (1985) recommendation,
typically reduces the size of the WLS confidence intervals, making these estimates
seem more precise and their t-values larger. But with excess heterogeneity, these
“fixed-effects” WLS coefficients will likely have more variation than these
formulas suggests. This is the central weakness of “fixed-effects” MRA compared
to “random-effects” MRA; that is, “fixed-effects” MRAs tend to report standard
errors that are too small for the actual uncertainty involved. Thus, we see no reason
to make this weakness worse by overriding standard WLS reported results by
forcing σ2 to be 1. Our view is to allow standard WLS packages to use the data to
determine σ2 and to compensate for at least some of the likely excess heterogeneity.
See the Appendix to this chapter for a further discussion of this issue.
The weakness of our MRA model resides in the case where the original regression
models (6.1) are misspecified in a manner that biases the estimated variances.
Recall that the design matrix, M, accounts for potential biases, but it does not
further correct for inconsistent estimates of the standard errors. An alternative
approach for potential inconsistencies in estimating the standard errors is to use a
conventional feasible GLS estimate of the WLS-MRA given in (6.3); that is, one
that uses the estimated individual residuals to estimate σ2i . The real possibility that
SE2i may be biased suggests that we should be conservative in calculating MRA
standard errors. We recommend that meta-analysts use heteroskedasticity-robust
and/or cluster-robust standard errors in conjunction with MRA model (6.4) as
additional insurance.12
This statistical theory of MRA has been corroborated in simulations where
dummy variables are used in MRAs to estimate and accommodate misspecification
biases as represented by M in MRA model (6.2) (Koetse et al., 2010). In these
simulations, both random unobserved effects on individual estimates and
regression misspecifications were introduced, and yet the WLS-MRA model (6.4)
that uses dummy variables to identify the presence of potential misspecification
bias outperformed a mixed-effects estimator that explicitly allows for individual
random effects. Our WLS-MRA is found to do a remarkable job in estimating both
the misspecification bias and the underlying true empirical effect.13 Thus, MRA
has its foundation in well-established econometric and statistical theory. Because
MRA models are derived from statistical theory, they can easily be corroborated
and modified, when necessary, by Monte Carlo experiments.
“Fixed-effects” panel methods can be considered the more general approach that
allows for correlation between the study-level effects and the moderator variables.
To better understand why “fixed-effects” methods are more general and robust,
we first consider the “fixed-effects” approach to estimation. MRA model (6.5)
can be estimated by a “fixed-effects” panel model in two equivalent ways, both
of Kwhich use OLS. The most obvious is to replace Ss by K dummy variables,
∑ s=1δsDis, assuming that one omits the intercept. This least-squares dummy vari-
able (LSDV) approach also allows us to 1/SE2i as the analytic weights.16 A second
equivalent approach to “fixed-effect” panel estimation subtracts study averages
from all observed values (Wooldridge, 2002: 267):
edis = ∑ βM j
d
jis + εisd ,
i = 1, 2, ..., ms, s = 1, 2, ..., K (6.6)
j=1
__ ___ __ ___
where edis = eis − e s, Mdjis = Mjis − M js, and the bar variables, e s and M s, are the sth
study average of the reported effect and moderator variables, respectively. Note
that the study-level effects, Ss, disappear from this model entirely, because Ss is
constant within each study. Moderator variables that do not vary at all within stud-
ies will also drop out, and their effects cannot be estimated by fixed-effects __ panel
methods. Subtracting the study average of Ss makes each difference (Ss − S s) equal
to zero. No part of Ss will be contained in the error terms; hence correlation of Ss
with the moderator variables causes no bias or inconsistency. Further, note that all
influences from any observed or unobservable variable that is constant for each
study, such as study quality, is entirely eliminated by this model. This fact has
important implications for the quality of our MRA inferences, which are explored
further in Section 6.2.2.
In conjunction with panel models, the meta-analyst should also use cluster-robust
standard errors and the WLS multiple MRA(equation (5.6)). Unlike conventional
econometric panels, it is very unlikely that estimates within studies will exhibit
the type of dependence routinely seen in time series. However, the variance within
studies might well differ from study to study even after the systematic variation is
fully accounted for. Thus, it is prudent to use cluster-robust standard errors.
In contrast, “random-effects” panel methods replace Ss with a random effect, vs,
and a feasible GLS strategy is employed to estimate a rather complex variance–
covariance matrix, Ω (Wooldridge, 2002).17 However, one must further assume
that the study-level effects, Ss, and moderator variables, Mjis, are independent,
if the resulting estimates are to be consistent and unbiased. If Ss and Mjis are
correlated, “random-effects” panel estimates become biased. To see this, define a
new composite error term, vis = vs + εis, and (6.5) becomes:
Econometric theory and meta-regression analysis 115
Note that if Mjis is correlated with vs it will also be correlated with the composite
error term in (6.7), vis. As discussed in every econometrics textbook, whenever
the independent variable of a regression model is correlated with the regression’s
error terms, estimates will be biased and inconsistent. Essentially, the overlap
between the random and deterministic components of the regression model does
not permit a clean separation or estimation.
Because there is no reason to rule out correlations between moderator variables
and study effects in a meta-regression context, meta-analysts should use “random-
effects” unbalanced panel models with great caution. Conventional “fixed-effects”
panel methods are the more general and robust approach, thus they earn the
preferential position unless there are good reasons to the contrary.18 Or, as discussed
in Chapter 5, one can use the Hausman test to test for this correlation and thereby
to choose between fixed- and random-effects models (recall Figure 5.3). However,
we have further reasons to suspect widespread correlations between study-level
effects and moderator variables in MRA. In most cases, unobservable study
quality will be correlated with observed and coded research choices of methods
and variables, which will induce bias if a “random-effects” model is used.
Here, we expect that Qs will be correlated with some of the moderators, Mjis,
perhaps highly so. After all, if “study quality” is to deserve this name, it should be
correlated with, for example, the precision of the reported empirical effects, the
choices of econometric models that researchers make or whether researchers fail
to include obvious important variables into their models.
Without loss of generality, we can embed unobserved study quality into the
study effect. Define vs = Ss + γQs, and replace this in (6.8):
This MRA model is now identical to our previous equation (6.5), and fixed-effects
panel methods can consistently and unbiasedly estimate the MRA regression coef-
ficients by entirely filtering out the study-level effects, whether vs or Ss. Recall that
fixed-effects panel methods work even when these study-level effects (including
quality here) are correlated with the included moderator variables. However, this
is not true for “random-effects” panel methods or with pooled OLS. If OLS is
used to estimate (6.8) but Qs cannot be observed, then we have the classic case of
omitted-variable bias. Only fixed-effects panel models can fully avoid the poten-
tial bias from omitting study quality.20
This brief note is meant to complement our previous discussion of research
quality in Chapter 2. It is our view that objective and observable dimensions of
study quality should be coded and included in MRA. Potentially more pernicious
are those aspects of research quality that are more difficult or impossible to measure
objectively. Nonetheless, we find it quite comforting that such unobservable factors
will not contaminate MRA panel estimates, even under the worst circumstances
where they are highly correlated with the estimated MRA effects.
It may also be worth pointing out that these desirable properties of panel MRAs
hold for a broader class of unobservable and “unmentionable” study effects such
as researcher ideology, research funding source, their institutional affiliation and
the strength of the commitment that researchers have for a given theory. Because
such potentially contaminating influences do not vary across estimates within
a study, they are swept up into the study effect, vs in (6.9). Regardless of the
strength of the influence that such factors might have on research, like study
quality, they can be folded into the study effects, and fixed-effect panel methods
Econometric theory and meta-regression analysis 117
can estimate the remaining observable and objective dimensions unbiasedly.
Even the most “politically sensitive” issues of economic and business research
can be accommodated without contaminating the remaining MRA estimates when
multiple estimates are reported per study.
This meta-regression of an estimate and its standard error was first introduced
by Egger et al. (1997) to serve as a test for the presence of publication bias. This
Egger meta-regression is itself a generalization of the well-known Galbraith dia-
grams (Galbraith, 1988) that adds a constant term to the WLS version of (4.1):
ti = β1 + β0(1/SEi) + vi (4.2)
In spite of the intuitive appeal of these relations, they lack a rigorous statistical
foundation. Although it seems apparent that smaller studies with correspondingly
larger standard errors would need to search harder over different data subsets, alter-
native specifications and methods in order to achieve statistical significance when
the underlying phenomenon is small or non-existent, this connection between a
reported estimate and its standard error begs for a more rigorous grounding. The
purpose of this section is to provide a mathematical argument for this dependence
of a reported estimate and its standard error when there is publication selection for
statistical significance. In the process, we hope to provide a better understanding
of this relation and its limitations.
Publication selection is analogous to the better-known bias that arises through
sample selection, famously addressed by Heckman (1979).21 Take the example
of gender wage discrimination. The gender wage gap is estimated by comparing
the estimated returns to worker productivity measures from samples of male
and female worker wages (e.g. Oaxaca, 1973; Jacobsen, 1994). A problem
arises because wages are only observed for employed workers, those who have
reservation wages lower than the observed market wage rate. The decision to
participate in the labor market is itself a function of wages, but wages might
be differentially affected by gender discrimination. Therefore, a regression on
just employed workers may provide biased regression estimates. Observing a
worker’s wages and discrimination may be endogenously related. To address this
issue, labor economists have long employed a Heckman correction for this sample
selection bias, and a meta-regression of the gender wage gap finds that using a
Heckman correction greatly increases (by approximately 18 percentage points)
the reported gender wage discrimination (Stanley and Jarrell, 1998).
118 Econometric theory and meta-regression analysis
Publication selection involves a similar case of incidental truncation. It is
“incidental truncation” because the magnitude of the reported effect (like worker
wage) is not directly selected but rather some other variable, the estimate’s t-value
(labor market participation); see Wooldridge (2002: 552). Incidental truncation
differs from censored sampling where the dependent variable is itself selected and
there is data on the independent variables for both the selected and the unreported
samples (Heckman, 1979; Wooldridge, 2002: 552).
With publication selection for directional statistical significance, we observe an
estimated effect only if effecti/SEi > a, where a is the critical value of the standard
normal distribution. By referring to the well-known conditional expectation of a
truncated normal distribution, it is easy to show that observed effects will depend
on the population (or “true”) effect plus a term that reflects the selection bias,
which is equal to the standard error times the inverse Mills ratio:
where λ(c) is the inverse Mills ratio, α1 is the “true” effect, which is the expected
value of the original distribution, σi is the standard error of the estimated effect, and
c = a − α1/σi. Because effecti is asymptotically normal with mean α1 and standard
deviation σi, equation (6.10) follows directly from Theorem 21.2 of Green (1990)
(see also Johnson and Kotz, 1970). Relation (6.10) has the same general form as
a Heckman regression (Davidson and MacKinnon, 2004: 488).
When we replace sample estimates for the population values in (6.10) we get
If one further assumes that the inverse Mills ratio is constant, we have our FAT-
PET-MRA equation (4.1). Like (6.11), the more familiar Heckman regression
adds a term containing the inverse Mills ratio and σi. Thus, statistical models of
truncation and selection offer a simple meta-regression relation between observed
effect and its standard errors, giving us a rigorous foundation for the publication
selection methods described and applied in Chapter 4.
However, the linear MRA defined by equation (4.1) further assumes that λ(c)
is approximately constant with respect to σi. Unfortunately, we know better. In
general, λ(c) is not constant, and variations of λ(c) can cause the MRA estimate of
the mean of the full distribution of effects, β̂0, to be biased and inconsistent. This
complication causes considerable difficulty in finding an unbiased and consistent
corrected estimate of the empirical effect in question.
To understand this problem in context, consider how the conventional
correction for sample selection works. When we have data on the explanatory
variables for both the selected and unreported samples, the conventional Heckman
regression consistently estimates the corrected effect using a two-step method,
where the first step models the probability of being selected, and in the second
step, estimates from the selection equation are used to calculate the inverse Mills
ratio in a Heckman regression. In effect, the estimated selection relation gives a
Econometric theory and meta-regression analysis 119
sample estimate of the selection bias term, σi · λ(c).22 To state the obvious, this
conventional approach is not available to the meta-analyst, because we have no
information on the characteristics of the unreported values; thus, no direct way to
model the selection process or to estimate the Heckman regression.
So what is the “second” best strategy? Somehow we need to estimate the
publication bias term, σi · λ(c), however crudely, using information contained only
in reported research findings. Further, we know that λ(c) is itself a function of σi,
but, unfortunately, it is not a simple function of σi.
To see what type of relation we must approximate, take the derivative of (6.10)
with respect to σi:
However, Heckman (1979: 159) shows that ∂λ(c)/∂c = λ(c)2 − cλ(c), which gives
Inspection of the limiting relations suggests that the bottom of this parabola should
occur when SEi =0 (recall Section 4.3.4 and Box 4.8). Constraining a second-
order power series to have its perigee at SEi = 0 implies that β1 = 0, removes the
linear term from (6.14), and gives the PEESE-MRA (4.4):
Two separate simulation studies have confirmed the viability of using β̂0 from
the WLS version of (4.4) as a corrected estimate of empirical effect. Stanley
and Doucouliagos (2011) compare simple and weighted averages (recall FEE
and REE from Chapter 3) to both the linear and quadratic FAT-PET-PEESE-
MRAs and find that PEESE has the smallest bias and MSE when there is a
genuine empirical effect. These simulations also show that quadratic or cubic
power series that are not constrained to have β1 = 0 have large bias and MSEs.
Unconstrained power series are clearly dominated by PEESE (4.4). Secondly,
a team of medical researchers report a “comprehensive simulation study” on
14 different approaches, including “trim-and-fill,” to estimating effect when
there might be publication bias (Moreno et al., 2009a). Their simulations find no
120 Econometric theory and meta-regression analysis
better approach to publication bias correction than β̂0 on a combination of four
criteria (bias, MSE, variance and coverage percentage).23 When Moreno et al.
(2009b) apply these publication correction methods to randomized clinical trials
of antidepressants, they use PEESE and found it to be the best way to correct for
publication selection bias.
There is an important special case for the relation between the expected value
of a reported effect and its standard error that must be mentioned. When the
underlying empirical effect is zero, β0 = 0, equation (6.13) simplifies to λ(c).
Recall that c = a − β0/σi, and ∂E(effecti | truncation) / ∂σi reduces to the inverse
Mills ratio evaluated at critical value of the standard normal distribution, a, which,
of course, is just a constant. Thus, when there is no genuine empirical effect, the
expected reported effect will be a multiple of its standard error, and the linear
MRA model used in Chapter 4, MRA (4.1), will be correct.24 This observation is
important because it further validates the precision-effect test (H0: β0 = 0), which
tests for the presence of a genuine underlying empirical effect beyond publication
selection bias. The PET’s null hypothesis assumes that β0 = 0; thus, the FAT-
PET-MRA is correctly specified as a linear relation for testing whether there is
a genuine non-zero empirical effect. As a result of this special case, simulations
further confirm that the PET estimate from (4.1) is superior to PEESE (4.4) when
we accept H0: β0 = 0, but that PEESE (4.4) is statistically more accurate when this
hypothesis is rejected (Stanley and Doucouliagos, 2011). As a result, one should
only use the PEESE correction if there is first evidence of some genuine effect
(i.e. reject H0: β0 = 0).
When MRA model (4.1) is estimated without including Xi, E(β̂1)= β1 + γ1β2.
One interpretation of this second term, γ1β2, is omitted-variable bias; another is
the portion of publication bias operating more indirectly through the variable X.
β1 may be seen as the “direct” publication bias that results from resampling
and re-estimation when the researcher’s first estimate proves insignificant or
of the “wrong sign.” However, more typical in econometrics, researchers will
respecify their econometric models by using a different set of independent
variables, a different functional form, some new econometric technique, etc.
Variations in such research dimensions create heterogeneity and are repre-
sented by X. When a study is imprecise (i.e. has a high SE), greater effort will
likely be needed to obtain statistical significance. In these cases, a researcher is
more likely to use some highly influential research dimension, X. Such selected
heterogeneity contributes to publication selection bias and is evidenced by a
correlation between X and SE. Thus, γ1β2 may be regarded as a component of
publication bias.
By this interpretation, β̂1 from the simple MRA is not biased, but rather it
estimates total publication bias coming from a variety of channels – recall
E(β̂1) = β1 + γ1β2. It is our view that this interpretation will often be appropriately
true in economics and business research. Of course, this is only one interpretation.
Another is that X imparts an important effect on the target phenomenon and any
correlation between X and SE is just coincidence. In practice, it is impossible
to know which interpretation is correct because it will depend on whether
heterogeneity is created largely as a byproduct of publication selection or not. In
our past experience over dozens of areas of research, simple MRAs of publication
selection have always provided a satisfactory characterization of a given area of
122 Econometric theory and meta-regression analysis
economics research, because these characterizations are also confirmed by more
complex multiple MRA and methods.
As we show in Chapter 5, multiple MRA provides a more complex and
nuanced estimate of publication bias. Nonetheless, for both of the multiple MRA
examples reported in Chapter 5, the simple models of publication selection
provide summaries that are corroborated by complex and robust multiple MRAs
results. In the case of the value of a statistical life (VSL), both simple and
complex methods find strong evidence of publication selection for statistically
positive VSLs. Also, the associated estimates of corrected VSL are quite close
to one another. Likewise for minimum-wage research. Both simple and complex
multiple MRAs find evidence of selection for an adverse employment effect but
no evidence of any practical employment effect, once allowance is made for
this selection (see Chapter 5). Regardless of the interpretation chosen, we will
still need to conduct several multiple meta-regression analyses to ensure that
any simple interpretation is robust and/or to investigate how sensitive it is to
more complex potential influences. Although substantial publication selection
may allow one to ignore heterogeneity, in practice it is always a good idea to
explore fully the more complex and nuanced multivariate landscape for the sake
of robustness.
In previous chapters we presented, derived and applied the basic MRA model and
several variations. The aim of this chapter is to discuss a few additional dimen-
sions of its structure and application. Section 7.1 explores some of the alternative
applications of MRA in economics. In certain branches of economics, most
notably environmental economics, MRA is used principally to derive improved
estimates of key parameters, such as the willingness to pay. In other areas, the
focus of MRA is mainly on the testing of competing economic theories, while
other applications of MRA concentrate on modeling the heterogeneity among
empirical findings. MRA is flexible enough to accommodate all of these facets of
economics and business. In Section 7.2 we discuss the choice of MRA variables
when there are more variables than observations. This is followed by a brief dis-
cussion of the functional form of the MRA in Section 7.3. We then discuss the use
of MRA for identifying exclusion restrictions in Section 7.4. Section 7.5 looks at
the forecasting performance of MRA in both time and space. Section 7.6 investi-
gates the treatment of effect sizes that involve MRA models with interaction and
non-linear terms.
The second part of the chapter explores the treatment of multiple effect sizes and
the results from multiple meta-analyses. While most meta-analyses investigate a
single effect size, there are many cases where researchers may be interested in the
results of several related effects. In Section 7.7, we illustrate the use of systems
estimators, such as seemingly unrelated regression (SUR) and three-stage least
squares (3SLS), for dealing with multiple but related effect sizes. In Section 7.8,
we show that the MRA model can be used to analyze the results from several meta-
analyses of the same empirical phenomenon (the M2RA model). This section also
discusses the results of meta-analyses of unrelated literatures.
effecti = β0 + ∑βZ
k ki + β1SEi + ∑ δ SE K
j i ji + εi (5.5)
Versions of these MRA models can be used for a range of applications, such as
summarizing and qualifying estimates of policy-relevant parameters, correcting
these estimates for any number of potential biases inherent in observational eco-
nomics research, testing economic theories, explaining heterogeneity, modeling
the research process itself, and giving direction to future empirical investigation.
These applications are not mutually exclusive. MRA can in fact be used to inform
on all of these dimensions simultaneously. Indeed, we have argued throughout
this book that MRA is best seen from a broad perspective encompassing several
of these dimensions. In particular, we have argued that in order to derive improved
estimates of policy-relevant parameters, it is essential that the MRA summarizes
and explains past research, but also accommodates and minimizes publication and
misspecification biases.
7.5.1 How well does meta-regression analysis explain the research record?
Like any regression, the explanatory power of the MRA is limited by the amount
of variation in the underlying data that can be potentially explained – systematic
heterogeneity. Because the dependent variable in an MRA is a statistical estimate,
part of its variation from study to study is random sampling error and, hence,
innately unexplainable.
The explanatory power of reported MRAs ranges from 0.08 to 0.98, depending
on the research issue and the specification of the MRA.9 Most of the MRAs do
a reasonable job at explaining a significant portion of the heterogeneity in the
research record. Indeed, half of the MRA models we have reviewed report an R2
(or an adjusted R2) greater than 0.50.10 Unfortunately, few of the studies we have
reviewed actually explore whether the remaining variation is solely due to random
error. So it is difficult to assess fully how well extant MRAs explain the variation in
reported economics and business research.11 One exception is the study by Stanley
(1998) whose meta-regression model explains all the heterogeneity, leaving only
random sampling error unexplained.
where H is the key variable of interest for the meta-analysis and X is a vector of
other factors that affect the dependent variable, Y. The interaction, Hit · Kit, and
non-linear terms, H it2 , are important in identifying the marginal effect of H on Y,
which in this case is given by: ∂Y/∂H = α1 + α2K + 2α3H. The interaction term
Hit · Kit measures the effect of H on Y conditional on the value of K. Similarly, the
term Hit2 causes the effect of H on Y to be conditional on its own value.
If this marginal effect can be calculated, it can be used in the meta-analysis.13
The problem most meta-analysts will face is that this marginal effect is usually
not reported. Typically, only the regression coefficients (α1, α2, α3) and their
standard errors will be available, rather than the marginal effect.14 It appears that
many authors are concerned only with the statistical significance of the individual
interaction and non-linear terms rather than the practical significance of the
overall effect. In some cases the marginal effect will be reported, and its standard
error will not; it is rare for both to be reported. The biggest hurdle here is that the
covariances are almost never reported.15 This makes it difficult to include effects
with interactions or polynomial terms and yet also accommodate publication
selection. Meta-analysts could, however, use other weights, such as sample size
or journal impact factors instead of standard errors. In practice, the meta-analyst
will likely be forced to consider two strategies for dealing with this issue.
She can ignore the interaction and non-linear terms. The MRA can be applied
to only those estimates from models that do not include any interactions or non-
linear terms:
The main disadvantage here is that part (perhaps a very important part) of the
literature is discarded, and this might introduce systematic bias into the MRA.
Alternatively she can conduct separate meta-analyses. The conditional terms
can be used in a separate meta-analysis that explores the existence of a genuine
empirical interactive term. For example, a meta-analysis can be carried out on
the Hit · Kit term and separately for the H it2 term. If these meta-analyses reveal
that these terms are important, then their results could be combined with those
from the linear meta-analysis. If the meta-analyses indicate that these terms are
not statistically significant, or if the size of the effect is very small and of little
practical significance, then the interaction terms can be ignored and the MRA
136 Further topics in meta-regression analysis
conducted only on the linear terms from all studies. That is, estimates of the
effect of H on Y from equation (7.1) can be combined in the one meta-analysis,
given that the interaction terms and the squared term can be taken to have no
effect on Y. Note, however, that dummy variables should still be included in the
MRA to identify those estimates derived from models with interactions and/or
squares.
A One One
Multiple effect sizes
B One Several
C Several One
D Several Several
Further topics in meta-regression analysis 137
effects differ because they involve different explanatory variables. It would not
be appropriate to pool the different effect sizes (from X1, X2 and X3) together
and conduct a standard WLS-MRA on the pooled data because they measure
conceptually different dimensions. Nonetheless, there is dependence in the effects
of X1, X2 and X3, because they are drawn from the same studies using the same data
and dependent variable.
Scenario C occurs where multiple effect sizes arise from studies reporting the
econometric analysis of more than one dependent variable, while scenario D
involves analysis of related though different explanatory variables and different
dependent variables. As an example of scenario C, consider the literature on
economic freedom. Economic theory predicts that economic freedom will affect
factor accumulation, especially investment in physical capital and human capital,
as well as economic growth. Hence, theory predicts that economic freedom will
impact on economic growth directly and also indirectly via capital accumulation.
Empirical studies offer estimates on the effects of economic freedom on different
dependent variables (growth (Y1), human capital (Y2), and physical capital (Y3)).
Researchers might be interested in analyzing all or some of these effects. In this
case, there is a single explanatory variable (economic freedom) that has an effect
on various dependent variables. Instead of running the MRA separately for each
dependent variable, multivariate MRA can be applied allowing the joint estimation
of several MRA equations, one for each dependent variable.16
Obviously, where there is interest in multiple effect sizes, the appropriate
data will need to be collected and coded. In many cases, this information can be
collected from the same pool of studies. In other cases, the information will need
to be collected from different but overlapping literatures. Here we illustrate the use
of two related estimators for dealing with such cases of cross-correlated MRAs,
seemingly unrelated regressions (SUR) and three-stage least squares (3SLS).17
where BEER, WINE and LIQUOR are the own price elasticities for these different
alcoholic beverages, SEbeer, SEwine and SEliquor are the standard errors for the
elasticities of beer, wine and liquor, respectively, and εji are the error terms for a
given product and study, i = 1,2,..., L. In system (7.3), the right-hand-side vari-
ables are specific to each of the equations; every explanatory and every dependent
variable is different. This corresponds to scenario D of Table 7.1. More generally,
the MRA model might take the form of:
where the vector Z contains variables such as the country sample, the estima-
tor used, the omission of important explanatory variables in the original study
and the source of data. This system of equation can easily be extended to include
the K-vector of variables interacted with standard error (recall Chapter 5). The
Z-vector may contain the same type of variables across the three equations. Note
that system (7.4) can easily be extended to include other equation-specific vari-
ables. While OLS still produces consistent estimates, the benefit of SUR over
OLS is that it can result in efficiency gains;20 SUR enables the cross-correlation in
the errors to be incorporated in the estimation.21
We illustrate the application of SUR with 258 estimates on the own price
elasticity of alcohol. The data come from 59 studies that report elasticities for
all three types of alcohol.22 Each of these studies reports estimates for all three
Further topics in meta-regression analysis 139
types of alcohol. Within each study, estimates for the different types of alcohol
are derived using the same time period, country and estimator. Hence, there
is every reason to expect that there is significant data dependence within each
study. Table 7.2 presents the results of the OLS estimates of each of the FAT-PET
regressions (columns 1 and 2) and SUR (columns 3 and 4), for equation (7.3).
While the coefficients are similar (with overlapping confidence intervals), the
advantages of using SUR in this case are that it provides more efficient estimates
and enables us to test whether the coefficients are the same across the equations.
The Breusch–Pagan test strongly rejects the independence of the three equations
(χ2(3) = 128.63; p < 0.001), suggesting that the SUR estimates are preferred to
the OLS ones. The correlation between the errors in the beer and spirits equations
is 0.64, between spirits and wine it is 0.26, and between beer and wine it is 0.13.
That is, there is a rather strong correlation between the beer and spirits elasticities
but not for the other combinations.
Testing the null hypothesis that all selection bias corrected alcohol elasticities
are zero (H0: α0 = β0 = γ0 = 0) is easily rejected ( χ2(3) = 1107.66; p < 0.001), as is
the null that the alcohol elasticities are identical (H0: α0 = β0 = γ0, χ2(3) = 485.43;
p < 0.001). Similarly, the null that the publication bias terms are zero (H0: α1 =
β1 = γ1 =0) is also rejected ( χ2(3) = 26.99; p < 0.001). Note that the MRA presented in
Table 7.2 is meant as an illustration only. The full meta-regression analysis should
obviously be extended to include a wide range of controls (versions of equation
(7.4)) as discussed in earlier chapters. It is worth noting that this finding appears
to run counter to Hunter and Schmidt (2004) who conjecture that publication
bias is less likely where a literature revolves around multiple associations. This
remains an important research area. Our own conjecture is that publication bias is
a complex research phenomenon, and it is entirely possible that selection occurs
across a wide range of associations. For example, it is plausible to suspect that
a relatively “good” regression coefficient for beer might not be reported if the
associated regression coefficient for wine is “undesirable”.
An example of the application of this estimator is Kotchen and Schulte’s (2009)
meta-study of the fiscal impact of alternative land uses. The authors estimate a
SUR model involving three equations for three land-use categories (residential,
commercial and open space). The OLS results are not reported, but the authors
note that OLS and SUR gave substantially similar results. A second example is the
here wbij and ebij denote the elasticity of wages and employment with respect to
immigration, respectively. In this framework, the elasticity of employment affects
the elasticity of wages and vice versa. The authors accommodate this endogeneity
by estimating the model using three-stage least squares and find that the results
differ from OLS. Naturally, when estimating such systems it is important to be
mindful of identification and to incorporate appropriate exclusion restrictions in
the MRA.
This framework appears to be important when econometric studies focus
on more than one effect size. This applies to both the estimates of the genuine
empirical effect, as well as publication bias. For example, when estimating
production functions, authors need to account for the marginal products of both
capital and labor.24 When a literature provides joint estimates of two or more
related effects, the MRA should be modeled accordingly.
7.8 Meta-meta-analysis
As the number of meta-analyses grows, so does the pool of information on effect
sizes that can potentially be compared within and between literatures. This wealth
of information remains relatively untapped. Much can be learned from past
Further topics in meta-regression analysis 141
MRAs, and the tools of meta-analysis can be employed here also. We have seen
in previous chapters and in literally hundreds of studies that MRA is an effective
tool for modeling research heterogeneity. Perhaps, meta-meta-regression analysis
can help identify patterns among meta-studies as well?
Surveys of meta-analyses can be accomplished by a traditional narrative review,
by a meta-meta-analysis of the same effect sizes, or by a meta-meta-analysis
across different effect sizes. Wherever possible, our own preference is for the
more objective statistical analysis of a meta-analysis. It is important to note that
we are not advocating the use of cumulative statistics for conceptually different
hypotheses. Rather, we are advocating combining meta-analyses that test the same
hypothesis (e.g. the effects of population on growth) or some other comparable
effect between meta-analyses (such as the degree of publication bias).
where ηij is the ith meta-estimate of the income elasticity from the jth meta-
analysis, Xij is a vector of explanatory variables and εij is the error term. Table
7.3 presents estimates from this M2RA. For this illustrative example, moderator
variables, the vector Xij , includes whether the meta-analysis corrects for selection
bias, whether it considers only wage risk or only stated preference studies (with all
types of studies combined used as the base) and whether the study was published.
15
10
Precision
5
0
0 1 2 3 4
Income elasticity
7.9 Summary
In this chapter we have explored several aspects, complications and potential
applications of meta-regression analysis for economics and business. Like econo-
metrics, the findings from MRA have many scientific and practical uses, and their
models have great flexibility and potential in the modeling of empirical economic
research. Because MRA is a relatively young discipline, much of its potential
remains underutilized. Thus the future of meta-regression analysis in economics
and business is bright.
146 Further topics in meta-regression analysis
This chapter has outlined alternative uses of MRA, strategies for specifying an
MRA model, the success of MRA predictions, and how MRA may be generalized
to explain and estimate multiple empirical effects and multiple MRAs. This
discussion is only a sketch of the many facets of meta-regression analysis, and
much more remains to be explored and further developed.
MRA models have the potential to contribute to the understanding of research
well beyond that of single effect sizes in isolated research literatures. We show
how MRA can be used to assist with the analysis of multiple effect sizes. It
can also be employed to analyze the findings of meta-analyses. As more meta-
analyses become available, it will be important to assess their findings to guide
both new primary studies and new meta-analyses. This presents new opportunities
and challenges for meta-analysts.
8 Summary and conclusions
Contemporary research on any important topic tends to be vast, and its findings
are often disparate and widely dispersed. This “flood of numbers” threatens to
drown economic knowledge and sensible policy action (Heckman, 2001). Yet,
without some objective and comprehensive understanding of economics research,
informed policy is impossible.1 Unfortunately, conventional narrative reviews are
not up to the task. It is all too easy for conventional reviewers to ignore find-
ings that do not fit into their theories or ideology. Without objective standards,
it is a hopeless task to assess what the genuine message of a given conventional
review might actually be. Hence, experienced researchers often discount narrative
reviews. And yet, there is an indisputable need for research synthesis.
Meta-analysis offers a critical and objective methodology that integrates
conflicting research findings and can filter out some of the biases routinely found
among reported research results. In hundreds of applications, meta-analysis
has proven to be effective at cleansing the often murky and muddy waters of
the ever-growing pool of econometric results. We believe that meta-analysis is
indispensable for a clear understanding of actual economic phenomena.
In the preceding chapters, we have shown how meta-regression analysis can
be used to:
Researchers must avoid relying on their own priors or the research communi-
ty’s norms regarding factors such as journal quality and the authors’ institutional
affiliations when developing criteria for study inclusion. Applying such a lens to
the inclusion of research results is likely to distort a meta-analysis in unknown
ways. If these or other potential dimensions of study “quality” are deemed to be
important, they should be coded and included explicitly in the meta-regression
analysis itself. An inclusive and replicable approach to the collection and coding
of a research record offers a scientific basis for the evaluation of research and,
hence, provides an objective evidence base to policy and the understanding of
economics and business.
This meta-regression model is estimated using weighted least squares with preci-
sion squared used as weights. The model contains tests for both publication bias
(the funnel-asymmetry test, FAT; H0: β1 = 0), and the presence of a genuine effect
beyond publication selection (the precision-effect test, PET; H0: β0 = 0).2
The precision-effect estimate with standard error (PEESE) is the estimated
β0 in:3
The PEESE provides a better estimate of the actual empirical effect corrected for
publication bias, when there is one.
Regardless of the outcome of the FAT, the PET should be used to test for the
existence of a genuine effect beyond potential contamination from publication
bias. If the meta-analysis does not permit a rejection of H0: β0 = 0, then this area
of research fails to provide clear evidence of any genuine empirical effect. The
PEESE should be used to estimate the magnitude of the empirical effect when the
PET provides evidence that one exists.
This area of publication selection bias requires further research and will no
doubt continue to evolve. Indeed, our own approach to meta-analysis has changed
over the years. For example, we no longer advocate the use of the MST test
(Chapter 4) even in conjunction with other publication selection MRA methods.
However, a consensus seems to be emerging that FAT-PET-PEESE and similar
meta-regression models provide the best correction for publication bias (Stanley,
2008; Moreno et al., 2009a, 2009b, Rücker et al., 2011).
150 Summary and conclusions
It is also very important to evaluate the size of the corrected effect for practical
economic or policy significance. Statistically significant effects that are practically
small need to be given correspondingly little weight.
effecti = β0 + ∑βZ
k ki + β1SEi + ∑ δ SE K
j i ji + εi (5.5)
This list is adapted from Nelson and Kennedy’s (2009) excellent review of
meta-analyses in environmental economics. It is our sincere hope that the
advice and discussions given in this book will help novice researchers to avoid
these pitfalls and thereby contribute to the rigor and acceptance of MRA in
business and economics.
Coda
Meta-regression analysis gives economists and business researchers the ability
to summarize and evaluate their areas of empirical inquiry using the same tools
and methods that produced the research in question. Our approach empow-
ers empirical researchers themselves to evaluate their fields of specialization
using their own methods, or similar ones, rather than relying on historians or
philosophers of science to document and evaluate progress in economics and
business.
Decades ago, logical positivists and naïve falsificationists sought to employ
strictly logical criteria to assess a scientific research program and to indicate
fruitful directions for its future progress (Popper, 1959; Lakatos, 1970). Since
then, economic methodologists and philosophers of science have taken a more
“naturalistic” turn; that is, a “turn away from a priori philosophy and towards a
philosophical vision that is informed by contemporary scientific practice” (Hands,
2001: 129). This is precisely what MRA offers empirical economics. A well-
conducted MRA may also serve as the basis for an internal philosophical appraisal
152 Summary and conclusions
of the scientific progress obtained in a given area of business or economics
research.4 MRA’s potential for deeper philosophical reflection and evaluation has
also been largely untapped.
Rather than attempting to develop a new logic of “induction” or “adduction”
(Blaug, 1980) to justify econometric inferences or employing some a priori
normative methodology of empirical inquiry, MRA takes econometrics as it is.
Using econometric methods, MRA summarizes, corrects, tests and evaluates
empirical econometric results. Economists cannot object that this approach is
inappropriate or invalid, unless they also reject empirical econometrics itself.
Of course, econometrics and meta-econometrics (MRA) have their limitations
and weaknesses. But a rejection of econometrics, en masse, would be quite
uneconomical, a great waste of the massive research resources used to produce
it. Furthermore, we believe that econometrics often provides the best empirical
evidence that we have for many important questions of economic theory and
policy. Readers of this book will realize that we are not naïve proselytizers of
econometrics. Rather, we believe that econometric analysis often offers much
less than is claimed. However, its modest findings usually add up to much more
than nothing. Even when little remains in a given area of empirical inquiry
after likely misspecification and selection biases are accommodated, this too is
important to know.
Understandably, many applications in meta-analysis in economics have
followed methodologies borrowed from other disciplines, most notably medicine
and psychology. However, we have pointed out in this book that the meta-analysis
of econometric studies requires a fresh meta-analytic approach. The aim of this
book is to map out some of the key components of such methodological advance
and differentiation. First, we have highlighted the need to treat observational
data differently from experimental data. Second, selection bias is as much, if not
more, a problem in economics as it is in other disciplines. Hence, it is imperative
that meta-analyses examine their areas of research for selection bias, just as they
should explore likely misspecification biases. Third, we caution researchers from
applying widely used estimators, such as random-effects weighted averages and
random-effects MRA models, to econometrics estimates.
Meta-analysis in economics is still relatively young. With youth comes hope
and great promise. Just as econometric practice has evolved, so too will meta-
analysis. The challenge for researchers is to continue to apply existing methods,
to question them, and to develop new and improved approaches as needed.
Much work lies ahead. We need to understand more about the various MRA
models, especially the Z/K multiple MRA framework. Many more simulations
along the lines of Stanley (2008), Koetse et al. (2010), Callot and Paldam (2011)
and Stanley and Doucouiagos (2011) are needed to validate and test MRA
methodology. The replication of prior meta-studies is also important. Do the
conclusions from prior meta-studies survive when re-estimated with the ever-
growing economics and business research? Perhaps new insights and lessons
can be drawn from such replications. Because much progress has been made in
recent years, current “best practice” specification of MRAs and the variables
Summary and conclusions 153
they should include requires further validation. It therefore remains important
for researchers to explore the sensitivity of alternative MRA specifications.
While new paths have been discovered, many require exploration and further
development.
We see an evolving, yet bright, future for meta-econometrics. Its past successes
and enormous potential, on so many levels, guarantee that meta-econometrics
will continue to extend its reach and be further developed. May it shine its light
on your area of research.
Notes
1 Introduction
1 Doucouliagos and Stanley (2012) conduct a meta-meta-analysis on the magnitude of
publication selection bias (see Chapter 4 below) and the degree to which there is a
competition of ideas about the prevailing economic theory in question. We find that
debate and theory competition reduce the severity of this bias and its distorting effect
on policy-relevant empirical magnitudes.
2 Experiments with human subjects cannot eliminate all potential threats to the validity
of their inferences either; however, it is likely that a well-designed RCT will avoid
some of the confounding influences that remain a concern to econometric analysis.
3 There are of course model specification tests, and they can help to reduce this ambi-
guity. However, they are infrequently employed, and their results will always leave
some substantial uncertainty. These tests must, by necessity, make some assumptions
about generic specification, and, even in the best of circumstances, there will remain
the possibility of committing a type I or type II error.
4 Dozens of economic meta-analyses have corroborated the existence and importance
of publication selection bias. See Chapter 4 for relevant references and an extended
discussion of publication selection.
5 We use this example only as one clear illustration of how ideology often influences
economics. It is not meant to be a comprehensive statement about the causes of the
“Great Recession.”
6 For example, UK’s Department for International Development, in concert with other
agencies, has funded several dozen systematic reviews and meta-analyses in 2010–11.
7 For two decades, the UK has been a world leader in attempting to better align chief
executive pay to the performance of their companies and the interests of their sharehold-
ers. UK’s regulations are generally called “comply or explain,” because compliance is
not technically mandatory, but failure to comply with government regulations requires
the company to explain why, publicly (Arcot et al., 2010).
8 See Chapter 3 for an extended discussion of the weaknesses of vote counting. Vote
counts can be greatly distorted by publication selection, which is known to be a seri-
ous problem in economics and business (Chapter 4). Furthermore, when statistical
tests have low power, the probability that vote counts come to the wrong conclusion
increases as research accumulates (Hedges and Olkin, 1985).
9 Statisticians have long argued that Glass’s g is biased and inefficient, largely due to
using a poor statistical measure of within group variation (Hedges and Olkin, 1985).
Nonetheless, more sophisticated versions of effect size based on g remain widely in use.
10 Here, we use the term “multivariate” in its broadest, multidimensional sense, as mul-
tiple regression or similar statistical analyses. In statistics, multivariate analysis is often
limited to situations where multiple dependent variables are being jointly modeled, or
what econometricians would call simultaneous or structural equation systems. Meta-
Notes 155
regression analysis can be employed in both of these ways. Chapter 7 discusses the use
of multiple MRA equation systems (e.g. three-stage least squares) to explain related
empirical effects. In other chapters, “multivariate” is meant only as a generic, non-tech-
nical substitute for “multiple” MRA.
11 To give an accurate count of the number of meta-studies over time would require the
type of comprehensive literature searching that characterizes meta-analysis. See the
next chapter (Chapter 2) for guidance on how to conduct such an exhaustive search.
Because our intention here is merely illustrative, we have made no effort to provide
a comprehensive search of meta-analyses in economics. Nonetheless, we conjecture
that such a meta-meta-analysis would reveal a similar pattern of growth.
12 The VSL is not a measure of the value of an actual life. Rather, it is the marginal will-
ingness to pay for infinitesimal risk reductions for different people that are aggregated
to a single statistical life.
Abreu, M., de Groot, H.L.F. and Florax, R.G.M. (2005) A meta-analysis of beta-
convergence: The legendary two-percent, Journal of Economic Surveys, 19: 389–420.
Adams, D.C., Gurevitch, J. and Rosenberg, M.S. (1997) Resampling tests for meta-analysis
of ecological data, Ecology, 78: 1277–83.
Akerlof, G.A. (1982) Labor contracts as partial gift exchange, Quarterly Journal of
Economics, 97: 543–69.
Albers, S., Mantrala, M.K. and Sridhar, S. (2010) Personal selling elasticities: A meta-
analysis, Journal of Marketing Research, 47: 840–53.
Alston, J.M., Marra, M.C., Pardey, P.G. and Wyatt, T.J. (2000) Research returns redux: A
meta-analysis of the returns to agricultural R&D, Australian Journal of Agricultural
and Resource Economics, 44: 185–215.
Andrews, E.L. (2008) Greenspan concedes error on regulation, New York Times, October 23.
Arcot, S., Bruno, V. and Faure-Grimaud, A. (2010) Corporate governance in the UK: Is the
comply or explain approach working?, International Review of Law and Economics,
30: 193–201.
Arnould, R.J. and Nichols, L.M. (1983) Wage-risk premiums and worker’s compensation:
a refinement of estimates of compensating wage differentials, Journal of Political
Economy 91: 332–40.
Ashenfelter, O., Harmon, C. and Oosterbeek, H. (1999) A review of estimates of the
schooling/earnings relationship, with tests for publication bias, Labour Economics, 6:
453–70.
Ayers, I. (2007) Super Crunchers, New York: Bantam.
Ballal, S. and Rubenstein, R. (2009) The effect of tax and expenditure limitations on public
education resources: A meta-regression analysis, Public Finance Review, 37: 665–85.
Baltagi, B.H., Garvin, S. and Kerman, S. (1989) Further Monte Carlo evidence on seemingly
unrelated regressions with unequal number of observations, Annales d’Economie et de
Statistique, 14: 103–15.
Banzhaf, S.H. and Smith, K.V. (2007) Meta-analysis in model implementation: Choice
sets and the valuation of air quality improvements, Journal of Applied Econometrics,
22: 1013–31.
Bateman, I.J. and Jones, A.P. (2003) Contrasting conventional with multi-level modeling
approaches to meta-analysis: Expectation consistency in U.K. woodland recreation
values, Land Economics, 79: 235–58.
Becker, B.J. and Wu, M-J. (2007) The synthesis of regression slopes in meta-analysis,
Statistical Science, 22: 414–29.
References 169
Begg, C.B. and Berlin, J.A. (1988) Publication bias: A problem in interpreting medical
data, Journal of the Royal Statistical Society, Series A, 151: 419–63.
Bellavance, F., Dionne, G. and Lebeau, M. (2009) The value of a statistical life: A meta-analysis
with a mixed effects regression model, Journal of Health Economics, 28: 444–64.
Bergstrom, J.C. and Taylor, L.O. (2006) Using meta-analysis for benefits transfer: Theory
and practice, Ecological Economics, 60: 351–60.
Berwick, D.M., Calkins, D.R., McCannon, C.J. and Hackbarth, A.D. (2006) The 100,000
lives campaign: Setting a goal and a deadline for improving health care quality, Journal
of the American Medical Association, 295: 324–27.
Bijmolt, T.H.A., van Heerde, H.J and Pieters, R.G.M. (2005) New empirical generalizations
on the determinants of price elasticity, Journal of Marketing Research, 42: 141–56.
Bland, J.M. (1988) Discussion of the paper by Begg and Berlin, Journal of the Royal
Statistical Society, Series A, 151: 450.
Blaug, M. (1980) The Methodology of Economics, Cambridge: Cambridge University Press.
Boockmann, B. (2010) The combined employment effects of minimum wages and labor
market regulation: A meta-analysis. IZA Discussion Paper No. 4983.
Borenstein, M., Hedges, L.V., Higgins, J.P.T. and Rothstein, H.R. (2009) Introduction to
Meta-Analysis, Chichester: Wiley.
Bowland, B.J. and Beghin, J.C. (2001) Robust estimates of value of a statistical life for
developing economies, Journal of Policy Modeling, 23: 385–96.
Brander, L.M., Florax, R.J.G.M. and Vermaat, J.E. (2006) The empirics of wetland valuation:
A comprehensive summary and a meta-analysis of the literature, Environmental and
Resource Economics, 33: 223–50.
Brons, M., Nijkamp, P., Pels, E. and Rietveld, P. (2008) A meta-analysis of the price
elasticity of gasoline demand: A SUR approach, Energy Economics, 30: 2105–22.
Brown, S.P. and Stayman, D.M. (1992) Antecedents and consequences of attitude toward
the ad: A meta-analysis, Journal of Consumer Research, 19: 34–51.
Burkhauser, R., Couch, K.A. and Wittenburg, D.C. (2000) A reassessment of the new
economics of the minimum wage literature with monthly data from the Current
Population Survey, Journal of Labor Economics, 18: 653–80.
Callot, L. and Paldam, M. (2011) The problem of natural funnel asymmetries: a simulation
analysis of meta-analysis in macroeconomics, Research Synthesis Methods, 2: 84–102.
Capelle-Blancard, G. and Couderc, N. (2007) How do shareholders respond to downsizing?
A meta-analysis. Paper presented at the MAER Network Colloquium, September 27–30,
2007, Sønderborg, Denmark.
Card, D.E. and Krueger, A.B. (1995a) Time-series minimum-wage studies: A meta-analysis,
American Economic Review, 85: 238–43.
Card, D.E. and Krueger, A.B. (1995b) Myth and Measurement: The New Economics of the
Minimum Wage, Princeton, NJ: Princeton University Press.
Card, D.E., Kluve, J. and Weber, A. (2010). Active labor market policy evaluations: A
meta-analysis, Economic Journal, 120: F452–77.
Chalmers, T.C., Matta, R.J., Smith, H. and Kunzler, A.M. (1977) Evidence favoring the
use of anticoagulants in the hospital phase of acute myocardial infarction, New England
Journal of Medicine, 297: 1091–6.
Charemza, W.W. and Deadman, D.F. (1997) New Directions in Econometric Practice, 2nd
edn, Cheltenham: Edward Elgar.
Cohen, J. (1988) Statistical Power Analysis in the Behavioral Sciences, 2nd edn, Hillsdale,
NJ: Erlbaum.
170 References
Colegrave, A.D. and Giles, M.J. (2008) School cost functions: A meta-regression analysis,
Economics of Education Review, 27: 688–96.
Connor, J.M. and Bolotova, Y. (2006) Cartel overcharges: Survey and meta-analysis,
International Journal of Industrial Organization, 24: 1109–37.
Cooper, H.M. and Hedges, L.V. (eds.) (1994) Handbook of Research Synthesis, New York:
Russell Sage.
Copas, J. (1999) What works? Selectivity models and meta-analysis, Journal of the Royal
Statistical Society, Series A, 162: 95–109.
Coric, B. and Pugh, G. (2010) The effects of exchange rate variability on international
trade: A meta-regression analysis, Applied Economics, 42: 2631–44.
Dalhuisen, J.M., Florax, R.J.G.M., de Groot, H.L.F. and Nijkamp, P. (2003) Price and
income elasticities of residential water demand: A meta-analysis, Land Economics, 79:
292–308.
Davidson, J. (2000) Econometric Theory, Malden, MA: Wiley- Blackwell.
Davidson, R. and MacKinnon, J.G. (2004) Econometric Theory and Methods, Oxford:
Oxford University Press.
Day, B.H. (1999) A meta-analysis of wage-risk estimates of the value of statistical life’,
Centre for social and economic research on the global environment. Working Paper.
De Blaeij, A., Florax, R.J.G.M., Rietveld, P. and Verhoef, E.T. (2003) The value of statistical
life in road safety: A meta-analysis, Accident Analysis and Prevention, 35: 973–86.
De Dominicis, L., Florax, R.J.G.M. and De Groot, H.L.F. (2008) Meta-analysis of the
relationship between income inequality and economic growth, Scottish Journal of
Political Economy, 55: 654–82.
De Long, J.B. and Lang, K. (1992) Are all economic hypotheses false? Journal of Political
Economy, 100: 1257–72.
De Mooij, R.A. and Ederveen, S. (2003) Taxation and foreign direct investment: A synthesis
of empirical research, International Tax and Public Finance, 10: 673–93.
Dekker, T., Brouwer, R., Hofkes, M. and Moeltner, K. (2008) The effect of risk context
on the value of a statistical life: A Bayesian meta-model. Institute for Environmental
Studies, Working Paper W08/23.
Demsetz, H. (1974) Two systems of belief about monopoly. In H.J. Goldschmid, H.M.
Mann and J.F. Weston (eds), Industrial Concentration: The New Learning, Boston:
Little, Brown and Company, pp.164–84.
Dionne, G. and Michaud, P.C. (2002) Statistical analysis of value-of-life estimates using
hedonic wage method. Working Paper 02–01. Ecole des Hautes Etudes Commerciales,
Montreal.
Disdier, A-C. and Head, K. (2008) The puzzling persistence of the distance effect on
bilateral trade, Review of Economics and Statistics, 90: 37–48.
Djankov S. and Murrell, P. (2002) Enterprise restructuring in transition: A quantitative
survey, Journal of Economic Literature, 40: 736–92.
Doucouliagos, C. (H.) (1995) Worker participation and productivity in labor-managed and
participatory capitalist firms: A meta-analysis, Industrial and Labor Relations Review,
49: 58–77.
Doucouliagos, C. (H.) and Laroche, P. (2003) What do unions do to productivity: A meta-
analysis, Industrial Relations, 42: 650–91.
Doucouliagos, C. (H.) and Laroche, P. (2009) Unions and profits: A meta-analysis,
Industrial Relations, 48: 146–84.
Doucouliagos, C. (H.) and Paldam, M. (2006) Aid effectiveness on accumulation: A meta
study, Kyklos, 59: 227–54.
References 171
Doucouliagos, C. (H.) and Paldam, M. (2008) Aid effectiveness on growth: A meta study,
European Journal of Political Economy, 24: 1–24.
Doucouliagos, C. (H.) and Paldam, M. (2009) The aid effectiveness literature: The sad
results of 40 years of research, Journal of Economic Surveys, 23: 433–61.
Doucouliagos, C. (H.) and Paldam, M. (2010) Conditional aid effectiveness: A meta study,
Journal of International Development, 22: 391–410.
Doucouliagos, C. (H.) and Paldam, M. (2011a) The robust result in meta-analysis of aid
effectiveness: A response to Mekasha and Tarp. Aarhus University, Department of
Economics Working Paper 2011–15.
Doucouliagos, C. (H.) and Paldam, M. (2011b) Does development aid reward good behavior?
A meta-analysis of the effects of human rights and democracy. Paper presented at the
2011 European Public Choice Society Conference, April, Rennes, France.
Doucouliagos, C. (H.) and Stanley, T.D. (2008) A tale of two biases. Paper presented at
the Nancy Workshop on Meta-Analysis in Economics and Business, 2008, University
of Nancy.
Doucouliagos, C. (H.) and Stanley, T.D. (2009) Publication selection bias in minimum-
wage research? A meta-regression analysis, British Journal of Industrial Relations, 47:
406–28.
Doucouliagos, C. (H.) and Stanley, T.D. (2012) Theory competition and selectivity: Are all
economic facts greatly exaggerated? Journal of Economic Surveys, forthcoming. Also
available as Deakin University, Economics Working Paper, 2008–06.
Doucouliagos, C. (H.) and Ulubasoglu, M. (2006) Economic freedom and economic
growth: Does specification make a difference? European Journal of Political Economy,
22: 60–81.
Doucouliagos, C. (H.) and Ulubasoglu, M. (2008) ‘Democracy and economic growth: A
meta-analysis, American Journal of Political Science, 52: 61–83.
Doucouliagos, C. (H.), Laroche, P. and Stanley, T.D. (2005) Publication bias in union-
productivity research, Relations Industrielles/Industrial Relations, 60: 320–46.
Doucouliagos, C., Haman, J. and Stanley, T.D. (2012a) Pay for performance and corporate
governance reform, Industrial Relations, forthcoming. Available as Deakin University,
Economics Working Paper No. 2010–04.
Doucouliagos, C. (H.), Stanley, T.D. and Giles, M. (2012b) Are estimates of the value of a
statistical life exaggerated? Journal of Health Economics, 31: 197–206.
Duval, S. and Tweedie, R. (2000) A nonparametric “trim and fill” method of accounting
for publication bias in meta-analysis, Journal of the American Statistical Association,
95: 89–98.
Efendic, A., Pugh, G. and Adnett, N. (2011) Institutions and economic performance: A
meta-regression analysis, European Journal of Political Economy, 27: 586–599.
Égert, B. and Halpern, L. (2006) Equilibrium exchange rates in Central and Eastern Europe:
A meta-regression analysis, Journal of Banking and Finance, 30: 1359–74.
Egger, M., Smith, G.D., Schneider, M. and Minder, C. (1997) Bias in meta-analysis detected
by a simple, graphical test, British Medical Journal, 315: 629–34.
Feige, E.L. (1975) The consequence of journal editorial policies and a suggestion for
revision, Journal of Political Economy, 83: 1291–6.
Feld, L.P. and Heckemeyer, J.H. (2011) FDI and taxation: A meta-study, Journal of
Economic Surveys, 25: 233–72.
Fidrmuc, J. and Korhonen, I. (2006) Meta-analysis of the business cycle correlation between
the Euro area and the CEECs, Journal of Comparative Economics, 34: 518–37.
172 References
Fischer, C. and Morgenstern, R.D. (2006) Carbon abatement costs: Why the wide range of
estimates, Energy Journal, 27: 73–86.
Fisher, R.A. (1932) Statistical Methods for Research Workers, 4th edn, London: Oliver
and Boyd.
Florax, R.J.G.M. (2002) Accounting for dependence among study results in meta-
analysis: methodology and applications to the valuation and use of natural resources.
Series Research Memoranda 0005, VU University Amsterdam, Faculty of Economics,
Business Administration and Econometrics.
Florax, R. and Poot, J. (2007) Learning from the flood of numbers: Meta-analysis in
economics. Paper presented at the Aarhus Colloquium of Meta-analysis in Economics,
September 27–30, Sandbjerg Manor, Sønderborg, Denmark.
Fortin, B. and Lanoie, P. (2000) Effects of workers’ compensation: A survey. In G. Dionne
(ed.), Handbook of Insurance. Boston: Kluwer Academic Publishers.
Galbraith, R.F. (1988) A note on graphical presentation of estimated odds ratios from
several clinical trials, Statistics in Medicine, 7: 889–94.
Gallet, C.A. (2007) The demand for alcohol: A meta-analysis of elasticities, Australian
Journal of Agricultural and Resource Economics, 51: 121–35.
Gallet, C.A. (2010) The income elasticity of meat: A meta-analysis, Australian Journal of
Agricultural and Resource Economics, 54: 477–490.
Gallet, C.A. and List, J.A. (2003) Cigarette demand: A meta-analysis of elasticities, Health
Economics, 12: 821–35.
García-Quevedo, J. (2004) Do public subsidies complement business R&D? A meta-
analysis of the econometric evidence, Kyklos, 57: 87–102.
Gerber, A.S. and Malhorta, N. (2008) Publication bias in empirical sociological research,
Sociological Methods and Research, 25: 1–28.
Gerber, A.S., Green, D.P. and Nickerson, D. (2001) Testing for publication bias in political
science, Political Analysis, 9: 385–92.
Glass, G.V. (1976) Primary, secondary, and meta-analysis of research, Educational
Researcher, 5: 3–8.
Glass, G.V., McGaw, B. and Smith, M.L. (1981) Meta-Analysis in Social Research, Beverly
Hills, CA: Sage.
Görg, H. and Strobl, E. (2001) Multinational companies and productivity spillovers: A
meta-analysis, Economic Journal, 111: F723–39.
Greenberg, D.H., Michalopoulos, C. and Robins, P.K. (2003) A meta-analysis of government-
sponsored training programs, Industrial and Labor Relations Review, 57: 31–53.
Greene, W.H. (1990) Econometric Analysis, New York: Macmillan.
Greenspan, A. (2007) The Age of Turbulence: Adventures in a New World, London: Penguin
Press.
Griliches, Z. (1977) Estimating the returns to schooling: Some econometric problems,
Econometrica, 45: 1–22.
Gujarati, D.N. (1995) Basic Econometrics, New York: McGraw-Hill.
Hands, D.W. (2001) Reflection without Rules, Cambridge: Cambridge University Press.
Harlow, L.L., Mulaik, S.A. and Steiger, J.H. (eds.) (1997). What If There Were No
Significance Tests? Mahwah, NJ: Erlbaum.
Hausman, J.A. (1978) Specification tests in econometrics, Econometrica, 46: 1251–271.
Havránek, T. (2010) Rose effect and the Euro: Is the magic gone? Review of World
Economics, 146: 241–61.
Heckman, J.J. (1979) Sample selection bias as a specification error, Econometrica, 47:
153–61.
References 173
Heckman, J.J. (2001) Micro data, heterogeneity, and the evaluation of public policy: Nobel
lecture, Journal of Political Economy, 109: 673–748.
Hedges, L.V. (1992) Modeling publication selection effects in meta-analysis, Statistical
Science, 7: 246–55.
Hedges, L.V. and Olkin, I. (1985) Statistical Methods for Meta-Analysis, Orlando, FL:
Academic Press.
Hedges, L.V. and Vevea, J.L. (1996) Estimating effect size under publication bias: Small
sample properties and robustness of a random effects selection model, Journal of
Educational and Behavioral Statistics, 21: 299–332.
Hennessy, P. (2011) Prime Ministers unite against Tory right, The Telegraph, June 4.
Higgins J.P.T. and Green, S. (eds) (2008) Cochrane Handbook for Systematic Reviews of
Interventions, Chichester: Wiley.
Higgins J.P.T. and Thompson, S.G. (2002) Quantifying heterogeneity in a meta-analysis,
Statistics in Medicine, 21: 1539–58.
Hirschberg, J.G., Lye, J.N. and Slottje, D.J. (2008) Inferential methods for elasticity
estimates, Journal of Econometrics, 147: 299–315.
Holmgren, J. (2007) Meta-analysis of public transport demand, Transportation Research,
Part A: Policy and Practice, 41: 1021–35.
Hopewell, S., Loudon, K., Clarke, M.J., Oxman, A.D. and Dickersin, K. (2009) Publication
bias in clinical trials due to statistical significance or direction of trial result, Cochrane
Review, Issue 1. http://www.thecochranelibrary.com
Huang, J., van den Brink, H.M. and Groot, W. (2009) A meta-analysis of the effect of
education on social capital, Economics of Education Review, 28: 454–64.
Hunt, M.M. (1997) How Science Takes Stock: The story of meta-analysis, New York:
Russell Sage Foundation.
Hunter, J.E. and Schmidt, F.L. (2004) Methods of Meta-Analysis: Correcting error and
bias in research findings, New York: Sage.
Iamsiraroj, S. (2009) FDI and growth. Unpublished PhD dissertation, Deakin University.
Iyengar, S. and Greenhouse, J.B. (1988) Selection models and the file drawer problem,
Statistical Science, 3: 109–17.
Jacobsen, J.P. (1994) The Economics of Gender, Cambridge, MA: Blackwell.
Jarrell, S.B. and Stanley, T.D. (1990) A meta-analysis of the union-nonunion wage gap,
Industrial and Labor Relations Review, 44: 54–67.
Jarrell, S.B. and Stanley, T.D. (2004) Declining bias and gender wage discrimination?
A meta-regression analysis, Journal of Human Resources, 39: 828–38.
Jensen, P.S. (2010) Testing the null of a low dimensional growth model, Empirical
Economics, 38: 193–215.
Jensen, P.S. and Würz, A.H. (2006) On determining the importance of a regressor with
small and undersized samples. Aarhus Working Paper No. 2006–8.
Johnson, H.G. (1975) On Economics and Society: Selected Essays, Chicago: University
of Chicago Press.
Johnston, N.L. and Kotz, S. (1970) Distributions in Statistics: Continuous Univariate
Distribution, New York: Wiley.
Johnston, R.J. and Duke, J.M. (2009) Characterizing welfare patterns associated with
study-invariant factors: spatial data supplemented meta-regression. Paper presented at
the Oregon State University MAER Network Colloquium, October 2009, Corvallis.
Johnston, R.J. and Rosenberger, R.S. (2010) Methods, trends and controversies in
contemporary benefit transfer, Journal of Economic Surveys, 24: 479–510.
174 References
Judge, G.G., Hill, R.C., Griffiths, W.E., Lütkepohl, H. and Lee, T.C. (1982) Introduction to
the Theory and Practice of Econometrics, New York: Wiley.
Klomp, J.G. and de Haan, J. (2010) Inflation and central bank independence: A meta-
regression analysis, Journal of Economic Surveys, 24: 593–621.
Kluve, J. and Schaffner, S. (2008) The value of life in Europe: A meta-analysis, Sozialer
Fortschritt, 10: 279–87.,
Knell, M. and Stix, H. (2005) The income elasticity of money demand: A meta-analysis of
empirical results, Journal of Economy Surveys 19: 513–33.
Kochi, I., Hubbell, B. and Kramer, R. (2006) An empirical Bayes approach to combining
and comparing estimates of the value of a statistical life for environmental policy
analysis, Environmental and Resource Economics 34, 385–406.
Koetse, M.J., de Groot, H.L.F. and Florax, R.J.G.M. (2009) A meta-analysis of the
investment-uncertainty relationship, Southern Economic Journal, 76: 283–306.
Koetse, M.J., Florax, R.J.G.M. and de Groot, H.L.F. (2010) Consequences of effect size
heterogeneity on meta-analysis: A Monte Carlo experiment, Statistical Methods and
Applications, 19: 217–36.
Konstantopoulos, S. and Hedges, L.V. (2004) Meta-analysis. In D. Kaplan (ed.) Quantitative
Methodology for the Social Sciences, Thousand Oaks, CA: Sage Publications, pp.
281–300.
Kotchen, M.J. and Schulte, S.L. (2009) A meta-analysis of cost of community service
studies, International Regional Science Review, 32: 376–399.
Krakovsky, M. (2004) Register or perish, Scientific American, 291: 18–20.
Krassoi-Peach, E. and Stanley, T.D. (2009) Efficiency wages, productivity and simultaneity:
A meta-regression analysis, Journal of Labor Research, 30: 262–8.
Krueger, A.B. (2003) Economic considerations and class size, Economic Journal, 113:
F34–63.
Laird, N. and Mosteller, F. (1988) Discussion of the paper by Begg and Berlin, Journal of
the Royal Statistical Society, Series A, 151: 456.
Lakatos, I. (1970) Falsification and the methodology of scientific research programmes. In
I. Lakatos and A. Musgrave (eds), Criticism and the Growth of Knowledge, Cambridge:
Cambridge University Press.
Leamer, E.E. and Leonard, H.B. (1983) Reporting the fragility of regression estimates,
Review of Economics and Statistics, 65: 306–17.
Leonhardt, D. (2007) Economist’s life, scored with a jazz theme, New York Times Book
Review, September 18.
Lewis, H.G. (1986) Union Relative Wage Effects: A survey, Chicago: University of Chicago
Press.
Lewis, S. and Clarke, M. (2001) Forest plots: Trying to see the wood and the trees, British
Medical Journal, 322: 1479–80.
Lindhjem, H. and Navrud, S. (2008) How reliable are meta-analyses for international
benefit transfers? Ecological Economics, 66: 425–35.
Lindhjem, H., Navrud, S. and Braathen, N.A. (2010) Valuing Lives Saved From
Environmental, Transport and Health Policies: A meta-analysis of stated preference
studies. Environment Directorate, OECD, February.
Lipsey, M.W. and Wilson, D.B. (2001) Practical Meta-Analysis, Thousand Oaks, CA: Sage.
Liu, J-T., Hammitt, J.K. and Liu, J-L. (1997) Estimated hedonic wage function and value
of life in a developing country, Economics Letters, 57: 353–8.
Longhi, S., Nijkamp, P. and Poot, J. (2010) Joint impacts of immigration on wages and
employment: review and meta-analysis, Journal of Geographical Systems, 12: 355–87.
References 175
Loomis, J.B. and White, D.S. (1996) Economic benefits of rare and endangered species:
Summary and meta-analysis, Ecological Economics, 18: 197–206.
Lovell, M.C. (1983) Data mining, Review of Economics and Statistics, 65: 1–12.
McCloskey, D.N. (1985) The loss function has been mislaid: The rhetoric of significance
tests, American Economic Review, 75: 201–05.
McCloskey, D.N. (1995) The insignificance of statistical significance, Scientific American,
272: 32–3.
Mayo, D. (1996) Error and the Growth of Empirical Knowledge, Chicago: Chicago
University Press.
Mekasha, T.J. and Tarp, F. (2011) Aid and growth: What meta-analysis reveals, UNU-
WIDER Working Paper No. 2011/22.
Melo, P.C., Graham, D.J. and Noland, R.B. (2009) A meta-analysis of estimates of urban
agglomeration economies, Regional Science and Urban Economics, 39: 332–42.
Miller, T.R. (2000) Variations between countries in values of statistical life, Journal of
Transport Economics and Policy, 34: 169–88.
Mishel, L. and Rothstein, R. eds. (2002) The Class Size Debate, Washington: Economic
Policy Institute.
Mookerjee, R. (2006) A meta-analysis of the export growth hypothesis, Economics Letters
91: 395–401.
Moreno, S.G., Sutton, A.J., Ades, A., Stanley, T.D., Abrams, K.R., Peters, J.L. and Cooper,
N.J. (2009a) Assessment of regression-based methods to adjust for publication bias
through a comprehensive simulation study, BMC Medical Research Methodology, 9:2,
http://www.biomedcentral.com/1471–2288/9/2.
Moreno, S.G., Sutton, A.J., Turner E.H., Abrams, K.R., Cooper, N.J, Palmer, T.M. and Ades,
A.E. (2009b) Novel methods to deal with publication biases: Secondary analysis of
antidepressant trials in the FDA trial registry database and related journal publications,
British Medical Journal, 339: 494–98.
Mrozek, J.R. and Taylor, L.O. (2002) What determines the value of life? A meta-analysis,
Journal of Policy Analysis and Management, 21: 253–70.
Mundlak, Y. (1978) On the pooling of time series and cross section data, Econometrica,
46: 69–85.
Nelson, J.P. (2004) Meta-analysis of airport noise and hedonic property values: Problems
and prospects, Journal of Transport Economics and Policy, 38: 1–28.
Nelson, J.P. (2011) Alcohol marketing, adolescent drinking, and publication bias in
longitudinal studies: A critical appraisal using meta-analysis, Journal of Economic
Surveys, 25: 191–232.
Nelson, J.P. and Kennedy, P.E. (2009) The use (and abuse) of meta-analysis in
environmental and natural resource economics: An assessment, Environmental and
Resource Economics, 42: 345–77.
Nijkamp, P. and Poot, J. (2004) Meta-analysis of the effect of fiscal policies on long-run
growth, European Journal of Political Economy, 20: 91–124.
Nijkamp, P. and Poot, J. (2005) The last word on the wage curve? A meta-analytic
assessment, Journal of Economic Surveys, 19(3): 421–50.
Oaxaca, R. (1973) Male-female wage differentials in urban labor markets, International
Economic Review, 14: 693–709.
Oosterbeek, H., Sloof, R. and van de Kuilen, G. (2004) Cultural differences in ultimatum
game experiments: Evidence from a meta-analysis, Experimental Economics, 7: 171–88.
Papke, L.E. and Wooldridge, J.M. (2005) A computational trick for delta-method standard
errors, Economics Letters, 86: 413–17.
176 References
Pavlides, M.G. and Perlman, M.D. (2009) How likely Is Simpson’s paradox? American
Statistician, 63: 226–33.
Pearson, K. (1904) Report on certain enteric fever inoculation statistics, British Medical
Journal, 2: 1243–46.
Peloza, J. and Steel, P. (2005) The price elasticities of charitable contributions: A meta-
analysis, Journal of Public Policy and Marketing, 24: 260–72.
Popper, K.R. (1959) Logic of Scientific Discovery, London: Hutchinson.
Popper, K.R. (1963) Conjectures and Refutations: The Growth of Scientific Knowledge,
New York: Basic Books.
Poteete, A.R. and Ostrom, E. (2008) Fifteen years of empirical research on collective
action in natural resource management: Struggling to build large-n databases based on
qualitative research, World Development, 36: 176–95.
Public Broadcasting Service. (2009) The warning, Frontline, October 20.
Ridhwan, M.M., De Groot, H.L.F., Nijkamp, P. and Rietveld, P. (2010) The impact of
monetary policy on economic activity: Evidence from a meta-analysis. Tinbergen
Institute Discussion Paper 10–043/3.
Riley, R.D. (2009) Multivariate meta-analysis: The effect of ignoring within-study
correlation, Journal of the Royal Statistical Society, Series A, 172: 789–811.
Rose, A.K. and Stanley, T.D. (2005) A meta-analysis of the effect of common currencies on
international trade, Journal of Economic Surveys, 19: 347–65.
Rosenberger, R.S. and Johnston, R.J. (2009) Selection Effects in Meta-Analysis and Benefit
Transfer: Avoiding unintended consequences, Land Economics, 85: 410–28.
Rosenberger, R.S. and Loomis, J.B. (2000a) Using meta-analysis for benefit transfer:
In-sample convergent validity tests of an outdoor recreation database, Water Resources
Research, 36: 1097–107.
Rosenberger, R.S. and Loomis, J.B. (2000b) Panel stratification in meta-analysis of
economic studies: an investigation of its effects in the recreation valuation literature,
Journal of Agricultural and Applied Economics, 32: 459–70.
Rosenberger R.S. and Stanley T.D. (2006) Measurement, generalization, and publication:
Sources of error in benefit transfers and their management, Ecological Economics, 60:
372–78.
Rosenthal, R. (1979) The “file drawer problem” and tolerance for null results, Psychological
Bulletin, 86: 638–41.
Rücker, G., Schwarzer, G., Carpenter, J.R., Binder, H. and Schumacher, M. (2011)
Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis,
Biostatistics, 12: 122–42.
Rusnák, M., Havránek, T. and Horváth, R. (2011) How to Solve the Price Puzzle? A Meta-
Analysis. CERGE-EI Working Paper No. 446. Available at SSRN: http://ssrn.com/
abstract=1942999
Sala-i-Martin, X.X. (1997) I just ran two million regressions, American Economic Review,
87: 178–83.
Sally, D. (1995) Conversation and cooperation in social dilemmas: A meta-analysis of
experiments from 1958 to 1992, Rationality and Society, 7: 58–92.
Sandy, R. and Elliott, R.F. (1996) Unions and risk: their impact on the compensation for
fatal risk, Economica, 63: 291–309.
Scargle, J.D. (2000) Publication bias: The “file drawer” problem in scientific inference,
Journal of Scientific Exploration, 14: 91–106.
Schmidt, P. (1977) Estimation of seemingly unrelated regressions with unequal numbers of
observations, Journal of Econometrics, 5: 365–77.
References 177
Schulze, R. (2004) Meta-Analysis: A comparison of approaches, Göttingen: Hogrefe and
Huber.
Sethuraman, R, Tellis, G.J., and Briesch, R.A. (2011) How well does advertising work?
Generalizations from meta-analysis of brand advertising elasticities, Journal of
Marketing Research, 48: 457–71.
Shen, Y-S., Eggelston, K., Lau, J. and Schmid, C. (2005) Hospital ownership and financial
performance: A quantitative research review. NBER Working Paper No. 11662.
Shinew, K.J., Floyd, M.F. and Parry, D. (2004) Understanding the relationship between
race and leisure activities and constraints: Exploring an alternative framework, Leisure
Sciences, 26: 181–199.
Shrestha, R.K. and Loomis, J.B. (2001) Testing a meta-analysis model for benefit transfer
in international outdoor recreation, Ecological Economics, 39: 67–83.
Sidik, K. and Jonkman, J.N. (2007) A comparison of heterogeneity variance estimators in
combining results of studies, Statistics in Medicine, 26: 1964–81.
Simons, R.A. and Saginor, J.D. (2006) A meta-analysis of the effect of environmental
contamination and positive amenities on residential real estate values, Journal of Real
Estate Research, 28: 71–104.
Smith, M.L. and Glass, G.V. (1977) Meta-analysis of psychotherapy outcome studies,
American Psychologist, 32: 752–60.
Smith, V.K. and Huang, J-C. (1995) Can markets value air quality? A meta-analysis of
hedonic property value models, Journal of Political Economy, 103: 209–27.
Smith, V.K. and Kaoru, Y. (1990a) What have we learned since Hotelling’s letter? A meta-
analysis, Economics Letters, 32: 267–72.
Smith, V.K. and Kaoru, Y. (1990b) Signals or noise? Explaining the variation in recreation
benefit estimates, American Journal of Agricultural Economics, 72: 419-33.
Smith V.K. and Pattanayak, S.K. (2002) Is meta-analysis a Noah’s Ark for non-market
valuation? Environmental and Resource Economics, 22: 271–96.
Stanley, T.D. (1998). New wine in old bottles: A meta-analysis of Ricardian equivalence,
Southern Economic Journal, 64: 713–27.
Stanley, T.D. (2001) Wheat from chaff: Meta-analysis as quantitative literature review,
Journal of Economic Perspectives, 15: 131–50.
Stanley, T.D. (2002) When all are NAIRU: Hysteresis and behavioral inertia, Applied
Economic Letters, 9: 753–57.
Stanley, T.D. (2004) Does unemployment hysteresis falsify the natural rate hypothesis? A
meta-regression analysis, Journal of Economic Surveys, 18: 589–612.
Stanley, T.D. (2005a) Beyond publication bias, Journal of Economic Surveys, 19: 309–45.
Stanley, T.D. (2005b) Integrating the empirical tests of the natural rate hypothesis: A meta-
regression analysis, Kyklos, 58: 611–34.
Stanley, T.D. (2008) Meta-regression methods for detecting and estimating empirical effect
in the presence of publication bias, Oxford Bulletin of Economics and Statistics, 70:
103–27.
Stanley, T.D. and Doucouliagos, C. (H.) (2007) Identifying and correcting publication
selection bias in the efficiency-wage literature: Heckman meta-regression. School
Working Paper, Economics Series 2007–11, Deakin University.
Stanley, T.D. and Doucouliagos, C. (H.) (2010) Picture this: A simple graph that reveals
much ado about research, Journal of Economic Surveys, 24: 170–91.
Stanley, T.D. and Doucouliagos, C. (H.) (2011) Meta-regression approximations to reduce
publication selection bias. School Working Paper, Economics Series 2011–4, Deakin
University.
178 References
Stanley, T.D. and Jarrell, S.B. (1989) Meta-regression analysis: A quantitative method of
literature surveys, Journal of Economic Surveys, 3: 161–70.
Stanley, T.D. and Jarrell, S.B. (1998) Gender wage discrimination bias? A meta-regression
analysis, Journal of Human Resources, 33: 947–73.
Stanley, T.D. and Rosenberger, R.S. (2009) Are recreation values systematically
underestimated? Reducing publication selection bias for benefit transfer, Bulletin of
Economics and Meta-Analysis. http://www.hendrix.edu/maer-network/default.aspx?
id=15206
Stanley, T.D., Doucouliagos, C. (H.) and Jarrell, S.B. (2008) Meta-regression analysis as
the socio-economics of economics research, Journal of Socio-Economics, 37: 276–92.
Stanley, T.D., Jarrell, S.B. and Doucouliagos, C. (H.) (2010) Could it be better to discard
90% of the data? A statistical paradox, American Statistician, 64: 70–7.
Sterling T.D. (1959) Publication decisions and their possible effects on inferences drawn
from tests of significance or vice versa, Journal of the American Statistical Association,
54: 30–4.
Sterling, T.D., Rosenbaum, W.L. and Weinkam, J.J. (1995) Publication decisions revisited:
The effect of the outcome of statistical tests on the decision to publish and vice versa,
American Statistician, 49: 108–12.
Sterne, J.A. and Egger, M. (2001) Funnel plots for detecting bias in meta-analysis:
Guidelines on choice of axis, Journal of Clinical Epidemiology, 54: 1046–55.
Stroup, D.F., Berlin, J.A., Morton, S.C., Olkin, I., Williamson, G.D., Rennie, D., Moher,
D., Becker, B.J., Sipe, T.A. and Thacker, S.B. (2000) Meta-analysis of observational
studies in epidemiology: A proposal for reporting, Journal of American Medical
Association, 283: 2008–12.
Sutton, A.J and Higgins, J.P.T. (2007) Recent developments in meta-analysis, Statistics in
Medicine, 27: 625–50.
Sutton, A.J., Abrams, K.R., Jones, D.R., Sheldon, T.A. and Song, F. (2000) Methods for
Meta-analysis in Medical Research, Chichester: Wiley.
Tellis, G.J. (1988) The price elasticity of selective demand: A meta-analysis of econometric
models of sales, Journal of Marketing Research, 25: 331–41.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing:
Three suggested reforms, Educational Researcher, 25: 26–30.
Thompson, B. (2004) The “significance crisis” in psychology and education, Journal of
Socio-Economics, 33: 607–13.
Todorovic, Z.W. and Ma, J. (2008) A review of minimum wage regulation effect: The
resource-based view perspective, Journal of Collective Negotiations, 32: 57–75.
Tosi, H.L., Werner, S., Katz, J.P. and Gomez-Mejia, L.R. (2000) How much does
performance matter? Journal of Management, 26: 301–39.
Tullock, G. (1959) Publication decisions and tests of significance – A comment, Journal of
the American Statistical Association, 54: 593.
United State Environmental Protection Agency (2010) Valuing Mortality Risk Reductions
for Environmental Policy: A white paper. Science Advisory Board–Environmental
Economics Review Draft.
Valentine, T.J. (1979) Hypothesis tests and confidence intervals for mean elasticities
calculated from linear regression equations, Economics Letters, 4: 363–67.
Van Houwelingen, H.C., Arends, L.R. and Stijnen, T. (2002) Advanced methods in
meta-analysis: Multivariate approach and meta-regression, Statistics in Medicine, 21:
589–624.
References 179
Verlegh, P.W.J. and Steenkamp, J.E.B.M. (1999) A review and meta-analysis of country-
of-origin research, Journal of Economic Psychology, 20: 521–46.
Viscusi, W.K. (1993) The value of risks to life and health, Journal of Economic Literature,
31: 1912–46.
Viscusi, W.K. and Aldy, J.E. (2003) The value of a statistical life: A critical review of
market estimates throughout the world, Journal of Risk and Uncertainty, 27: 5–76.
Vista, A.B. and Rosenberger, R. 2009. Primary study aggregation effects: Meta-analysis of
sportfishing values in North America. Paper presented at the Oregon State University
MAER Network Colloquium, October 1–4 2009, Corvallis.
Wagenaar, A.C., Salois, M.J. and Komro, K.A. (2009) Effects of beverage alcohol price and
tax levels on drinking: A meta-analysis of 1003 estimates from 112 studies, Addiction,
104: 179–190.
Waldorf, B. and Byun, P. (2005) Meta-analysis of the impact of age structure on fertility,
Journal of Population Economics, 18: 15–40.
Weichselbaumer, D. and Winter-Ebmer, R. (2005) A meta-analysis of the international
gender wage gap, Journal of Economic Surveys, 19: 479–511.
Welkowitz, J., Ewen, R.B. and Cohen, J. (1982) Introductory Statistics for the Behavioral
Sciences, San Diego, CA: Harcourt Brace Jovanovich.
Whitehead, A. (2002) Meta-Analysis of Controlled Clinical Trials, Chichester: Wiley.
Wooldridge, J.M. (2002) Econometric Analysis of Cross Section and Panel Data,
Cambridge: MIT Press.
Wooldridge, J.M. (2006) Introductory Econometrics: A Modern Approach, Cincinnati:
South-Western.
Wooster, R.B. and Diebel, D.S. (2010) Productivity spillovers from foreign direct
investment in developing countries: A meta-regression analysis, Review of Development
Economics, 14: 640–55.
Young, N.S., Ioannidis, J.P.A. and Al-Ubaydli, O. (2008) Why current publication practices
may distort science, PLoS Med, 5: doi:10.1371/journal.pmed.0050201.
Zelmer, J. (2003) Linear public goods experiments: A meta-analysis, Experimental
Economics, 6: 299–310.
Ziliak, S.T. and McCloskey, D.N. (2004) Size matters: The standard error of regressions in
the American Economic Review, The Journal of Socio-Economics, 33: 527–46.
Index
3SLS see three stage least squares 71, 73, 76–7, 81, 84–6, 93, 102, 109–10,
112, 115–16, 121, 126–31, 136, 147,
Abreu, M. 13, 75, 144, 159n 152, 160n, 165n; publication (selection)
Adams, D. 158n 4, 9, 11, 16–9, 27, 29, 32, 39–42, 44, 47,
Akerlof, G. 99 50–79, 82–6, 90–106, 108–10, 117–23,
Albers, S. 26 125–7, 129, 135–6, 139–41, 144–5,
alchohol 18, 126–7, 136–40 147–51, 154–67n
Aldy, J. 158n, 167n Bijmolt, T. 26, 38
Alston, J. 19 binary dependent variable 16–17
Andrews, E. 4 Bland, J. 51
AR(1) 57, 163n Blaug, M. 152
Arcot, S. 154n Boockmann, B. 165n
Arnould, R. 162n Bolotova, Y. 29
Ashenfelter, O. 75 Borenstein, M. 22, 49
Ayers, I. 2 Bowland, B. 158n
Brander, L. 29, 38, 126
Babel 1–5 Breusch–Pagan test 103–5, 139, 163n
Ballal, S. 166n Brons, M. 140, 165n, 167n
Baltagi, B. 166n Brown, S. 24
Banzhaf, S. 155n Burkauser, R. 99
Bateman, I. 69, 100, 126 Business Source Premier 9, 162n
Becker, B. 23, 164n, 166n Byun, P. 155n, 156n, 157n
Begg, C. 51, 74
Beghin, J. 158n Callot, L. 152
beliefs 2, 53, 60 Cameron 4
Bellavance, F. 9, 10, 38, 57, 86, 89–90, 94, Campbell Collaboration 14
126, 156–9n, 162n Capelle-Blancard, G. 40
benefit transfer 22, 25, 126–7, 131–4, Card, D. 16, 51, 55, 57, 59–60, 76–7, 85,
156n, 166n 94, 99, 127, 133, 155n, 159n
Bergstrom, J. 126, 134 Chalmers, T. 2
Berlin, J. 51, 74 Charemza, W. 91
Berwick, D.M. 2 Clarke, M. 40
best practice 35, 93, 98–9, 129, 152, 162–3n cluster-robust 33, 37, 70–2, 97, 100,
bias 2–4, 10, 13–9, 21, 24, 31, 33–4, 45, 103–5, 112, 114, 150–1, 164–5n
48, 81–3, 86, 93, 95, 100, 108–10, Cochrane Collaboration 14
112, 114–21, 126–7, 135, 144, 147, Cochrane Reviews 52
152, 154–6n, 158n, 160n, 163–5n; Cochran’s Q-test 49, 81–2
misspecification (omitted-variable) 3, Cohen, J. 5, 7, 22, 48, 157n
6–7, 13, 16, 21, 27, 36, 38, 45, 53, 67–8, Colegrave, A. 29
Index 181
compensation insurance, 86–7, 89–90, EconLit 7, 13, 162n
92–4, 102, 162n Econometrics: as the subject of meta-
competition 154n regression analysis 2–10, 12–37
Connor, J. 29 (chapter 2); see also conventional
conventional 7, 10, 21, 34, 43, 52, 60, 62–3, econometrics
71, 88, 99, 106, 165; econometrics 4, 11, Econometrics methods see Heckman
37, 68, 72, 75, 81, 84, 88, 92–3, 106–15, correction, seeming unrelated
118–19, 122–3, 141, 161–3n, 165n; regressions (SUR), three stage least
narrative reviews 2, 9, 21, 36, 52, 88, 141, squares (3SLS), weighted least
154n; practice (meta-analysis) 33, 41, 47, squares (WLS)
49, 82, 89, 106, 141, 158n, 161n, 164n economic growth 14, 42, 129, 137–8, 144
convergence 75, 128, 144 economic significance see significance:
Cooper, H. 46, 49, 82 practical
Copas, J. 51 Ederveen, S. 38
Coric, B. 157n Efendic, A. 144
corporate performance 5 effect size 6–7, 12, 15–16, 20–33, 38,
correlation 5, 16, 23–5, 41, 83–4, 103–4, 41–3, 46–50, 122, 125, 127, 131–2,
114–15, 121, 136, 138, 150, 156–7n, 135–41, 143–6, 148, 151, 154n,
160n, 165n; auto- 36, 68; partial 3, 156–8n, 166n
23–9, 31, 39–40, 42, 53–4, 57, 102, 132, efficiency: economic 29; statistical 71–2,
144–5, 156–8n; zero-order 24 137–8, 164n, 166n
cross-sectional 20–1, 23, 82, 87, 95, 113, efficiency wage 18, 21, 99, 128, 131
156n Égert, B. 18
Egger, M. 61, 64, 117, 158n, 165n
Dalhuisen, J. 9, 26, 38, 55 elasticity 9–10, 16, 23–4, 26–9, 31, 35,
data dependence 32–3, 36–7, 68–9, 71, 3–9, 44, 46–9, 55–6, 58–60, 62–4, 67,
99–101, 112–13 69, 75, 80, 82–3, 87, 94–102, 109–10,
data mining 91, 104, 131 126–8, 132, 136–40, 142–3, 145,
Davidson, J. 107 156–7n, 159n, 162–3n, 167n; delta
Davidson, R. 84, 107, 108,111, 118, 123 method 27, 157n
Day, B. 127, 158n Elliott, R. 93
Deadman, D. 91 employment 5, 10, 26, 30, 47–8, 52,
debate 68, 73, 94–5, 150, 154n 59–60, 62–3, 67, 70–1, 77, 80, 86–9,
De Blaeij, A. 158n 94–103, 113, 122, 127, 133, 140,
De Dominicis, L. 29, 144 158–9n, 162–3n
de Groot, H. 13, 75, 76, 144, 159n environmental economics 13, 25, 29,
de Haan, J. 19, 35, 156n, 157n 125–6, 134, 151; planning 9, 56, 67;
De Long, J. 51 policy 10, 145; values 3, 57, 73, 126–7,
De Mooij, R. 38 133, 158n
Demsetz, H. 2, 60 evidence-based practice 2–3
Dekker, T. 158n exaggeration 1, 52, 71, 92
Department for International Development explanatory variables 15, 18, 23, 27–8, 35,
154n 80, 82, 86, 90–1, 103–4, 106–7, 110,
Diebel, D. 166n 114–15, 118, 130–3, 136–8, 140, 142–4,
Dionne, G. 57, 158n 161n
discrimination 42–3, 86, 117, 134
Disdier, A. 35, 156n failsafe N 73–4
Djankov, S. 156n FAT-PET-MRA 69, 73, 76, 78–9, 84–5, 90,
Doucouliagos, C (KH) 2, 4–5, 9–11, 19–20, 101, 103–5, 117–22, 127, 138–9, 149,
22, 25–6, 32–4, 38, 40, 42–3, 48, 52–3, 158n, 161–2n, 165n; see also funnel-
57, 59, 61, 63, 65–8, 76–7, 86, 88–90, asymmetry test (FAT), precision-effect
92–9, 119–20, 127, 129, 133, 141, 144–5, test (PET)
149–50, 154–60n, 162n, 165–6n Federal Reserve 4
Duval, S. 74, 171 Feige, E. 51
182 Index
Feld, L. 28, 103 Heckman correction 75, 84, 88, 117–19,
Fidrmuc, J. 19, 24 159n, 165n
file drawer problem 73 Heckman, J. 1, 4, 117–19, 147, 165n,
Fischer, C. 29 172–3
Fisher, R.A. 5, 45 Hedges, L. 45–6, 49, 51, 63, 75–6, 82,
Fisher test 5–6, 45 111–12, 124, 154n, 160–1n, 169–70,
Fisher z-transform 25, 156n 173–4
fixed-effects (weighted averages) (FEE) hedonic 10, 57, 59, 86–7, 89, 91–2, 102
43, 46–50, 64, 69–71, 81, 149, 152, Hennessy, P. 4
158n; meta-regression 83–4, 101, heteroskedasticity 28, 61, 62, 67, 69, 81,
103–4, 111–16, 122–3, 152, 160–1n, 85, 108–10, 112, 122, 151, 163–4n
163–5n Higgins, J. 14, 49, 82
Florax, R. 68, 75, 162n Hirschberg, J. 157n
foreign aid 4, 42, 52, 133, 144, 150 Holmgren, J. 38
Fortin, B. 162n Hopewell, S. 52
funnel-asymmetry test (FAT) 61–2, 64–5, Huang, J. 38, 75–6
78–9, 94, 120, 149 Hunt, M. 6–7
funnel graph 40–2, 53–60 Hunter, J. 22, 24, 32, 34, 44, 47, 72, 139,
156n, 158n
Galbraith, R. 38, 117
Gallet, C. 20, 35, 156n, 159n Iamsiraroj, S. 144, 158n
García-Quevedo, J. 16 ideology 1, 4, 51, 113, 116, 147, 154n
Gauss–Markov theorem 123 impact factor 22, 34, 46, 157n
generalized least squares (GLS) 100, independently and identically distributed
111–12, 114, 123, 164n, 166n (i.i.d.) 61, 68, 106–7, 110
general-to-specific (G-to-S) 90–2, 96, information technology 1, 155n
104–5 intervention 7, 26, 48, 147
Genesis 1 inverse Mills ratio 84, 118, 120, 159n
Gerber, A. 52 Ioannidis, J. 157n
Giles, M. 29 Iyengar, S. 74
Glass, G. 6–7, 22, 51, 154n
Glass’s g 6–7, 22, 154n Jacobsen, J. 117
Görg, H. 156n Jarrell, S. 3–4, 7, 21–3, 28, 30, 36, 42, 68,
Great Depression 4 86, 88–9, 100, 117, 134, 156n, 162n,
Great Recession 154n 167n
Green, D. 52 Jensen, P. 166n
Green, S. 14 Johnson, H. 60
Green, W. 111, 118, 157n, 159n, 165n Johnston, N. 118
Greenberg, D. 31 Johnston, R. 22, 126, 134, 144, 166n
Greene, W. 157n, 159n Jones, A. 69, 100, 126
Greenhouse, J. 74, 173 Judge, G. 123
Greenspan, A. 4, 168, 172
Gresham’s law 51 Kaitz index 87–8, 95, 97–9, 102
Griliches, Z. 115 Kaoru, Y. 29
Gujarati, D. 157n Kennedy, P. 13, 151
Klomp, J. 19, 35, 156–7n
Halpern, L. 18 Kluve, J. 16, 20, 158n
Hands, D. 151 Knell, M. 26
Harlow, L. 73 Kochi, I. 158n
Hausman test 71, 103, 105, 115 Koetse, M. 112, 152, 155n
Havránek, T. 30, 40 Konstantopoulos, S. 111, 161n
health 2, 83; mental 6–7; policy 9, 127 Korhonen, I. 19, 24
Heckemeyer, J. 28, 103 Kotchen, M. 139
Index 183
Kotz, S. 118 Meta-Significance (MST) 76–8, 149, 161n
Krakovsky, M. 52 methodology 2, 141, 147, 150, 152, 167n
Krassoi-Peach, E. 18, 21, 88, 99, 128, 131 Michaud, P. 57, 158n
Krueger, A. 32, 51, 55, 57, 59–60, 76–7, Miller, T. 158n
85, 94, 99, 127, 133, 159n minimum wage 5, 9–10, 19, 26, 38–9,
K-variables 85, 90–1, 94, 96–8, 101, 104, 43–4, 47–8, 52, 59–60, 61–4, 67, 69–72,
109, 138, 150, 163n 77, 80, 82, 86–9, 94–103, 113, 122, 127,
133, 145, 155–6n, 158–9n, 162–3n,
Lagrange multiplier test see Breusch– 165n
Pagan test Mishel, L. 72
Laird, N. 52 misspecification see bias
Lakatos, I. 151 mixed-effects 101, 112, 161n, 164n
Lang, K. 51 moderator variables 15–16, 24, 28, 49, 82,
Laroche, P. 9, 22, 25–6, 33, 38, 89, 159n 86–98, 101–4, 107, 109–10, 114–16,
Leamer, E. 3 134, 141–2, 150, 162n, 164–5n
Lebeau, M. 57 Monte Carlo 103, 112; see also
Leonard, H. 3 simulations
Leonhardt, D. 4 Mookerjee, R. 144, 156n, 157n
Lewis, H. 21 Moreno, S. 119–20, 149, 165n
Lewis, S. 40 Morgenstern, R. 29
Lindhjem, H. 126, 134, 158n Mosteller, F. 52
Lipsey, M. 22, 166n Mrozek, J. 158n
List, J. 35, 159n, 165n multicollinearity 91, 131, 161n
Liu, J. 129, 158n multidimensional 12, 69, 93, 100, 113,
logit 16–17, 44 141, 154n, 164n
Longhi, S. 140 multilevel 37, 49, 69–71, 82, 100–1,
Loomis, J. 17, 29, 69, 100, 113, 126, 134 103–4, 106, 113, 122, 150, 160–2n,
Lovell, M. 51 164n; fixed-effects (FEML) 69–71, 82,
97, 101, 103–5; random-effects (REML)
McCloskey, D. 48, 73 69–71, 82, 97, 101, 103–5
MacKinnon, J. 84, 107, 108, 111, 118, 123 multiple MRA 11, 16, 18, 20, 28, 35, 43,
Ma, J. 165n 48, 80–105 (Chapter 5)
Malhorta, N. 52 multiple regression 3, 7, 25
maximum likelihood 68, 70, 75–6, 160n multivariate analysis 3, 7, 50, 65, 75,
Mayo, D. 167n 79–81, 84–5, 89, 96, 98, 104, 120, 122,
medical research 2, 7, 14, 16, 36, 51–2, 126, 136–7, 154–5n, 161–4n, 166n
74–5, 83–4, 103, 119, 161n, 166n Mundlak, Y. 103
Melo, P. 26 Murrell, P. 156n
mental health see health
meta-analysis: definition 2 natural rate hypothesis 128, 165n
meta-analysis: methods see fixed-effects Navrud, S. 126, 134
(weighted averages) (FEE), random- Nelson, J. 13, 18, 151
effects (weighted averages) (FEE), Nichols, L. 162n
trim-and-fill Nickerson, D. 52
meta-meta-analysis (M2RA) 140–145, Nijkamp, P. 75, 158n
154–5n Nobel prize 1
meta-regression analysis: definition 3 non-English studies 15–16
meta-regression analysis: methods nonstationarity 109
see funnel-asymmetry test (FAT),
K-variables, meta-meta-analysis Oaxaca, R. 117
(M2RA), precision-effect estimate with Olkin, I. 45–6, 51, 63, 75, 111–12, 124,
standard errors (PEESE), precision- 154n, 161n
effect test (PET), Z-variables omitted-variable bias see bias
184 Index
Oosterbeek, H. 29 Rand, A. 4
ordinary least squares (OLS) 16, 21, 48, random-effects (weighted averages) (FEE)
60–4, 68, 71, 85, 91, 110–11, 113–16, 43, 46–8, 50, 64, 69–70, 74, 149, 152,
122–4, 138–41, 160n, 164n 158; meta-regression 82–84, 101,
Ostrom, E. 17 103–4, 111–16, 122–3, 152, 160–1n,
163–4n
Paldam, M. 22, 42–3, 52, 89, 133, 144, randomized clinical trials (RCT) 2, 52, 120
150, 152, 156n, 158n, 162n random sampling 13, 47, 56, 60, 80, 108,
panel 13, 20–1, 23, 33, 37, 69–71, 80, 82, 133, 141
87–8, 95–100, 102–4, 106, 112–17 regulation 4–5, 9–10, 83, 136, 154n
Papke, L. 157n reviewers 4, 32, 52–3, 99, 104
paradox 19, 24, 56, 120 reviews: conventional (narrative) 2–3,
parameter 1, 10, 38–9, 44, 48, 57, 60, 66, 13–14, 21, 36–7, 52, 63, 88, 141, 143,
74–6, 80, 109, 111, 125–9, 134, 147 147, 151; systematic 2–5, 11, 14, 33, 52,
partial correlations see correlation 63, 134, 154n, 167n
Pattanayak, S. 28, 126 Ridhwan, M. 30
Pavlides, M. 24 Riley, R. 166n
Pearson, K. 5, 45 risk and wages 10, 56, 63, 86–7, 89–90,
Peloza, J. 158n 127, 142–3, 155, 162
policy 1–5, 7, 9–10, 12, 15, 26, 30, 38, 44, robustness 15, 25, 32–3, 72, 93–4, 99–101,
48, 50–1, 64–5, 67, 73, 79, 83, 94, 126, 103, 105, 113, 122, 130, 150, 158n,
129, 134, 145, 147–8, 150, 152, 154n, 163n
167n Rose, A. 24, 28, 159n
political science 52 Rosenberger 29, 57, 69, 73, 100, 113,
politics 51, 63 126–7, 134, 144
Poot, J. 75, 158n, 162n Rosenthal, R. 51, 73–4
Popper, K. 37, 99, 128, 151 Rothstein, R. 72
Poteete, A. 17 Rubenstein, R. 166n
precision 3, 18–19, 27, 34–5, 40–2, 46, 49, Rücker, G. 149
53–4, 56–73, 76, 78, 81–2, 85–6, 90, Rusnák, M. 30
108–11, 115–16, 127, 139, 142, 148,
149, 156–60n, 162–4n safety 9, 56, 67
precision-effect estimate with standard Sala-i-Martin 3, 74, 90, 98, 102
errors (PEESE) 65–7, 75, 78–9, 83, 85, Sally, D. 29
92–4, 101, 103–5, 119–22, 127, 149, Sandy, R. 93
159–61n, 163n, 165n Scargle, J. 74
precision-effect test (PET) 62–7, 75, 78–9, Schaffner, S. 20, 158n
93, 120, 149, 160n Schmidt, F. 22, 24, 32, 34, 44, 47, 72, 139,
probit 16–17, 44, 84, 155n, 159n 156n, 158n
psychology 7, 36, 73–4, 152 Schulte, S. 139
psychotherapy 6–7 Schulze, R. 156n, 158n
Public Broadcasting Service 4 science 63, 99, 151, 161n, 167n
publication bias see bias scientific knowledge
publication selection see bias scrutiny
Pugh, G. 157 seemingly unrelated regressions (SUR)
p-values 5, 30–1, 43, 45–6, 50, 75–6, 107, 137–140
157n, 160n selection see bias
self-fulfilling 2
quality 7, 10–1, 17–18, 20, 30, 33–5, 46, Sethuraman, R. 157n
69, 76, 85, 96, 106, 113–16, 134, 138, Shen, Y. 158n
141, 144, 148, 160n, 163–5n Shinew, K. 14
Shrestha, R. 126, 134
Index 185
Sidik, K. 49 theory (economic) 2, 12–3, 21, 34, 51, 60,
Significance: practical 6–7, 10,18–22, 81, 99, 102, 116, 127–9, 131, 137, 145,
25–6, 35, 37, 46, 48, 55, 60, 64–5, 67, 150, 152, 154–5n, 163n; competition
69, 71, 73, 83, 91, 96, 99–100, 102–4, 154n; econometric 11, 61, 106–24
122, 128, 135, 144–5, 150, 155n, 157n, (chapter 6), 160n, 163–5n; meta-
159–60n, 162n; statistical 5, 16, 21, 26, regression 81, 106–12, 164n
31, 35, 42–5, 48, 52, 60, 63–4, 66, 69, Thompson, B. 73
71, 73, 75–6, 90–1, 96, 100–1, 123, 135, Thompson, S. 82
150, 157n, 159–60n, 163n, 165n thought experiment 61, 66, 165n
Simons, R. 29 three stage least squares (3SLS) 122, 125,
simulations 47, 56, 61, 65–6, 75–6, 83, 85, 137, 144
103, 112, 119–20, 127, 152, 159n, 161n time series 57 , 82, 87, 96, 113–14, 158n
simultaneous equations see seemingly Todorovic, Z. 165n
unrelated regressions (SUR), three stage Tosi, H. 24
least squares (3SLS) transformation 3, 7, 25, 28–9, 57, 127,
skewed 57, 62–3, 108 131–2, 156n, 159n
Smith, K. 28, 29, 38, 126 trim-and -fill 74–5, 119
Smith, M. 7 Tullock, G. 51
specification 98; model 3, 21, 35–6, 60, Tweedie, R. 74
69, 76, 87–8, 95–6, 99, 102–6; tests 72, type I error 64–5, 72, 77–8, 90, 154n
103–5, 115, 117, 123 type II error 154n
standard error (SE) 14–15, 20, 22–31,
34–6, 37, 40–1, 46, 53, 55, 57, 60–2, Ulubasoglu, M. 33, 129, 144, 166n
64–7, 69–73, 78–9, 82–7, 91–3, 100–1, unemployment 21, 128, 165n
103–4, 108, 110–12, 114, 117–21, unions 21–2, 26, 54, 87, 158; productivity
123–4, 129, 135, 138, 143, 149, 151, 9, 36, 38–41, 44, 47, 53, 55, 62–3, 65,
157–61n, 164–5n 67–8, 82, 102, 159n
Stanley, T.D. 2–5, 7, 9–11, 13, 17–24; 26, United Kingdom 4
28, 30, 32–4, 36, 38, 40, 42–3, 47–8, unobserved effect 69, 71, 80, 100–4, 107,
52, 57, 59, 61–8, 72–4, 76–8, 83, 85–6, 112, 114–16, 160n, 164–5n
88–9, 93–5, 99, 108–10, 117, 119–20, unpublished studies 17–20, 73–74, 143
123, 126–9, 131, 133–4, 145, 152, US Congress 4
154–6n, 158–63n, 165n, 167n
STATA 64, 69, 71, 82, 84, 103, 123–4; Valentine, T. 27, 157n
metareg 64, 82, 84, 161n value added 20–1, 88–9, 92
Stayman, D. 24 value of a statistical life (VSL) 9–10, 29,
Steel, P. 158n 38–9, 44, 47, 52, 56–7, 59–65, 67, 83,
Sterling, T. 51–2 86–94, 102, 122, 126–7, 129, 132, 155n,
Sterne, J. 158n 157n, 159n, 162n; income elasticity of
Stix, H. 26 141–3, 155, 157, 159, 162
Strobl, E. 156n van den Brink, H. 75–6
Stroup, D. 14 variance-covariance matrix 109, 111, 114,
structural equations 80, 138, 154n 138, 164n, 166n
study selection criteria 14–20, 37 Verlegh, P. 38
Sutton, A. 22, 38, 49, 53, 82 Vevea, J. 75
symmetry (asymmetry) 25, 54–5, 57–64, Viscusi, K. 10, 158n, 167n
108, 110, 158–9n, 162n Vista, A. 158n
vote-counting 6, 43–45, 154n
Tarp, F. 158n
Taylor, L. 126, 134 Wagenaar, A. 165n, 166n
tax 9, 14, 28, 126–7, 166n Waldorf, B. 155n, 156n, 157n
Tellis, G. 26 water 9, 26, 38–9, 44, 47, 55–6, 60, 62–3, 65,
67, 75, 80, 82–3, 102, 126–7, 141, 147
186 Index
Weichselbaumer, D. 20, 42, 134 Wooldridge, J. 84, 111, 113–14, 118
weighted least squares (WLS) 28–9, 48, Wooster, R. 166n
61–2, 65–6, 69–71, 81–5, 91–4, 96–7, Wu, M. 23, 164n, 166n
99–101, 103–4, 110–12, 114, 117, 119,
122–4, 136, 137, 143, 149–50, 159–67n Young, J. 157n
Welkowitz, J. 48
White, D. 17 Zelmer, J. 18, 19, 29
Whitehead, A. 22 Ziliak, S. 73
Wilson, D. 22, 166n Z-variables 85, 90–2, 94, 96–8, 101, 104,
Winter-Ebmer, R. 20, 42, 134 109, 138, 150, 162–3n