Academy of Management The Academy of Management Journal
Academy of Management The Academy of Management Journal
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms
Academy of Management is collaborating with JSTOR to digitize, preserve and extend access
to The Academy of Management Journal
This content downloaded from 43.231.60.18 on Fri, 06 Dec 2019 10:03:11 UTC
All use subject to https://about.jstor.org/terms
® Academy of Management Journal
2011, Vol. 54, No. 4, 657-660.
This editorial continues a seven-part series, "Publishing in AMJ," in which the editors give suggestions and advice
improving the quality of submissions to the Journal. The series offers "bumper to bumper" coverage, with installmen
ranging from topic choice to crafting a Discussion section. The series will continue in October with "Part 3: Setting th
Hook."- J.A.C.
Most scholars, as part of their doctoral education, and macro research. Rejection does not happen be
take a research methodology course in which they cause such data are inherently flawed or because
learn the basics of good research design, including reviewers or editors are biased against such data. It
that design should be driven by the questions being happens because many (perhaps most) research ques
asked and that threats to validity should be avoided. tions in management implicitly—even if not framed
For this reason, there is little novelty in our discus as such—address issues of change. The problem with
sion of research design. Rather, we focus on common cross-sectional data is that they are mismatched with
design issues that lead to rejected manuscripts atresearch questions that implicitly or explicitly deal
AMJ. The practical problem confronting researchers with causality or change, strong tests of which require
as they design studies is that (a) there are no hard andeither measurement of some variable more than once,
fast rules to apply; matching research design to re or manipulation of one variable that is subsequently
search questions is as much art as science; and (b)linked to another. For example, research addressing
external factors sometimes constrain researchers'
such topics as the effects of changes in organizationa
ability to carry out optimal designs (McGrath, 1981). leadership on a firm's investment patterns, the effects
Access to organizations, the people in them, andof CEO or TMT stock options on a firm's actions, or
rich data about them present a significant challenge the effects of changes in industry structure on behav
for management scholars, but if such constraints ior implicitly addresses causality and change. Sim
become the central driver of design decisions, the larly, when researchers posit that managerial behav
outcome is a manuscript with many plausible ior al affects employee motivation, that HR practices
ternative explanations for the results, which leadsreduce turnover, or that gender stereotypes constrain
ultimately to rejection and the waste of consider the advancement of women managers, they are also
able time, effort, and money. Choosing the appro implicitly testing change and thus cannot conduc
priate design is critical to the success of a manu adequate tests with cross-sectional data, regardless of
script at AMJ, in part because the fundamental whether that data was drawn from a pre-existing dat
design of a study cannot be altered during the base re or collected via an employee survey. Researchers
vision process. Decisions made during the researchsimply cannot develop strong causal attributions
design process ultimately impact the degree of con
with cross-sectional data, nor can they establish
fidence readers can place in the conclusions drawn
change, regardless of which analytical tools they use.
from a study, the degree to which the results pro
Instead, longitudinal, panel, or experimental data are
vide a strong test of the researcher's arguments, and
needed to make inferences about change or to estab
the degree to which alternative explanations can be
lish strong causal inferences. For example, Nyberg,
discounted. In reviewing articles that have been
Fulmer, Gerhart, and Carpenter (2010) created a pane
rejected by AMJ during the past year, we identified
set of data and used fixed-effects regression to mode
three broad design problems that were common
the degree to which CEO-shareholder financial align
sources of rejection: (a) mismatch between research
ment influences future shareholder returns. This data
question and design, (b) measurement and opera structure allowed the researchers to control for cross
tional issues (i.e., construct validity), and (c) inap
firm heterogeneity and appropriately model how
propriate or incomplete model specification.
changes in alignment within firms influenced share
holder returns.
Our point is not to denigrate the potential use
Matching Research Question and Design
fulness of cross-sectional data. Rather, we point out
Cross-sectional data. Use of cross-sectional data the importance of carefully matching research design
is a common cause of rejection at AMJ, of both microto research question, so that a study or set of studies
657
Copyright of the Academy of Management, all rights reserved. Contents may not be copied, emailed, posted to a listserv, or otherwise transmitted without the copyright holder's
written permission. Users may print, download or email articles for individual use only.
This content downloaded from 43.231.60.18 on Fri, 06 Dec 2019 10:03:11 UTC
All use subject to https://about.jstor.org/terms
Academy of Management Journal
is capable of testing the question of interest. Research thor developing a new construct must clearly artic
ers should ask themselves during the design stage ulate the definition and boundaries of the new con
whether their underlying question can actually be struct, map its association with existing constructs,
answered with their chosen design. If the question and avoid assumptions that scales with the same
involves change or causal associations between vari name reflect the same construct and that scales with
ables (any mediation study implies causal associa different names reflect different constructs (i.e., jingle
tions), cross-sectional data are a poor choice. jangle fallacies [Block, 1995]). Failure to define the
Inappropriate samples and procedures. Much or core construct often leads to inconsistency in a man
ganizational research, including that published in uscript. For example, in writing a paper, authors may
AMJ, uses convenience samples, simulated business initially focus on one construct, such as organization
situations, or artificial tasks. From a design stand al legitimacy, but later couch the discussion in terms
point, the issue is whether the sample and procedures of a different but related construct, such as reputation
are appropriate for the research question. Asking stu or status. In such cases, reviewers are left without a
dents with limited work experience to participate in clear understanding of the intended construct or its
experimental research in which they make executive theoretical meaning. Although developing theory is
selection decisions may not be an appropriate way to not a specific component of research design, readers
test the effects of gender stereotypes on reactions and reviewers of a manuscript should be able to
to male and female managers. But asking these same clearly understand the conceptual meaning of a con
students to participate in a scenario-based experi struct and see evidence that it has been appropriately
ment in which they select the manager they would measured.
prefer to work for may present a good fit between Inappropriate adaptation of existing measures.
sample and research question. Illustrating this notion A key challenge for researchers who collect field
of matching research question with sample is a study data is getting organizations and managers to com
on the valuation of equity-based pay in which Devers, ply, and survey length is frequently a point of con
Wiseman, and Holmes (2007) used a sample of exec cern. An easy way to reduce survey length is to
utive MBA students, nearly all of whom had experi eliminate items. Problems arise, however, when
ence with contingent pay. The same care used in researchers pick and choose items from existing
choosing a sample needs to be taken in matching scales (or rewrite them to better reflect their unique
procedures to research question. If a study involves context) without providing supporting validity ev
an unfolding scenario wherein a subject makes a se idence. There are several ways to address this prob
ries of decisions over time, responding to feedback lem. First, if a manuscript includes new (or sub
about these decisions, researchers will be well served stantially altered measures), all the items should be
by collecting data over time, rather than having a included in the manuscript, typically in an appen
series of decision and feedback points contained in a dix. This allows reviewers to examine the face va
single 45 minute laboratory session. lidity of the new measures. Second, authors might
Our point is not to suggest that certain samples include both measures (the original and the short
(e.g., executives or students) or procedures are inher ened versions) in a subsample or in an entirely
ently better than others. Indeed, at AMJ we explicitly different sample as a way of demonstrating high
encourage experimental research because it is an ex convergent validity between them. Even better
cellent way to address questions of causality, and we would be including several other key variables in
recognize that important questions—especially those the nomological network, to demonstrate that the
that deal with psychological process—can often be new or altered measure is related to other similar
answered equally well with university students or and dissimilar constructs.
organizational employees (see AM/s August 2008 Inappropriate application of existing mea
From the Editors [vol. 51: 616-620]). What we ask of sures. Another way to raise red flags with review
authors—whether their research occurs in the lab or ers is to use existing measures to assess completely
the field—is that they match their sample and proce different constructs. We see this problem occurring
dures to their research question and clearly make the particularly among users of large databases. For
case in their manuscript for why these sample or example, if prior studies have used an action such
procedures are appropriate. as change in format (e.g., by a restaurant) as a mea
sure of strategic change, and a submitted paper uses
this same action (change in format) as a measure of
Measurement and Operationalization
organizational search, we are left with little confi
Researchers often think of validity once they be dence that the authors have measured their in
gin operationalizing constructs, but this may be too tended construct. Given the cumulative and incre
late. Prior to making operational decisions, an au mental nature of the research process, it is critical
This content downloaded from 43.231.60.18 on Fri, 06 Dec 2019 10:03:11 UTC
All use subject to https://about.jstor.org/terms
2011 Bono and McNamara
This content downloaded from 43.231.60.18 on Fri, 06 Dec 2019 10:03:11 UTC
All use subject to https://about.jstor.org/terms
Academy of Management Journal
experiments (e.g., Grant & Berry, 2011), or by test Kark, R., Shamir, B., & Chen, G. 2003. The two faces of
ing multiple industry data sets to assess the robust
transformational leadership: Empowerment and de
ness of findings (e.g., Beck, Bruderl, & Woywode, pendency. Journal of Applied Psychology, 88: 246—
2008). As noted in AMfs "Information for Contrib 255.
tential for an ultimately positive outcome. Wang, H., Law, K. S., Hackett, R. D., Wang, D., & Chen,
Joyce E. Bono Z. X. 2005. Leader-member exchange as a mediato
University of Florida of the relationship between transformational leader
ship and followers' performance and organizational
Gerry McNamara citizenship behavior. Academy of Management
Michigan State University Journal, 48: 420-432.
This content downloaded from 43.231.60.18 on Fri, 06 Dec 2019 10:03:11 UTC
All use subject to https://about.jstor.org/terms