Professional Documents
Culture Documents
Ideology or Situation Sense - An Experimental Investigation o
Ideology or Situation Sense - An Experimental Investigation o
2016
David Hoffman
Danieli Evans
Neal Devins
William & Mary Law School, [email protected]
Eugene Lucci
See next page for additional authors
Repository Citation
Kahan, Dan M.; Hoffman, David; Evans, Danieli; Devins, Neal; Lucci, Eugene; and Cheng, Katherine, "'"Ideology" or "Situation
Sense"? An Experimental Investigation of Motivated Reasoning and Professional Judgment" (2016). Faculty Publications. 1801.
https://scholarship.law.wm.edu/facpubs/1801
Copyright c 2016 by the authors. This article is brought to you by the William & Mary Law School Scholarship Repository.
https://scholarship.law.wm.edu/facpubs
Authors
Dan M. Kahan, David Hoffman, Danieli Evans, Neal Devins, Eugene Lucci, and Katherine Cheng
This article is available at William & Mary Law School Scholarship Repository: https://scholarship.law.wm.edu/facpubs/1801
ARTICLE
α Elizabeth K. Dollard Professor of Law and Professor of Psychology, Yale Law School.
β Murray H. Shusterman Professor of Transactional and Business Law, Temple University
Beasley School of Law.
χ Senior Fellow, National Constitutional Center and Analyst, Cultural Cognition Lab.
δ Goodrich Professor of Law, Cabell Research Professor, and Professor of Government,
William & Mary Law School.
π Judge, Ohio Court of Common Pleas.
σ Lab Member, Cultural Cognition Project, Yale Law School.
Funding for the study reported in this Article was generously provided by the Oscar M.
Ruebhausen Fund at Yale Law School. We are grateful to Justice Edward Chavez of the New
Mexico Supreme Court; Judge Melanie May of the Florida Fourth District Court of Appeal; and
Judge Marva L. Crenshaw of the Florida Second District Court of Appeal for facilitating judicial
participation in the study. We also wish to express our gratitude to both the National Center for
State Courts and the National Judicial College’s assistance in notifying judges of the online
continuing legal education seminars from which a portion of the judge study sample was recruited;
and in particular to Paula Hannaford-Agor at the National Center for State Courts for consultation,
advice, and encouragement. We are grateful, too, to Meredith Berger and Katie Carpenter for
production assistance.
(349)
350 University of Pennsylvania Law Review [Vol. 164: 349
marijuana legalization and other contested issues. When the study subjects were
assigned to analyze statutory interpretation problems, however, only the responses of
the general-public subjects and not those of the judges varied in patterns that reflected
the subjects’ cultural values. The responses of a sample of lawyers (n = 217) were
also uninfluenced by their cultural values; the responses of a sample of law students
(n = 284), in contrast, displayed a level of cultural bias only modestly less
pronounced than that observed in the general-public sample. Among the competing
hypotheses tested in the study, the results most supported the position that professional
judgment imparted by legal training and experience confers resistance to identity-
protective cognition—a dynamic associated with politically biased information
processing generally—but only for decisions that involve legal reasoning. The
scholarly and practical implications of the findings are discussed.
INTRODUCTION
Are judges politically motivated? Do they make decisions, at least in
ideologically charged cases, on the basis of “policy preferences” or
predispositions?
Most members of the American public think so. Opinion polls suggest
that about three-quarters of Americans believe that judges—U.S. Supreme
Court Justices and lower court jurists alike—base their decisions on their
“personal political views.”1 The charge that judges are “legislating from the
bench” is automatic after decisions involving culturally contested matters—
from gay rights to gun control, affirmative action to stem cell research.2 If a
1 See, e.g., D AVID ROTHMAN , T RUST AND C ONFIDENCE IN THE C ALIFORNIA C OURTS
32 (2005), http://www.courts.ca.gov/documents/4_37pubtrust1.pdf [http://perma.cc/QY7U-M9BE]
(“Overall, outcomes are seen by all respondents as least fair for persons who are low-income or who
do not speak English.”); Kathleen Hall Jamieson & Bruce Hardy, Public Understanding of and Support
for the Courts: Annenberg Public Policy Center Judicial Survey Results, ANNENBERG PUB. POL’Y
CENT. (Oct. 17, 2007), http://www.annenbergpublicpolicy center.org/wp-
content/uploads/Judicial_Findings_10-17-20071.pdf [http://perma.cc/FG9B-YG3S] (“75% of
Americans think that a state judge’s ruling is influenced by his or her politics to a great or modest
extent . . . .”); Opinions of the Supreme Court, N.Y. TIMES (June 7, 2012),
http://www.nytimes.com/interactive/2012/06/08/us/politics/ opinions-of-the-supreme-court.html
[http://perma.cc/K6XY-VLVE ] (summarizing a survey in which 76% of respondents believed that
Supreme Court justices “sometimes let personal or political views influence their decisions”). See
generally KEITH J. BYBEE, ALL J UDGES ARE POLITICAL—EXCEPT WHEN THEY ARE NOT:
ACCEPTABLE HYPOCRISIES AND THE RULE OF LAW 8, 112-13 (2010) (collecting and
summarizing survey results indicating that “[t]he notion that state judges are political is indeed
commonly held”).
2 See, e.g., Ramesh Ponnuru, Editorial, When Judicial Activism Suits the Right, N.Y. T IMES , June
23, 2009, at A29 (suggesting that “[j]udicial restraint has also been absent” in cases striking down
352 University of Pennsylvania Law Review [Vol. 164: 349
affirmative action plans); Editorial, Wrong Direction on Stem Cells, N.Y. TIMES, Aug. 24, 2010, at
A20 (“In a huge overreach, a federal judge has decided that the legal interpretation that has governed
federal support of embryonic stem cell research for more than a decade is invalid.”); Daniel Wilson,
Lawmakers Want Same-Sex Marriage Laws In State Hands, LAW360 (Feb. 11, 2015, 6:51 PM),
http://www.law360.com/articles/620597/lawmakers-want-same-sex-marriage-laws-in-state-hands
[http://perma.cc/YEE4-AG6A] (noting that supporters of state legislation banning same-sex
marriage “were struggling against ‘activist court judges overstepping their constitutional authority
by legislating from the bench’”); D. Robert Worley, Judicial Activism and the Second Amendment,
HUFFINGTON POST (Jan. 4, 2013, 6:37 PM), http://www.huffingtonpost.com/d-robert-
worley/judicial-activism-and-the_b_2412471.html [http://perma.cc/8ZEE-V7LS] (arguing that the
modern interpretation of the Second Amendment is a product of judicial activism).
3 See, e.g., Alex S. Vitale, Why Police Are Rarely Indicted for Misconduct, AL J AZEERA AM . (Nov.
24, 2014, 10:00 PM), http://america.aljazeera.com/opinions/2014/11/ferguson-police-
misconductdarrenwilsongrandjury.html [http://perma.cc/KT6U-QPBQ] (arguing that racial
attitudes stifle judicial regulation of police violence against minorities); Miranda S. Spivak, Marissa
Alexander's Supporters Converge in Florida, WOMENSEWS (Jan. 26, 2015),
http://womensenews.org/story/domestic-violence/150124/marissa-alexanders-supporters-converge-
in-florida [http://perma.cc/LLG8-FSKQ] (“[A] lot of women face [imprisonment] for defending
themselves [against domestic violence] . . . .”).
4 See J AMES Q. W ILSON , M ORAL J UDGMENT: D OES THE ABUSE E XCUSE T HREATEN
OUR LEGAL SYSTEM? 44-70, 111-12 (1997) (arguing that the battered-woman defense is an example
of the law’s receptivity to “excusing accountability for some group defined by their group
membership”); The New York Stop and Frisk Law: Judicial Activism, MYREALITYLAW.ORG (Aug. 14,
2013) http://www.myrealitylaw.org/the-new-york-stop-and-frisk-law-judicial-activism/
[http://perma.cc/Z3NA-3RPA] (characterizing a judge as “legislat[ing] from the bench” and
engaging in “judicial activism” in upholding a constitutional challenge to police stop-and-frisk
policy).
5 See generally LEE E PSTEIN, W ILLIAM M. LANDES & RICHARD A. P OSNER , T HE
BEHAVIOR OF FEDERAL JUDGES: A THEORETICAL AND EMPIRICAL STUDY OF RATIONAL
CHOICE (2013) (attempting to characterize judges’ decisions based on ideological scores); JEFFREY
ALLAN SEGAL, HAROLD J. SPAETH & SARA CATHERINE BENESH, THE SUPREME COURT
IN THE AMERICAN LEGAL SYSTEM (2005) (conducting studies on state and federal judges in an
attempt to explain their decisionmaking in different ideological categories).
6 See, e.g., E ILEEN B RAMAN , L AW , P OLITICS , AND P ERCEPTION : HOW P OLICY
PREFERENCES INFLUENCE LEGAL REASONING 4-5 (2009) (suggesting that even judges who
believe they are acting objectively are “predisposed to find authority consistent with their attitudes
more convincing than cited authority against desired outcomes”); Eileen Braman & Thomas E.
Nelson, Mechanism of Motivated Reasoning? Analogical Perception in Discrimination Disputes, 51 AM. J.
2016] “Ideology” or “Situation Sense”? 353
impression that judges (despite their own protestations to the contrary) are
just “junior varsity politicians.”7
We disagree—not necessarily with the conclusion that judges are
“politically biased,” but with the premise that existing empirical evidence
furnishes secure grounds for crediting it. Popular judgments obviously are
not conclusive: because members of the public lack legal training (indeed,
lack comprehension of even the most rudimentary elements of the legal
system),8 they necessarily are incapable of reliably assessing the validity of
judicial decisions. Inferences from experiments involving members of the
public—including law students—are suspect for exactly the same reason:
because the subjects in such studies are members of the public, their
vulnerability to unconscious ideological predispositions begs the question of
whether the training and experience that judges possess immunizes them
from such influences when they are engaged in legal reasoning.9
Observational studies that purport to correlate the decisions of real judges
with those judges’ “ideologies” have also been reasonably criticized for
methodological problems. One is the obvious selection bias involved in
POL. SCI. 940, 940 (2007) (“[L]egal training did not appear to attenuate motivated perceptions.”);
Eric Posner & Cass Sunstein, Institutional Flip-Flops (Chi. Pub. Law and Legal Theory Working
Paper No. 501, 2015), http://chicagounbound.uchicago.edu/cgi/
viewcontent.cgi?article=1953&context=public_law_and_legal_theory [https://perma.cc/2Z9X-26ZH]
(positing that short-term political commitments cloud decisionmakers’ long-term complex values).
7 See, e.g., Book Discussion on Making Our Democracy Work: A Judge’s View, C-SPAN (Sept.
13, 2010), http://www.c-span.org/video/?297408-1/book-discussion-making-democracy-work-judges-
view [http://perma.cc/ZPC2-5EBT] (interview with Justice Stephen Breyer) (cited material at
24:42 of video) (rejecting view that Supreme Court consists of “nine junior varsity politicians”).
Although there does appear to be growing public consensus that the U.S. Supreme Court and other
courts are “political,” whether judges are “too liberal” or “too conservative” is an issue that generally
divides ordinary Americans in patterns characteristic of partisan divides on contentious issues.
Supreme Court’s Favorable Rating Still at Historic Low: Few Conservatives View the Roberts Court as
Conservative, PEW RES. CTR. (Mar. 25, 2013), http://www.people-press.org/2013/03/25/ supreme-
courts-favorable-rating-still-at-historic-low/ [http://perma.cc/ Z8DC-94LE] (noting that Republicans
tend to see the Supreme Court’s rulings as “liberal” while Democrats tend to see them as
“conservative”).
8 See, e.g., Jamieson & Hardy, supra note 1 (finding that less than one-third of the American
public knows that U.S. Supreme Court rulings cannot be appealed).
9 See Dan M. Kahan, Laws of Cognition and the Cognition of Law, C OGNITION , Feb. 2015, at 56,
59-60 [hereinafter Kahan, Cognition of Law] (noting that studies examining influences of motivated
cognition on legal reasoning are not based on judges); Dan M. Kahan, The Supreme Court 2010
Term—Foreword: Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law,
126 HARV. L. REV. 1, 27-28 (2011) [hereinafter Kahan, Neutral Principles, Motivated Cognition]
(noting that it is a mistake to infer vulnerability of public to motivated reasoning generalizes to
judges).
354 University of Pennsylvania Law Review [Vol. 164: 349
studying only cases that are actually litigated.10 Another is the failure of
researchers to specify the measured outcome variable in a manner that
distinguishes the illicit contribution of political sensibilities extrinsic to law
from the licit contribution of political sensibilities intrinsic to conventional
legal reasoning.11 These methodological shortcomings, critics assert, raise doubts
about the strength of the inferences that can be drawn from these studies.
The most satisfactory way to overcome these limitations, we believe, is
through valid experiments performed on judges.12 In this Article, we present
the results of such an experiment. In this study, judges, lawyers, and law
students were instructed to assess legal problems designed to trigger
unconscious political bias in members of the general public.
The experimental results furnished evidence strongly at odds with the
conclusion that judges are influenced by political predispositions when they
engage in legal reasoning. Judges of diverse cultural outlooks—ones polarized
on their views of the risks of marijuana legalization, climate change, and other
contested issues—converged on results in cases that strongly divided
comparably diverse members of the public. Culturally diverse lawyers also
displayed a high degree of consensus in their legal determinations. Law
students, in contrast, did not; in addressing the legal problems featured in the
experiment, they polarized along the same lines that divided the legally
untrained members of the public, although to a lesser extent.
These results strongly support the hypothesis that professional judgment
can be expected to counteract “identity-protective cognition,” the species of
motivated reasoning known to generate political polarization over risks and
myriad policy and legally consequential facts.13 Legal training and experience,
on this view, endows judges and lawyers with a specialized form of cognitive
10 See generally Jonathan P. Kastellec & Jeffrey R. Lax, Case Selection and the Study of Judicial
Politics, 5 J. EMPIRICAL LEGAL STUD. 407 (2008) (arguing that case selection influences Supreme
Court decisionmaking).
11 See generally Harry T. Edwards & Michael A. Livermore, Pitfalls of Empirical Studies That
Attempt to Understand the Factors Affecting Appellate Decisionmaking, 58 DUKE L.J. 1895, 1945-46
(2008) (pointing out that in some cases the law requires moral judgments that, far from “tak[ing]
the form of personal whim or preference . . . can include a situated and disciplined elaboration of
the conventional norms of the American political community”).
12 See Avani Mehta Sood, Motivated Cognition in Legal Judgments-an Analytic Review, 9 ANN .
REV. L. & SOC. SCI. 307, 318 (2013) (“Although much can be gleaned from combining the
experimental literature on motivated cognition with existing work on judicial biases more generally,
there is a need for more targeted and systematic experiments that specifically investigate motivated
cognition in populations of judges.”).
13 See Dan Kahan, Fixing the Communications Failure, 463 N ATURE 296 (2010) (likening
“protective cognition” to “cultural cognition,” in which an individual’s group values influence her
or her risk perceptions and related beliefs); David K. Sherman & Geoffrey L. Cohen, The Psychology
of Self-Defense: Self-Affirmation Theory, 38 ADVANCES EXPERIMENTAL SOC. PSYCHOL. 183 (2006).
2016] “Ideology” or “Situation Sense”? 355
17 See Kahan, Neutral Principles, Motivated Cognition, supra note 9, at 4 (commenting that both
political parties criticize the Supreme Court for being excessively partisan).
2016] “Ideology” or “Situation Sense”? 357
A. Observational Studies
Associated with the disciplines of political science and economics, studies
that use observational methods make up the largest share of the literature on
the impact of ideological motivations on judicial decisionmaking. Such
studies use correlational analyses—in the form of multivariate regression
models—that treat the “ideology” of individual judges as one “independent
variable” the impact of which on case outcomes is assessed after “controlling
for” additional influences represented by other “independent variables.”18
There are different methods for measuring judges’ “ideologies,” including
(in the case of federal judges) the party of the appointing President19 and (in
the case of Supreme Court Justices) the covariance of votes among judges
who can be understood to be aligned along some unobserved or latent
ideological continuum.20 Such studies tend to find that “ideology” so
measured explains a “statistically significant” increment of variance in judicial
determinations. Studies looking at the decisions of federal courts of appeals,
which assign cases to three-judge panels for determination, also find that the
impact of ideology so measured can be either accentuated or muted
depending on the ideological composition of judges on the particular panel.21
18 See generally EPSTEIN, POSNER & LANDES, supra note 5 (using this method to classify
judicial ideologies).
19 See, e.g., C ASS R. SUNSTEIN, D AVID S CHKADE , LISA M. E LLMAN & ANDRES S AWICKI ,
ARE JUDGES POLITICAL? AN EMPIRICAL ANALYSIS OF THE FEDERAL JUDICIARY (2006).
Political scientists have enriched this measure in important and innovative ways that take account
of, among other things, interactions between the appointing President and the confirming Senate.
See generally Lee Epstein, Andrew D. Martin, Jeffrey A. Segal & Chad Westerland, The Judicial
Common Space, 23 J.L. ECON. & ORG. 303 (2007) (critiquing past efforts to quantify political
analyses of the judiciary and offering new methodology); Michael W. Giles, Virginia A Hettinger
& Todd Peppers, Picking Federal Judges: A Note on Policy and Partisan Selection Agendas, 54 POL. RES .
Q. 623 (2001) (using a methodology that analyzes federal courts of appeals and their appointments
in relation to senatorial preferences).
20 See generally Andrew D. Martin & Kevin M. Quinn, Dynamic Ideal Point Estimation Via
Markov Chain Monte Carlo for the U.S. Supreme Court, 1953–1999, 10 POL. ANALYSIS 134 (2002)
(explaining a formula-based study intended to reveal political preferences of Supreme Court Justices
over time). But see Joseph Bafumi, Andrew Gelman, David K. Park & Noah Kaplan, Practical Issues
in Implementing and Understanding Bayesian Ideal Point Estimation, 13 POL. ANALYSIS 171, 186 (2005)
(questioning the validity of the methodology used to array Justices and predict Supreme Court case
outcomes).
21 See, e.g., Richard L. Revesz, Congressional Influence on Judicial Behavior? An Empirical
Examination of Challenges to Agency Action in the D.C. Circuit, 76 N.Y.U. L. REV. 1100, 1104 (2001)
358 University of Pennsylvania Law Review [Vol. 164: 349
(examining D.C. Circuit decisions involving health and safety regulations to determine if political
party appointment affected case outcomes).
22 See Edwards & Livermore, supra note 11, at 1922 (noting several methodological issues with
empirical studies of judicial decisions).
23 See George L. Priest & Benjamin Klein, The Selection of Disputes for Litigation, 13 J. LEGAL
STUD. 1, 4 (1984) (arguing that litigants are economically motivated and likely choose to pursue
cases based on cost or likelihood of success).
2016] “Ideology” or “Situation Sense”? 359
27 See, e.g., FRANK B. C ROSS , D ECISION M AKING IN THE U.S. C OURTS OF APPEALS 20
(2007) (discussing examples of ideological codifications for different types of case outcomes as well
as the problems associated with these codifications); EPSTEIN, LANDES , & POSNER, supra note 5
at 107-16 (noting that the definitions of “liberal” and “conservative” have changed over time);
SUNSTEIN , SCHKADE, ELLMAN & SAWICKI, supra note 19 (explaining the methodology behind
a study of three-judge panel decisions that used “liberal” or “conservative” to classify case outcomes).
28 See e.g., Edwards & Livermore, supra note 11, at 1945-48 (arguing that judges’ ideologies are
not necessarily extrinsic to the law, and that studies using measures of “liberal” and “conservative”
fail to account for this); Carolyn Shapiro, Coding Complexity: Bringing Law to the Empirical Analysis
of the Supreme Court, 60 HASTINGS L.J. 477, 485-88 (2009) (criticizing previous studies that simplify
judicial opinions into a liberal–conservative dichotomy).
29 See generally RONALD D WORKIN , LAW ’ S E MPIRE (1986) (providing an overview of
different legal theories and their varying emphasis on judges’ normative reasoning); E DWARD H.
LEVI, AN INTRODUCTION TO LEGAL REASONING (2013) (providing a realist explanation to legal
reasoning).
30 See Neal Devins, How State Supreme Courts Take Consequences into Account: Toward a State-
Centered Understanding of State Constitutionalism, 62 STAN. L. REV. 1629, 1655-56 (2009) (describing
state court exercise of consequentialist judgment contemplated by conventional understandings of
constitutional jurisprudence).
31 See Henry J. Friendly, In Praise of Erie—and of the New Federal Common Law, 39 N.Y.U. L.
REV. 383, 405 (1964) (discussing persistence of “specialized federal common law”); Thomas W.
Merrill, The Common Law Powers of Federal Courts, 52 U. CHI. L. REV. 1, 5 (1985) (describing
ubiquity of statutes that implicitly delegate federal common lawmaking power).
32 See generally D WORKIN , supra note 29; Jeffrey A. Pojanowski, Reading Statutes in the Common
Law Tradition, 101 VA. L. REV. 1357 (2015).
2016] “Ideology” or “Situation Sense”? 361
all demand the use of the form of guided normative theorizing we are
describing. But the bare desire to use legal outcomes in particular cases (or in
large classes of them) to disadvantage those who subscribe to a disfavored
view of the best life or to advance the cause of a particular political party is
plainly outside the range of considerations that can validly be appealed to in
the exercise of normative reasoning intrinsic to law. Whether in the form of
regression coefficient correlations, law-enforcement wiretaps, or
anonymously leaked emails, evidence that judges of particular ideologies were
being influenced by such considerations would be a ground for intense
concern.
There is a distinction, in sum, between resort to normative considerations
that are internal to law and ones external to it. The former are licit, the latter
illicit, from the perspective that lawyers and judges in the U.S. system of
justice share of what counts as valid legal reasoning.37
The “prevailing party” outcome-classification scheme used in
observational studies of judicial ideology is blind to the distinction. As a
result, such studies will count in their estimates of the influence of “ideology”
perfectly mundane associations between the jurisprudential philosophies of
judges deciding cases on the basis of normative considerations internal to law
and the party of the Presidents who appointed them or the voting records of
those judges and judges who feel likewise about the normative theories that
inform labor law, free speech cases, criminal cases and the like.38
The correlations that these researchers report could also be capturing
judges’ reliance on illicit political considerations, external to the law. But, as
critics point out, there is no way to know whether this is the case, or to what
extent, given the indiscriminate coding of outcome variables that these
studies employ.39
Some candid adherents to the “ideology thesis”40 have acknowledged this
point.41 But they have not supplied a response to what critics would identify
mechanically,” but rather that “its scope depends on the structure and functions of the substantive
statute[,]” the interpretations of which may be guided by normative concerns).
37 See Edwards & Livermore, supra note 11, at 1947-48.
38 See id. at 1927 (noting that outcome-based coding of judicial decisions ignores important
purposes and factors of judicial decisionmaking).
39 See Carolyn Shapiro, The Context of Ideology: Law, Politics, and Empirical Legal Scholarship, 75
MO. L. REV. 1 (2010) (criticizing current ideological coding of judicial decisions).
40 See Edwards & Livermore, supra note 11, at 1959-62 (discussing points of support and
criticism towards the ideology thesis).
41 See Thomas J. Miles & Cass R. Sunstein, The New Legal Realism, 75 U. C HI . L. REV. 831,
844 (2008) (identifying as a “jurisprudential” issue worthy of “further exploration” the possibility
that variance between Democratic and Republican judicial appointees might be explained consistent
2016] “Ideology” or “Situation Sense”? 363
B. Experimental Studies
Studies that use experimental methods have been used to examine the
impact of motivated reasoning on legal decisionmaking. Motivated reasoning
refers to the tendency of people to conform their assessments of
information—from logical arguments to empirical data, from expert
judgments to their own sense impressions—to some end or goal extrinsic to
judgment accuracy.42
One such interest can be to protect the status of, or one’s own standing
in, an important affinity group, a form of motivated reasoning known as
identity-protective cognition.43 The impact of identity-protective cognition
in distorting assessments of evidence has been identified as an important
source of political polarization over issues like climate change, gun control,
the HPV vaccine, nuclear power, and the like.44
Studies have demonstrated that identity-protective cognition can affect
assessment of evidence relevant to legal decisions, too.45 In mock juror
studies, subjects have been shown to form different assessments of the facts
in cases involving alleged intimidation of pedestrians by political protestors,
excessive force by police, and violent confrontations between private citizens,
depending on the relationship of the group identities and values of the
parties, on the one hand, and the study subjects’ own group commitments,
on the other.46 Studies have also shown that identity-protective cognition can
with “Ronald Dworkin's account of law as a search for ‘integrity,’ through which judges seek both
to ‘fit’ and to "justif[y] preexisting legal decisions”).
42 Ziva Kunda, The Case for Motivated Reasoning, 108 PSYCHOL. B ULL. 480, 480-81 (1990).
43 See Sherman & Cohen, supra note 13, at 191-92.
44 See, e.g., Kahan, supra note 13, at 296.
45 See Sood, supra note 12.
46 See Yael Granot, Emily Balcetis, Kristin E. Schneider & Tom R. Tyler, Justice Is Not Blind:
Visual Attention Exaggerates Effects of Group Identification on Legal Punishment, 143 J. EXPERIMENTAL
PSYCHOL. 2196 (2014); Dan M. Kahan, David A. Hoffman, Donald Braman, Danieli Evans &
Jeffrey J. Rachlinski, They Saw a Protest: Cognitive Illiberalism and the Speech–Conduct Distinction, 64
364 University of Pennsylvania Law Review [Vol. 164: 349
STAN. L. REV. 851 (2012); Dan M. Kahan, David A. Hoffman & Donald Braman, Whose Eyes Are
You Going to Believe? Scott V. Harris and the Perils of Cognitive Illiberalism, 122 HARV. L. REV. 837
(2009).
47 See Dan M. Kahan, Culture, Cognition, and Consent: Who Perceives What, and Why, in
Acquaintance-Rape Cases, 158 U. PA. L. REV. 729, 804-06 (2010).
48 See Dan M. Kahan & Donald Braman, The Self-Defensive Cognition of Self-Defense, 45 AM .
CRIM. L. REV. 1, 34-35 (2008).
49 See e.g., id.; Kahan, Hoffman & Braman, supra note 46; Janice Nadler & Mary-Hunter
McDonnell, Moral Character, Motive, and the Psychology of Blame, 97 CORNELL L. REV. 255, 274
(2012); Avani Mehta Sood & John M. Darley, The Plasticity of Harm in the Service of Criminalization
Goals, 100 CALIF. L. REV. 1313, 1324-25 (2012).
50 See B RAMAN , supra note 6, at 87; Braman & Nelson, supra note 6, at 945; Ward Farnsworth,
Dustin F. Guzior & Anup Malani, Implicit Bias in Legal Interpretation (John M. Olin Program in Law &
Economics, Working Paper No. 577, 2011).
51 See Sood, supra note 12, at 318.
52 See Kahan, Neutral Principles, Motivated Cognition, supra note 9, at 4.
2016] “Ideology” or “Situation Sense”? 365
cultural orthodoxy,53 but also mandate strict scrutiny to flush out hidden
motivations for disadvantaging those who deviate from dominant norms.54
Procedural rules recognize the same vulnerability in jurors, forbidding the
introduction of evidence that might excite aversion to a party’s identity or
values.55 Instilling the reasoning skills and habits of mind essential to
administering these rules in a reliable and even-handed manner is one of—if
not the—central objective of legal training.
Because the entire point of this regime is to insulate the law from the
impact of identity-protective cognition on ordinary members of the public, it
is question-begging to cite the public’s vulnerability to that bias as reason to
believe that it distorts the reasoning of judges as well.56 It is certainly not
unreasonable—indeed, it is quite plausible—to hypothesize that identity-
protective cognition could be disabling judges from shielding liberal
democratic government from the threat posed to it by its diverse citizenry’s
own vulnerability to identity-protective cognition. But to the extent that such
conjecture is based on casual observation of how courts resolve controversial
cases, the experimental study of identity-protective cognition actually
furnishes strong evidence to be skeptical of such a surmise: those studies show
us that ordinary members of the public, precisely because they selectively
credit and discredit all manner of information in patterns congenial to their
own cultural predispositions, can be expected to perceive court decisions they
disagree with as biased even if those decisions reflect the impartial application
of neutral principles of law.57
53 See W. Va. State Bd. of Educ. v. Barnette, 319 U.S. 624, 642 (1943) (“If there is any fixed
star in our constitutional constellation, it is that no official, high or petty, can prescribe what shall
be orthodox in politics, nationalism, religion, or other matters of opinion . . . .”).
54 See J OHN HART E LY , D EMOCRACY AND D ISTRUST: A T HEORY 0F J UDICIAL REVIEW
146 (1980) (“[S]pecial scrutiny . . . turns out to be a way of ‘flushing out’ unconstitutional motivation
. . . .”); Elena Kagan, Private Speech, Public Purpose: The Role of Governmental Motive in First
Amendment Doctrine, 63 U. CHI. L. REV. 413, 431 n.55, 453-55, 500-01 (1996) (arguing that the strict
scrutiny standard is a means of measuring government motive in passing laws that disadvantage
certain groups); Jed Rubenfeld, Affirmative Action, 107 YALE L.J. 427, 436-37 (1997) (“[Strict
scrutiny’s] function . . . is to smoke out illegitimate purposes that cannot be a valid basis for state
action under the Equal Protection Clause.”).
55 See, e.g., FED . R. E VID . 404 (barring “character propensity” proof); F ED . R. E VID . 610
(barring “evidence of a witness’s religious beliefs or opinions . . . to attack or support the witness’s
credibility”).
56 Cf. B RAMAN , supra note 6, at 87-89; Braman & Nelson, supra note 6, at 940-45 (implying
that the results of an experiment with students support inferences on how judges reason in legal
decisions).
57 See Kahan, Cognition of Law, supra note 9, at 60. Some judges report believing that identity-
protective cognition or related forms of motivated reasoning also affect judicial reasoning. See, e.g.,
RICHARD A. POSNER, HOW JUDGES THINK 116 (2008) (asserting that judges rely on “cultural
cognition” in assessing empirical claims). We do not think that this is relevant evidence. The impact
366 University of Pennsylvania Law Review [Vol. 164: 349
The only valid means to test whether judges are prone to ideological bias
as a result of identity-protective cognition is to perform valid identity-
protective cognition studies on actual judges.58 No such studies have yet been
performed.59
But two are at least suggestive. In an important recent article, Andrew
Wistrich, Jeffrey Rachlinski, and Chris Guthrie report a series of experiments
that compared the determinations of judges randomly assigned to conditions
that featured either “sympathetic” or “unsympathetic” litigants.60 In one
involving the application of a medical marijuana statute, a larger proportion
of judges accepted the adequacy of a medical-need affidavit proffered by a
middle-aged man who was “married with three children,” “employed as an
accountant,” and who “lacked a “criminal record” than that of a 19-year old
“currently unemployed” man who was “on probation for beating his ex-
girlfriend, and had a juvenile record for drug possession and drug dealing.”
Similarly, a group of bankruptcy judges were less likely to discharge the debt
of a college student who used a credit card “for spring break, where she
charged her hotel room, meals, and rounds of drinks for friends,” than they
were to discharge that of a student who used the card to visit and buy
medicines for her mother, “who was battling cancer, lacked health insurance,
and needed assistance recovering from a recent surgery.”61 The authors treat
these and similar results as evidence that the judges’ had been “motivated” to
reach outcomes that matched their “emotional” evaluations of the parties.62
Significantly, however, judges of different political ideologies did not react
differently to the experimental stimuli.63 Indeed, Wistrich et al. express
surprise at “the lack of a political influence given the widespread findings [in
observational studies] that politics influences appellate judges.”64 Because
Wistrich et. al report that their experiments furnished “little support for the
proposition that political ideology drives much judicial decision making,”65 it
of unconscious biases on oneself cannot be reliably detected by introspection, much less discerned
in others via casual observation. See generally Emily Pronin, Daniel Y. Lin & Lee Ross, The Bias
Blind Spot: Perceptions of Bias in Self Versus Others, 28 PERSONALITY & SOC. PSYCHOL. BULL. 369
(2002).
58 Kahan, Cognition of Law, supra note 9, at 60; Kahan, Neutral Principles, Motivated Cognition,
supra note 9, at 27-28.
59 See Sood, supra note 12, at 318 (“[T]here is a need for more targeted and systematic
experiments that specifically investigate motivated cognition in populations of judges.”).
60 See Andrew J. Wistrich, Jeffrey J. Rachlinski & Chris Guthrie, Heart Versus Head: Do Judges
Follow the Law of Follow Their Feelings?, 93 Tex. L. Rev. 855 (2015).
61 Id. at 888.
62 See id. at 899-900.
63 See id. at 880, 889-90.
64 Id. at 880.
65 Id. at 899.
2016] “Ideology” or “Situation Sense”? 367
66 Id. at 899.
67 Some of Wistrich et al.’s results can fairly be read this way but many cannot. In the medical-
marijuana problem, for example, the statute expressly stated that the medical-need affidavit must
indicate that the “the medical use of marijuana” was necessary “to treat or alleviate the person’s
serious or debilitating medical condition or symptoms.” Id. at 914 (emphasis added). The “sympa-
thetic” party’s affidavit in fact described him as suffering from “severe pain caused by bone cancer,”
a “debilitating” condition that “would likely kill him in a year”; the “unsympathetic” defendant’s
affidavit, in contrast, stated that he was “being treated for occasional mild seizures” that were “not
debilitating and might abate within a year.” Id. (emphasis added). In this problem, then, the emo-
tional sensibilities that Wistrich et al. attribute to the judges in the two conditions matched up with
the language of the statute being construed. In the bankruptcy case, too, differing reactions to the
“sympathetic” and “unsympathetic” parties’ reasons for incurring debt were relevant to the credibility
of their respective denials of fraudulent intent, the honesty of which was the only issue for decision.
68 See Richard E. Redding & N. Dickon Reppucci, Effects of Lawyers’ Socio-Political Attitudes on
Their Judgments of Social Science in Legal Decision Making, 23 L. & HUM. BEHAV. 31 (1999).
69 See id. at 34-35.
70 Id. at 48.
368 University of Pennsylvania Law Review [Vol. 164: 349
71 See James N. Druckman, The Politics of Motivation, 24 C RITICAL REV . 199, 203-07 (2012);
Alan Gerber & Donald Green, Misperceptions About Perceptual Bias, 2 ANN. REV. POL. SCI. 189,
206 (1999); Kahan, Cognition of Law, supra note 9, at 60.
72 See supra notes 30–35 and accompanying text; cf. Redding & Reppucci, supra note 68, at 48.
This same difficulty—the use of designs that tested the influence of values intrinsic rather than
genuinely extrinsic to the legal problem being addressed—constrains the strength of the inferences
that can be drawn from various other studies involving law students and members of the public. See,
e.g., Braman & Nelson, supra note 6, at 947 (manipulating the authorities in a constitutional law case
pertinent to reconciling tension between anti-discrimination and free speech principles); Nadler &
McDonnell, supra note 49, at 73 (manipulating the moral quality of behavior legally relevant to
assessing issue of “causation”).
73 Sood, supra note 12, at 318.
2016] “Ideology” or “Situation Sense”? 369
74 See generally Christine Jolls, Cass R. Sunstein & Richard Thaler, A Behavioral Approach to
Law and Economics, 50 STAN. L. REV. 1471 (1998) (proposing an economic analysis of legal rules
and legal actors based in considerations of actual behavior that include fallacious and directional
decisionmaking).
75 See Chris Guthrie, Jeffrey J. Rachlinski & Andrew J. Wistrich, Blinking on the Bench: How
Judges Decide Cases, 93 CORNELL L. REV. 1, 19-27 (2007) (reviewing studies of judicial
decisionmaking which analyze the influence of anchoring, statistical inferences, and hindsight bias).
76 See generally Daniel Kahneman, Maps of Bounded Rationality: Psychology for Behavioral
Economics, 93 AM. ECON. REV. 1449 (2003); Daniel Kahneman & Shane Frederick, A Model of
Heuristic Judgment, in THE CAMBRIDGE HANDBOOK OF THINKING AND REASONING 267, 267-
68 (2005).
77 See Kahan, supra note 16, at 416; Kahan, Peters, Wittlin, Slovic, Ouellette, Braman &
Mandel, supra note 16.
370 University of Pennsylvania Law Review [Vol. 164: 349
are in fact the most politically polarized on contested issues like climate
change, gun control, nuclear power, fracking, and the like.78
Legal training and practice can reasonably be understood to cultivate
proficiency in conscious, analytical forms of reasoning. Thus, the work on
“motivated System 2 reasoning”—that portion of the literature that examines
the tendency of conscious, effortful information processing to magnify
identity-protective cognition79—might in fact be regarded as furnishing
strong support for the conjecture that unconscious cultural partisanship can
be expected to subvert judicial neutrality.80
Nevertheless, when judges decide cases, they are not merely engaging in
conscious, effortful information processing. They are exercising professional
judgment. Professional judgment consists of habits of mind—conscious and
effortful to some degree, but just as much tacit and perceptive—that are
distinctively fitted to reasoning tasks that fall outside ordinary experience.81
Indeed, it is characterized in many fields by resistance to all manner of error,
including ones founded on heuristic information processing that would defeat
the special form of decision that professional judgment facilitates.82
The dominant scholarly account of professional judgment roots it in the
dynamic of pattern recognition.83 Pattern recognition consists of the rapid un-
or pre-conscious matching of phenomena with mentally inventoried
prototypes. A ubiquitous form of information processing, pattern recognition
is the type of cognition that enables human beings to recognize faces and read
one another’s’ emotions.84 But it is also the basis for many forms of highly
78 See Dan M. Kahan, What is the “science of science communication”?, J. SCI . C OMM . (Aug. 25,
2015), http://jcom.sissa.it/archive/14/03/JCOM_1403_2015_Y04 [http://perma.cc/NL8M -U8MM].
79 See Dan M. Kahan, Ellen Peters, Erika Cantrell Dawson & Paul Slovic, Motivated Numeracy
and Enlightened Self-Government 25 (Cultural Cognition Project, Working Paper No. 116, 2013),
http://static1.1.sqspcdn.com/static/f/386437/23982003/1385735927633/
wp_draft_1.5_9_14_13.pdf?token=k3oG7Vfla1yerw5DPKyLzu6KN2A%3D [http://perma.cc/Q9UD-
CVWG]; supra note 16.
80 In fact, evidence suggests that judges score higher than average on the Cognitive Reflection
Test (CRT), see Shane Frederick, Cognitive Reflection and Decision Making, J. ECON. PERSP., Fall
2005, at 25, which is the standard measure of the capacity and disposition to use System 2 reasoning.
Guthrie, Rachlinski & Wistrich report that a large sample of state court judges (N = 250) attained
a mean CRT score of 1.23. Guthrie, Rachlinski & Wistrich, supra note 75, at 14. The mean score in
a general population sample is approximately 0.65. The score of the judges in the conference would
place them between the 75th and 90th percentile for the general population. Kahan, supra note 77,
at 410.
81 See MARGOLIS , supra note 15, at 35.
82 See id.
83 See generally HOWARD M ARGOLIS , P ATTERNS , T HINKING , AND C OGNITION : A
THEORY OF JUDGMENT (1987).
84 See P AUL M. C HURCHLAND, THE E NGINE OF REASON , THE SEAT OF THE S OUL : A
PHILOSOPHICAL JOURNEY INTO THE BRAIN 27-42, 123-32 (1995).
2016] “Ideology” or “Situation Sense”? 371
85 See Erik Dane & Michael G. Pratt, Exploring Intuition and Its Role in Managerial Decision
Making, 32 ACAD. MGMT. REV. 33, 42-43 (2007).
86 See Herbert A. Simon & William G. Chase, Skill in Chess: Experiments with Chess-Playing
Tasks and Computer Simulation of Skilled Performance Throw Light on Some Human Perceptual and
Memory Processes, 61 AM. SCIENTIST 394, 402 (1973).
87 See E LEANOR J. G IBSON , P RINCIPLES OF P ERCEPTUAL LEARNING AND
DEVELOPMENT 8 (1969).
88 See, e.g., Jean C. Bedard & Stanley F. Biggs, Pattern Recognition, Hypotheses Generation, and
Auditor Performance in an Analytical Task, 66 ACCT. REV. 622, 624 (1991).
89 See generally James A. Marcum, An Integrated Model of Clinical Reasoning: Dual-Process Theory
of Cognition and Metacognition, 18 J. EVALUATION CLINICAL PRAC. 954 (2012) (proposing a model
of reasoning by treating physicians which relies on initial pattern recognition); Geoff Norman,
Meredith Young & Lee Brooks, Non-Analytical Models of Clinical Reasoning: The Role of Experience,
41 MED. EDUC. 1140 (2007) (demonstrating experimentally that expert medical diagnosis relies on
initial recognition of significant patterns); Vimla L. Patel, David R. Kaufman & Jose F. Arocha,
Emerging Paradigms of Cognition in Medical Decisionmaking, J. BIOMEDICAL INFORMATICS, Feb.
2002, at 52 (suggesting additional research into intuitive, non-analytical components of medical
diagnostic decisionmaking); Vimla L. Patel & Guy J. Groen, Knowledge Based Solution Strategies in
Medical Reasoning, 10 COGNITIVE SCI. 91 (1986) (reporting that accurate diagnoses in an empirical
study were reached by physicians who relied on an initial experiential knowledge base before
immediately resorting to hypothesis testing).
372 University of Pennsylvania Law Review [Vol. 164: 349
Mertz, Ketti Mazzocco & Stephan Dickert, Numeracy and Decision Making, 17 PSYCHOL. SCI. 407
(2006).
94 See KARL L LEWELLYN , T HE C ASE L AW SYSTEM IN AMERICA 73-77, (Paul Gewirtz ed.,
Michael Ansaldi trans., 1989):
The words “legal certainty” seem to evoke in most lawyers’ minds an image of simply
being able to apply an existing rule of law deductively. We are used to thinking like
this, particularly since judicial opinions and legal discourse must always be dressed up
this way so as to be socially acceptable. My claim would be, though, that for the cases
which occasion difficulties, this kind of legal certainty never has existed and never will
exist. . . .
[Yet] [i]n spite of all this, the outcome of a dispute concerning the law is predict-
able to a truly amazing degree, and for that reason the law is (descriptively) certain.
95 See KARL N. LLEWELLYN , T HE C OMMON LAW T RADITION : D ECIDING APPEALS 59-
61, 121-57, 206-08 (1960) (examining a variety of cases through the “situation-sense” framework).
96 See id. at 183-91 (describing rules as forms of “singing reason” that reliably summon situation
types); Karl N. Llewellyn, On Reading and Using the Newer Jurisprudence, 40 COLUM. L. REV. 581,
590 (1940) (examining cases to reveal that the successful appellant did not identify logical constraints
on judges’ reasoning but “‘simply’ got the court to ‘see’ ‘the true’ rule and its bearing”). Some
observational study proponents of the “ideology thesis”—the view that ideological predispositions
shape judging independently of legal reasoning, see Edwards & Livermore, supra note 11, at 1945—
characterize their outlook as the “New Legal Realism,” and claim they are refining and sharpening
Llewellyn’s views of judicial decisionmaking. See Miles & Sunstein, supra note 41, at 831; see also
EPSTEIN, POSNER & LANDES, supra note 5, at 25-63 (characterizing the use of ideological variables
in multivariate regression models to constitute a “realistic theory of judicial behavior” and
castigating so-called “legalists” and “anti-realists” critics of such an approach). Legal Realism, of
course, comprised a diverse set of thinkers whose various theories of adjudication varied in their
systematization and ultimately were only loosely related. But the equation of Llewellyn’s position,
in particular, with the view that judges are motivated, consciously or unconsciously, by “political”
sensibilities seems out of keeping with how Llewellyn himself understood his own work. To be sure,
Llewellyn rejected the proposition that legal reasoning can be faithfully represented as the deductive
application of formal rules. Nevertheless, he vehemently insisted—in opposition, in fact, to the
views of certain other Legal Realists, most notably Jerome Frank, see Llewellyn, supra note 96, at
374 University of Pennsylvania Law Review [Vol. 164: 349
A. Design
1. Overview
Making valid inferences about how the professional judgment of lawyers
and judges interacts with identity-protective cognition requires comparing
593, 598-601—that judges can be expected to converge in their rulings based on a shared
apprehension of the pertinent features of recurring “situation types.” See id. at 589-91.
97 See MARGOLIS , supra note 81, at 67.
98 See id. at 49-63; M ARGOLIS , supra note 83, at 87-111.
99 See MARGOLIS , supra note 81, at 35-36, 67-68, 94-95.
2016] “Ideology” or “Situation Sense”? 375
how lawyers and non-lawyers reason with regard to both legal and nonlegal
decisionmaking tasks. That is what our study did.
2. Sample
Overall, there were 1554 subjects. Eight hundred of them consisted of
adult members of the U.S. general public. Recruited and stratified for
national representativeness,100 the sample was 49% male, and 76% of its
members were white, 11% African-American, and 12% Hispanic. The average
age was 52. The median educational attainment was “some college,” and the
median income was $40,000 to $49,000.
Two hundred and fifty-three judges participated in the study.
Approximately 200 responded to solicitations issued to judicial conference
attendees.101 The remainder were recruited from participants in two online
CLE lectures.102 All were state judges and consisted of a mix of roughly equal
numbers of trial and appellate court members.
The sample also included 225 lawyers and 250 law students. The former
were recruited via solicitations to attorneys in Connecticut, Illinois, New
York, Pennsylvania, Texas, Virginia and Washington, D.C. Law student
subjects were recruited from Harvard Law School, the Temple University
Beasley School of Law, William & Mary Law School, and Yale Law School.
The law students were a mix of first-, second-, and third-year students, but
those who had not yet completed at least one semester were excluded.
In all cases, the solicitation indicated that the study was designed to
enable study of legal and related forms of reasoning in members of the public,
law students, lawyers, and judges. Beyond that, no information was supplied
on the nature of the study.
100 The general public sample was recruited by the public opinion research firm YouGov, which
conducts online surveys and experiments on behalf of academic and governmental researchers and
commercial customers (including political campaigns). The firm’s general population recruitment
and stratification methods have been validated in studies comparing the results of YouGov surveys
with those conducted for American National Election Studies. See Stephen Ansolabehere & Douglas
Rivers, Cooperative Survey Research, 16 ANN. REV. POL. SCI. 307, 320 (2013).
101 The conferences included the 2014 Annual Conference of the Council of Chief Judges of
State Courts of Appeals; the 2014 Annual Education Program of the Florida Conference of District
Court of Appeal Judges; the 2013 Annual Judicial Education Conference of the Texas Center for
the Judiciary; and the September 2013 Advanced Science and Technology Adjudication Resource
(ASTAR) Center meeting (a seminar for judges on issues in forensic proof).
102 The lectures, for which notices were issued by the National Center for State Courts and the
National Judicial College, were furnished to enable judges to satisfy continuing legal education.
Study solicitations were sent to participating judges, whose participation was of course optional.
376 University of Pennsylvania Law Review [Vol. 164: 349
103 See Cultural Cognition Worldview Scales (CCWS)—Long & Short Forms, DECISION M AKING
INDIVIDUAL DIFFERENCES INVENTORY, http://www.sjdm.org/dmidi/Cultural
_Cognition_Worldview_Scales.html [http://perma.cc/624M-D7LB] (last visited Oct. 31, 2015).
104 This scheme is itself intended to operationalize the “cultural theory of risk” associated with
Mary Douglas and Aaron Wildavsky. See MARY DOUGLAS & AARON B. WILDAVSKY, RISK AND
CULTURE: AN ESSAY ON THE SELECTION OF TECHNICAL AND ENVIRONMENTAL DANGERS
(1982) see also Steve Rayner, Cultural Theory and Risk Analysis (using grid or group analysis to predict
and understand individual preferences), in SOCIAL THEORIES OF RISK 83, 87-91 (Sheldon
Krimsky & Dominic Golding eds., 1992).
105 Dan M. Kahan, Cultural Cognition as a Conception of the Cultural Theory of Risk, in
HANDBOOK OF RISK THEORY: E PISTEMOLOGY, DECISION THEORY, ETHICS AND SOCIAL
IMPLICATIONS OF RISK 725, 730-35 (R. Hillerbrand et al. eds., 2012).
106 See generally Kahan, supra note 13 (providing examples of studies that have used CCWS
measures to analyze respondents' perceptions of various social issues, including environmental
problems, nanotechnology, and human-papillomavirus (HPV) vaccinations for schoolgirls).
2016] “Ideology” or “Situation Sense”? 377
contested policy,107 risk,108 and legal issues. 109 They also have been shown to
display psychometric properties superior to other latent-disposition scales
commonly used to study public risk perceptions.110
Figure 1: Cultural Cognition of Risk
Note: “Cultural cognition of risk” refers to the tendency of individuals to form perceptions of risk that reflect and
reinforce their commitments to affinity groups whose members share values that can be characterized along two
orthogonal dimensions: “hierarchy–egalitarianism” and “individualism–communitarianism.”111
107 See John Gastil, Don Braman, Dan Kahan & Paul Slovic, The Cultural Orientation of Mass
Political Opinion, 44 PS: POL. SCI. & POL. 711 (2011) (concluding that cultural worldview scales
display more variance among respondents and are less sensitive to differences in political
sophistication on policy issues such as gun control, universal health care, elimination of estate taxes,
and restricting carbon emissions).
108 See Kahan, Peters, Wittlin, Slovic, Ouellette, Braman & Mandel supra note 16, at 732-35
(describing a study finding that cultural worldviews have greater explanatory power than political
measures for climate change and nuclear power risk perceptions).
109 See Kahan, Hoffman, Braman, Evans & Rachlinski, supra note 46, at 884 n.118 (finding that
cultural worldviews are more discerning of biased perceptions of fact in legal controversy).
110 See Wen Xue, Donald W. Hine, Natasha M. Loi, Einar B. Thorsteinsson, & Wendy J.
Phillips, Cultural Worldviews and Environmental Risk Perceptions: A Meta-Analysis, J. ENVTL.
PSYCHOL., Dec. 2014, at 249, 257.
111 See id.; Dan M. Kahan & Donald Braman, Cultural Cognition and Public Policy, 24 YALE L.
& POL’Y REV. 149, 151-57 (2006).
378 University of Pennsylvania Law Review [Vol. 164: 349
lawyer lawyer
student judge
student
judge public
public
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
Egalitarianism Hierarchy Communitarianism Individualism
Note: Distributions of subject types’ worldviews. Scores are standardized: 0 is the sample mean, and
units are standard deviations from the mean. Subject type explained 7% of the variance in the
hierarchy–egalitarianism scale and 9% of the variance in the individualism–communitarianism scale.
a. Littering
The first case, “Littering,” involved a statutory provision forbidding
“littering, disposing, or depositing any form of garbage, refuse, junk, or other
debris” on the grounds of any national wildlife preserve.115 The defendants
had admittedly left unattended in such a preserve—a portion of desert along
the United States–Mexico border—400 ten-gallon reusable plastic water
dispensers, which they intended to be used and periodically refilled. The
subjects were instructed to play the role of a trial court judge ruling on a
motion to dismiss a civil penalty action against the defendants. As spelled out
clearly in the case vignette, proper disposition of the motion turned on whether
the defendants’ conduct could be deemed “depositing . . . junk” or “debris.”
113 The statutory interpretation problems and other components of the study instrument appear
in Appendix B.
114 See supra notes 36–37 and accompanying text.
115 Cf. Disposal of Waste, 50 C.F.R. § 27.94(a) (2014) (containing slightly different language).
2016] “Ideology” or “Situation Sense”? 381
116 Rayner, supra note 104; Kahan & Braman, supra note 111.
117 See Dan Kahan, U.S. Risk-Perception/Polarization Snapshot, CULTURAL COGNITION
PROJECT AT YALE L. SCH. (Dec. 30, 2011, 10:00 AM), http://www.culturalcognition.net
/blog/2011/12/30/us-risk-perceptionpolarization-snapshot.html [http://perma.cc/565X-P86V].
382 University of Pennsylvania Law Review [Vol. 164: 349
b. Disclosure
The second case, “Disclosure,” involved a familiar “mistake of law” issue.
The statute in Disclosure made it a crime for a government official to
“knowingly violate” a prohibition on “disclosing” to the public “confidential
investigatory information relating to an identifiable private citizen.” The
defendant, a police officer, admittedly disclosed to a private party what the
officer knew to be “confidential investigatory information.” Nevertheless, he
claimed not to “know” that he was “violat[ing]” any statutory prohibition.
His ignorance of the prohibition, the vignette made clear, would not furnish
the officer a defense if the term “knowingly” was construed to relate only to
the confidential nature of the information, to its release, and to the status of
the information recipient as a member of the public rather than another
government official; if, in contrast, “knowingly” were construed to relate in
addition to the existence of the prohibition on disclosing such information,
then the defendant officer’s ignorance would be a defense.118 The subjects
were instructed to evaluate these alternative readings of the statute for the
purpose of ruling on the defendant’s appeal of the trial court’s refusal to
instruct the jury that unawareness of the legal prohibition on disclosure would
in fact defeat proof of a crime.
The experimental manipulation in this case concerned the identity of the
party to whom the defendant had made the disclosure. In the “pro-choice
118 Compare United States v. Int’l Minerals & Chem. Corp., 402 U.S. 558, 563 (1971) (holding
that the term “knowingly” applies only to the defendant’s mental state regarding her conduct, not
to her mental state regarding the regulation’s existence), with Liparota v. United States, 471 U.S.
419, 432 (1985) (holding that “in a prosecution for violation of [7 U.S.C.] § 2024(b)(1) [(1982)] the
Government must prove that the defendant knew that his acquisition or possession of food stamps
was in a manner unauthorized by statute or regulations”).
2016] “Ideology” or “Situation Sense”? 383
center” version of the problem, subjects were advised that the officer had
leaked the investigatory information to “a noncommercial ‘family planning’
center that provides free information on birth control and abortion services”; the
information supplied by the defendant police officer was that an individual
known to police to belong to an anti-abortion group had applied to the center
for employment under false pretenses. In the “pro-life center” version of the
problem, study subjects were advised that the information recipient was a
“religious ‘family planning’ center that counsels women on alternatives to
abortion”; the information was that an individual known to police to belong
to a pro-choice group had applied to the center for employment without
disclosing this part of his background.
Of no consequence to the “mistake of law” issue in the case, the
manipulation of the information-recipient’s identity in Disclosure was
expected to provoke identity-protective cognition in individuals who are
either relatively hierarchical and communitarian, on the one hand, or
relatively egalitarian and individualistic, on the other. The former adhere to
social norms that assign individuals role-based obligations (e.g., “mother” or
“religious adherent”) to contribute to the wellbeing of one or another
collective entity (e.g., “family,” or “church”) that is itself rich with status-
defined obligations. The latter, in contrast, chafe at distinctions in status and
authority that do not originate in voluntarily assumed private agreements,
and reject, too, the idea that individuals have unchosen obligations to
subordinate their own well-being to the interests of any collective entity.
The legal right to abortion is an issue that tends to divide Hierarchical
Communitarians and Egalitarian Individualists (particularly female ones119).
The former see abortion “at will” as devaluing the social status of women who
successfully occupy matriarchal roles; the latter view legal protection of the
“right to choose” as a token of society’s commitment to assuring that
individual women, just like individual men, should be afforded esteem for
mastering market and professional roles.120 Again, in line with cultural
cognition, people with these outlooks tend to credit or dismiss asserted
abortion procedure health risks in patterns reflecting the opposing cultural
meanings that such individuals attach to abortion rights.121
The identity-manipulation in Disclosure was aimed at varying the
identity-protective stake that individuals with these cultural worldviews
119 See generally KRISTIN LUKER, ABORTION AND THE POLITICS OF MOTHERHOOD (1984).
120 Id. at 158-92 (discussing the different worldviews of pro-choice and pro-life individuals).
121 See Dan M. Kahan, Donald Braman, John Gastil, Paul Slovic & C. K. Mertz, Culture and
Identity-Protective Cognition: Explaining the White-Male Effect in Risk Perception, 4 J. EMPIRICAL
LEGAL STUD. 465, 489-491 (2007).
384 University of Pennsylvania Law Review [Vol. 164: 349
would form in the outcome of the Disclosure case. In the “pro-life version”
of the problem, Hierarchical Communitarians, we surmised, would
experience an unconscious motivation to extend a defense to the police
officer: such an outcome, we reasoned, would affirm their worldview by
exonerating from criminal censure a state official who acted to protect a pro-
life family counseling center from subversion. Precisely because it treats such
an actor as unworthy of legal denunciation, Egalitarian Individualists, we
predicted, would be motivated to deny the officer the defense in that version
of the problem. These positions would be reversed in the “pro-choice
version”: legally condemning the officer for tipping off the pro-choice family-
counseling center would be identity-affirming for Egalitarian Individualists,
we surmised, and identity-denigrating for Hierarchical Communitarians.122
122 Cf. Kahan, Hoffman, Braman, Evans & Rachlinski, supra note 46 (finding that Hierarchical
Communitarians and Egalitarian Individualists polarized on perception of facts after watching a
video of a protest described as either an anti-abortion demonstration or demonstration protesting
discrimination on the basis of sexual orientation).
123 See Dan M. Kahan, Climate-Science Communication and the Measurement Problem, 36
ADVANCES IN P OL. PSYCHOL. 1, 8-9 (2015) (finding that risk perception of climate change is
correlated with the belief that human activity is its cause).
124 See Yoav Ganzach, Shmuel Ellis, Asya Pazy & Tali Ricci-Siag, On the Perception and
Operationalization of Risk Perception, 3 JUDGMENT & DECISION MAKING 317 (2008); Elke U.
Weber, Ann-Renée Blais & Nancy E. Betz, A Domain-Specific Risk-Attitude Scale: Measuring Risk
Perceptions and Risk Behaviors, 15 J. BEHAV. DECISION MAKING 263 (2002).
2016] “Ideology” or “Situation Sense”? 385
Items included in the risk perception battery were selected on the basis of
previous studies that had established them to be ones that trigger identity-
protective cognition in relation to the identities associated with the cultural
cognition worldview scales. We thus anticipated that the environmental risk
perception scale would divide subjects inclined toward an egalitarian–
communitarian worldview from ones inclined toward a hierarchical–
individualistic one. We expected the social-deviancy risk scale to divide
subjects inclined toward egalitarian-individualist and hierarchical-
communitarian worldviews, respectively.125
B. Hypotheses
1. Four Contenders
As indicated, the rationale for the design of the statutory interpretation
problems and the selection of risk-battery items was the expected impact they
would have in triggering identity-protective cognition. The central aim of
the study, however, was to test whether professional judgment of the sort
generally exercised by judges and lawyers would counteract this species of
motivated reasoning. Specifically, the array of problems, combined with the
mix of different types of study subjects, was geared to assessing the relative
plausibility of four distinct hypotheses.
a. Universal Vulnerability
One of the hypotheses was that the status of the subject—member of the
public, law student, lawyer, or judge—would make no difference. All of them
would display the same vulnerability to identity-protective reasoning in both
the legal-problem and risk-perception response measures. We will call this
the Universal Vulnerability (UV) hypothesis. Results consistent with UV
would vindicate the dominant scholarly view that judges are indeed
“ideologically biased”—or “politicians in robes.”
Indeed, corroboration of UV would help to reinforce the two main pillars
of research now thought to support the view that judicial decisionmaking is
“ideological.” Such results would constitute more persuasive grounds for
crediting the view that judges are politically biased than existing identity-
protective cognition studies, which, as indicated, attribute to judges
decisionmaking biases observed in general population samples. Findings
125 See generally Dan M. Kahan & Donald Braman, Cultural Cognition of Public Policy, 24 YALE
J.L. & PUB. POL’Y 147, (2006); Aaron Wildavsky & Karl Dake, Theories of Risk Perception: Who
Fears What and Why?, DAEDALUS, Fall 1990, at 41, 44.
386 University of Pennsylvania Law Review [Vol. 164: 349
c. Domain-Specific Immunity
A more plausible alternative to UV is what we will call the “domain-
specific immunity” (DSI) hypothesis. DSI predicts that lawyer and judge
members of the sample will display resistance to identity-protective cognition
but only in their responses to the legal-problem component of the study. In
lawyers and judges will uniformly agree on the outcomes of the Littering and
Disclosure problems, although one might expect that they will be largely in
agreement. It predicts only that differences of opinion among the lawyer and judge
study participants will not reflect differences in those judge’ cultural worldviews.
But again, DSI predicts that judges and lawyers will reliably converge on
decisionmaking factors independent of their cultural outlooks only in the
legal-problem component of the study because only that component of the
study features the exercise of legal professional judgment.134 The outcome
most consistent with DSI, then, would be one in which judges and lawyers
resist identity-protective cognition only with respect to the legal-problem
component and not the risk-perception component.
d. Acquired Impartiality
A final conjecture, which we call the Acquired Neutrality (AN)
hypothesis, predicts that only judges and not lawyers will display resistance
to identity-protective reasoning. On this account, what confers judges’
immunity to identity-protective reasoning is not the training and experience
common to membership in the legal profession, but instead the habitual,
willed exercise of neutrality that is peculiar to the task of judging.
Conceivably, this acquired immunity to identity-protective reasoning might
apply across all manner of domain; or alternatively, it might be more limited,
negating identity-protective cognition in judges only when they are engaged
in the form of reasoning that they use to decide cases. Under the former view,
AN predicts that judges, but not lawyers, will resist identity-protective
cognition in their response to both the legal-problem portion of the study and
the risk-perception portion. Under the latter, AN predicts judges alone will
avoid the influence of unconscious partisan reasoning, but only in the legal-
problem component of the study.
2. Law Students
What about law students? Obviously, if UV is correct, we should expect
them, like lawyers and judges and members of the public generally, to display
identity-protective reasoning in their responses to all of the study outcome
measures. The same would be true under AI.
134 See Dane & Pratt, supra note 85, at 43 (“Because complex schemas develop in a particular
domain (one’s area of expertise), they are more likely to lead to effective decisions in that domain
than when used in a different domain or context. Thus, complex managerial schemas may serve a
manager well at the office but may lead to inaccurate intuitive judgments at home.”).
2016] “Ideology” or “Situation Sense”? 389
ISI and DSI, however, do not necessarily imply that law students will
resist identity-protective cognition either. Students are at only an
intermediate stage of professionalization; their “situation sense,” one might
thus suppose, is imperfect. On this view, then, ISI and DSI could be
understood to predict that law students would display identity-protective
cognition, but to a smaller degree than members of the public generally.
The Redding–Reppucci study found that, while judges were influenced
by political commitments on a determination (the constitutionality of the
death penalty) that turned on normative commitments intrinsic to law, they
were unaffected by those commitments in making evidentiary rulings that
were analytically distinct from such commitments.135 The same was not true
of law students.136 As a result of that study’s design, there is ambiguity about
the inferences that can be drawn from the results.137 Nevertheless, a finding
in this study that the legal-problem responses of students, but not of judges,
display sensitivity to the cultural congeniality of the experimentally
manipulated case outcomes would supply reason to attribute the results in
Redding–Reppucci study to the power of professional judgment to
counteract unconscious political predispositions.
IV. RESULTS
A. Legal Reasoning
We consider first how the subjects responded to the legal-problem
component of the study. We start with summary analyses based on simple
tabulations of the responses of various subject types and worldviews in the
different versions of the two statutory interpretation problems. We then
present multivariate analyses designed to test the study hypotheses.
1. Summary Data
143 Psychologically speaking, the mistake of deriving the likelihood ratio or weight to be
assigned new evidence from one’s prior beliefs, or more generally from one’s willingness to assent
to the truth of the hypothesis, is confirmation bias. See generally STANOVICH, supra note 93;
Matthew Rabin & Joel L. Schrag, First Impressions Matter: A Model of Confirmatory Bias, 114 Q.J.
ECON. 37 (1999).
144 The conventional threshold for statistical power—a measure of the likelihood of observing
a posited effect size at a specified threshold of statistical significance, conditional on sample size—
is 0.80. See generally Stephen G. West, Jeremy C. Biesanz, & Steven C. Pitts, Causal Inference and
Generalization in Field Settings: Experimental and Quasi-Experimental Designs, in HANDBOOK OF
RESEARCH METHODS IN SOCIAL AND PERSONALITY PSYCHOLOGY 40, 53 (Harry T. Reis &
Charles M. Judd eds., 2000).
392 University of Pennsylvania Law Review [Vol. 164: 349
Public Students
100%
75%
50%
25%
0%
violation no violation violation no violation violation no violation violation no violation
Pro-choice center Pro-life center Pro-choice center Pro-life center
Lawyers Judges
100%
75%
50%
25%
0%
violation no violation violation no violation violation no violation violation no violation
Pro-choice center Pro-life center Pro-choice center Pro-life center
Note: Panels reflect the percentage of indicated type of subjects who selected indicated results in
specified versions of the Disclosure problem.
The situation was quite different, however, among both the lawyers and
the judges who participated in our study. Regardless of the version of
Disclosure they evaluated, decided majorities of both of these types of
subjects indicated that the defendant police officer had indeed violated the
statute (Figure 3). In Littering, decided majorities concluded that placing
unattended reusable water containers in the desert did not constitute
“discarding . . . debris” in a wildlife preserve—whether done by immigrant
aid or construction workers (Figure 4). Most lawyers and judges, then,
perceived something in each problem that guided them to a consensus
interpretation, notwithstanding the ambiguity of the statutory language.
2016] “Ideology” or “Situation Sense”? 393
The overall level of agreement among the law students was intermediate
between the ones observed among members of the public, on the one hand,
and among lawyers and judges, on the other. In Disclosure, the students, like
the lawyers and judges, were inclined to find a statutory violation in both the
“pro-life center” and “pro-choice center” versions (Figure 3).
Public Students
100%
75%
50%
25%
0%
violation no violation violation no violation violation no violation violation no violation
Immigrant aid Constr. workers Immigrant aid Constr. workers
Lawyers Judges
100%
75%
50%
25%
0%
violation no violation violation no violation violation no violation violation no violation
Immigrant aid Constr. workers Immigrant aid Constr. workers
Note: Panels reflect the percentage of indicated type of subjects who selected indicated results in
specified versions of the Littering problem.
75%
Hierarch individ
50%
Egal commun
Egal commun
25%
0%
Immigrant aid Constr. workers Immigrant aid Constr. workers
Lawyers Judges
100%
pct. finding violation
75%
Hierarch individ
50%
Hierarch individ
25%
Egal commun Egal commun
0%
Immigrant aid Constr. workers Immigrant aid Constr. workers
Note: Panels reflect the percentage of subjects of type and worldview who indicated support for
finding a violation in one or another version of the Littering problem.
Cultural divisions were more modest but still apparent among members
of the public in Disclosure (Figure 6). Egalitarian Individualist and
Hierarchical Communitarian subjects favored finding a violation of the
statute by narrow but roughly equal margins (52% to 48% and 53% to 47%,
respectively) in the “pro-choice center” version. But in the “pro-life center”
version, subjects with these identities diverged: whereas 60% of the
Egalitarian Individualists concluded the officer had violated the statute by
alerting the anti-abortion family-counseling center of its possible infiltration
by a pro-choice activist, only 39% of Hierarchical Communitarian subjects
did. While less dramatic than the effect in Littering, this disparity, too, fit
2016] “Ideology” or “Situation Sense”? 395
Public Students
100%
Egal Individ
pct. finding violation
75%
Egal Individ
Hierarch Commun
50%
Hierarch Commun
25%
0%
Pro-choice center Pro-life center Pro-choice center Pro-life center
Lawyers Judges
100% Egal Individ
Hierarch Commun
pct. finding violation
75%
Hierarch Commun Egal Individ
50%
25%
0%
Pro-choice center Pro-life center Pro-choice center Pro-life center
Note: Panels reflect the percentage of subjects of type and worldview who indicated support for
finding a violation in one or another version of the Disclosure problem.
2. Multivariate Regression
a. Generally
The impact of subject type, worldview, and experimental assignment was
probed more systematically with multivariate logistical regression analysis.
For each problem, a regression model was constructed to enable statistical
estimation of the probability that different subject types (member of the
public, law student, lawyer, or judge) would find a violation conditional on
the subject’s worldview and the version of the problem.145
Monte Carlo simulations based on the regression models were performed
to facilitate interpretation of the results. In a Monte Carlo simulation, the
regression model outcome variable is calculated and randomly adjusted by an
amount reflecting the measurement error associated with the model
parameters. This process is repeated a sufficient number of times to populate
the entire probability distribution for the outcome variable at specified values
of the model predictors.146 Using this technique, the probability that a
decisionmaker would find a violation was computed a thousand times for each
combination of subject type, cultural worldview, and experimental
assignment of interest (Appendix A, Figure A1, Figure A2).147 Differences in
the predicted probabilities of finding a violation conditional on worldview or
experimental assignment were determined in the same fashion.148
145 The regression analyses for the legal problems appear in Tables A1 and A2 in Appendix A.
146 See ANDREW GELMAN & JENNIFER HILL, DATA ANALYSIS USING REGRESSION AND
MULTILEVEL/HIERARCHICAL MODELS 137-51 (2007).
147 The continuous “hierarchy–egalitarianism” and “individualism–communitarianism”
predictors were set at +1 SD; +1 SD for “Hierarchical Individualist”; +1, -1 for “Hierarchical
Communitarian”; -1, +1 for “Egalitarian Individualist”; and -1, -1 for “Egalitarian Communitarian.”
148 See Gary King, Michael Tomz, & Jason Wittenberg, Making the Most of Statistical Analyses:
Improving Interpretation and Presentation, 44 AM. J. POL. SCI. 347, 349-353 (2000).
398 University of Pennsylvania Law Review [Vol. 164: 349
public and for a judge, respectively. There are separate estimates for each
problem and for the “average effect” based on responses to both problems.149
Public
Judge
149 The “average” effect was derived consistently with the methods prescribed in Robert
Rosenthal & Donald B. Rubin, Meta-Analytic Procedures for Combining Studies with Multiple Effect
Sizes, 99 PSYCHOL. BULL. 400 (1986), for aggregating the effect sizes of multiple single-study
measures of a single phenomenon of interest—here the disposition of different subject types to
display identity-protective cognition.
2016] “Ideology” or “Situation Sense”? 399
150 For this purpose, we are treating as an “identity-protective cognition effect” a difference in
probability of finding a violation in the direction corresponding to the hypothesized effect of the
experimental manipulation. In Figure 7, differences in that direction have a positive value;
differences in direction contrary to the hypothesized effect of identity-protective cognition, in
contrast, have a negative value.
151 See Lee Epstein, Andrew D. Martin, & Matthew M. Schneider, On the Effective
Communication of the Results of Empirical Studies, Part I, 59 VAND. L. REV. 1811, 1832-33 (2006); King,
Tomz & Wittenberg, supra note 148.
152 All confidence intervals reported in the text hereafter will reflect a 0.95 level of confidence.
153 Because each study problem constituted an indirect measure of an unobserved or latent
disposition—the propensity to process information in a manner that reflects identity-protective
cognition—the aggregated or “average” effect is more precise than either individual measure, and
thus has a smaller standard error. See Rosenthal & Rubin, supra note 149.
400 University of Pennsylvania Law Review [Vol. 164: 349
154 The negative values suggest that the “best estimate” of the effects of the judges’ cultural
outlooks on their decision making were in fact the opposite from what one would expect if judges
had been influenced by identity-protective cognition.
155 See generally Rivka M. de Vries & Richard D. Morey, Bayesian Hypothesis Testing for Single-
Subject Designs, 18 PSYCHOL. METHODS 165 (2013); Goodman, supra note 140; John K. Kruschke,
Bayesian Estimation Supersedes the t Test, 142 J. EXPERIMENTAL PSYCHOL. 573 (2013).
156 See, e.g., Lempert, supra note 141.
2016] “Ideology” or “Situation Sense”? 401
have drawn a red marble from B2. The joint probability of these independent
events—that is, the probability of the two occurring together, as they did—
is 3/4 x 3/4 or 9/16. If we assume that the hypothesis “B1 is the one that
contains mostly blue marbles” is false, then the joint probability of drawing a
blue marble from B1 followed by a red marble from B2 would be 1/4 x 1/4, or
1/16. Other possible combinations of colors could have occurred, of course
(indeed, there are four possible combinations for such a trial). But if we were
to repeat this “experiment” over and over (with the marbles being replaced
and the labels on the bags being randomly reassigned after each trial), then
we would expect the sequence “blue, red” to occur nine times more often
when the bag containing mostly blue marbles is the one labeled “B1” than
when it is the bag labeled “B2.” Because “blue, red” is the outcome we
observed in our trial, we should revise our estimate of the probability of the
hypothesis “B1 contains mostly blue marbles” by a factor 9—from odds of 1:1
(50%) to 9:1 (90%).
We can use precisely the same logic to assess the relative probability of
hypothesized judge and public IPCIs. In effect, one can imagine each subject
type as an opaque vessel containing some propensity to engage in identity-
protective cognition. Although the strengths of those propensities—the
subject types’ “true” IPCIs—are not amenable to direct inspection, we can
sample observable manifestations of them by performing this study’s
statutory interpretation experiment. Calculating the relative likelihood of the
observed results under competing hypotheses, we can construct a likelihood
ratio that conveys how much more consistent the evidence is with one
hypothesized subject type IPCI than with another.
Figure 8 illustrates the use of this method to test two competing
hypotheses about the public’s “true” IPCI: that members of the public would
be 25% more likely to find a violation when doing so is culturally affirming,
and alternatively that they would be only 15% more likely to do so. To make
the rival hypothesis commensurable with the study results, we can represent
each as a probability distribution with the predicted IPCI as its mean and a
standard error equivalent to the one observed in the experimental results.
Within any one such distribution, the relative probability of alternative
IPCIs (e.g., 15% and 25%) can be determined by assessing their relative
“heights” on that particular curve.157 Likewise, the relative probability of
observing any particular IPCI under alternative distributions can be
157 The units that appear on the y-axis are in fact completely irrelevant for this purpose.
Consistent with convention, we compare probability densities, which are the first derivative of (rate
of change in) cumulative probability distribution associated with the logistic regression function
that generates the underlying probabilities.
402 University of Pennsylvania Law Review [Vol. 164: 349
determined by comparing the ratio of the heights for the probability density
distributions in question.158
15
Hypothesis 2:
Hypothesis 1: IPCI = 25%
IPCI = 15%
10
observed
public IPCI
1
0
0 .1 .2 .3 .4
The public IPCI was 22%. The probability of observing such a result (or
any in close proximity to it) is eight times more likely under the more extreme
“public IPCI = 25%” hypothesis than it is under the more modest “public
IPCI = 15%” hypothesis (Figure 8).159 This is the Bayesian likelihood ratio,
or the factor in proportion to which one should modify one’s assessment of
the relative probability that the “true” public IPCI is 25% as opposed to 15%.
158 See Steven N. Goodman, Toward Evidence-Based Medical Statistics. 2: The Bayes Factor, 130
ANNALS INTERNAL MED. 1005 (1999).
159 See generally Goodman, supra note 140.
2016] “Ideology” or “Situation Sense”? 403
We will use the same process to assess the weight of four competing
hypotheses about the vulnerability of judges to identity protective cognition.
The first is that judges will be “unaffected” (IPCI = 0). This prediction, of
course, appears similar to the “null hypothesis.” But whereas “null hypothesis
testing” purports to specify only whether the null hypothesis can be rejected,
Bayesian methods can be used to obtain a genuine assessment of the strength
of the evidence in support of there being “no effect” if that is a genuine
hypothesis of interest, as it is here.160 The remaining three hypotheses, the
plausibility of which will be tested relative to the “IPCI = 0” hypothesis are
that judges will be “just as affected as the public” (IPCI = 22%); that judges
will be moderately affected (IPCI = 10%); and that judges will be affected to
only a comparatively mild degree (IPCI = 5%).161
The results are reflected in Figure 9. Not surprisingly, the experimental
data are much more supportive of the first hypothesis—that judges would be
unaffected by the experimental manipulation—than with the second—that
they would be “as affected as much as the public.” Indeed, because the
probability that we would have observed the actual experimental result if the
latter hypothesis is true is astronomically low, there is little practical value in
assigning a likelihood ratio to how much more strongly the evidence supports
the hypothesis that judges were “unaffected” by the experimental
manipulation.
5 5 5
1.8
.25
0 0
-.3 -.2 -.1 0 .1 .2 .3 .4 .5 -.3 -.2 -.1 0 .1 .2 .3 -.3 -.2 -.1 0 .1 .2 .3
160 See Kruschke, supra note 155, at 577; Richard D. Morey & Jeffrey N. Rouder, Bayes Factor
Approaches for Testing Interval Null Hypotheses, 16 PSYCHOL. METHODS 406 (2011).
161 See generally Goodman, supra note 158.
404 University of Pennsylvania Law Review [Vol. 164: 349
Note: The probability distributions reflecting the indicated hypotheses were constructed using the
standard error (0.06) associated with the observed Judge IPCI in the experiment. The horizontal
line intersecting the probability distributions is the observed judge IPCI (-5%).
Lawyer
The evidence for the students is more equivocal. In both problems, the
predicted student IPCI was greater than zero, and on average it was 12%
(± 14%). This effect—the difference in how likely a student decisionmaker is
to find a violation when that outcome is identity-affirming rather than
identity-denigrating—is not statistically significant at “p < 0.05”; it is
significant at only “p = 0.10.” But the difference between the student and
public IPCI’s was not statistically significant at p < 0.05 either (10%, ± 15%;
p = 0.20). Accordingly, if one uses “null hypothesis testing” criteria, one can
reject neither the hypothesis that students were unaffected by identity-
protective cognition nor the hypothesis that they were affected just as much
as members of the public—who by this same mode of assessment can be deemed
to have been strongly influenced by this same form of bias (22%, ± 6%).
162 See generally King, Tomz, & Wittenberg, supra note 148.
406 University of Pennsylvania Law Review [Vol. 164: 349
Note: The probability distributions reflecting the indicated hypotheses were constructed using the
standard error (0.07) associated with the observed student IPCI in the experiment. The horizontal
line intersecting the probability distributions is the observed student IPCI (12%).
very close to one, the experimental results are effectively equally consistent
with the “lawyer IPCI = 0” and “lawyer IPCI = 5%” hypotheses.
observed data >100 x more observed data 2x more observed data equally
consistent with consistent with IPCI = 0 consistent with IPCI = 0 as
IPCI = 0 than with IPCI = “as than with IPCI = 10% with IPCI = 5%
big as public”
H1: H2: H1: H3: H1: H4:
IPCI = 0 IPCI = 0.22 IPCI = 0 IPCI = 0.10 IPCI = 0 IPCI = 0.05
6 6 6
5.6 5.6 5.6
5.2
2.8
0 0 0
-.25 -15 -.05 .05 .15 .25 .35 .45 -.3 -.2 -.1 0 .1 .2 .3 -.3 -.2 -.1 0 .1 .2 .3
Identity protective cognition impact
Note: The probability distributions reflecting the indicated hypotheses were constructed using the
standard error (0.07) associated with the observed lawyer IPCI in the experiment. The horizontal
line intersecting the probability distributions is the observed lawyer IPCI (2%).
judge
-5%, ± 12% 3.9
lawyer “AI””
2%, ± 14% 3.15 IPCI∆ = 0.10
“DSI”:
2 IPCI∆ = 0.00
0
-.3 -.2 -.1 0 .1 .2 .3 -.5 -.4 -.3 -.2 -.1 0 .1 .2 .3 .4 .5
Identity protective cognition impact IPCI differential
Note: The left-hand panel juxtaposes probability density distributions for predicted lawyer and judge
IPCIs (see supra Figures 7 & 10). The probability distributions reflecting the indicated hypotheses
were constructed using the standard error (0.10) associated with the observed difference in judge
and lawyer IPCIs.
This point is illustrated in Figure 13, which juxtaposes the probability dis-
tributions for two competing hypotheses about the size of the difference be-
tween the judge and lawyer IPCIs: 0% vs. 10%. The meager 1.2 likelihood
ratio in favor of the latter hypothesis signifies that the evidence in support of
it is only trivially greater than the evidence in support of the former “equal
IPCI” hypothesis.
B. Risk Perceptions
As indicated, the subjects responded to a battery of items measuring their
perceptions of various societal risks. These were combined to form separate
“environmental risk” and “social deviancy risk” scales.163
public public
student student
lawyer lawyer
judge judge
Note: The bars reflect standardized (z-score) means of indicated risk perceptions for subjects holding
indicated worldviews (as determined by relationship of their scores to means on the Hierarchy–
Egalitarianism and Individualism–Communitarianism scales). Error bars reflect 0.95 level of
confidence for “true mean.”
164 See generally Kahan & Braman, supra note 111; Wildavsky & Dake, supra note 125.
410 University of Pennsylvania Law Review [Vol. 164: 349
V. TAKING STOCK
168 Remarkably, some “ideology thesis” proponents use information on judges’ voting behavior
to characterize judges as “liberal” or “conservative” and then use the resulting measures to “test”
hypotheses about “ideological voting” by those judges in the very cases from which their “ideology”
classifications or scores were derived. See EPSTEIN, POSNER & LANDES, supra note 5, 113-16 & n.13.
169 See, e.g., Guthrie, Rachlinski & Wistrich, supra note 75, at 27-28.
170 See supra notes 77–79 and accompanying text.
171 See generally Guthrie, Rachlinski & Wistrich, supra note 75.
2016] “Ideology” or “Situation Sense”? 413
That law students would not enjoy the same capacity to resist identity-
protective reasoning as judges and lawyers is not surprising and is consistent
with the Domain-Specific Immunity hypothesis. Students enjoy an immature
form of the professional judgment that fully trained and experienced lawyers
possess. It stands to reason, then, that culturally diverse students would
display less convergence in their assessment of culturally fraught legal
problems.
Indeed, our finding that students are less resistant to politically motivated
reasoning than are judges and lawyers corroborates one important component
of the Redding–Reppucci study and helps to put the full results of that study
in perspective. As we did, Redding and Reppucci found that students, but
not judges, were inclined to conform legal rulings—in their study, the
admissibility of evidence on the deterrent efficacy of the death penalty—to
their political outlooks.172
Redding and Reppucci also found that both judges and students tended
to resolve constitutional challenges to capital punishment in a manner
consistent with their political values, regardless of whether the evidence
challenged their prior views on capital punishment’s deterrent efficacy.173 But
as we have explained,174 that legal issue itself requires judges to make
normative judgments. Normative judgments, unsurprisingly, will vary across
judges of differing political outlooks. Insofar as the role of those judgments
in legal decisionmaking is intrinsic to law, however, it is not valid to treat any
correlation between legal rulings of that sort and decisionmakers’ political
outlooks as evidence of “politically” or “ideologically” biased reasoning.175
For it to be important—indeed, for it to be analytically coherent—the
claim that judges are “politically biased” requires demonstrating the
responsiveness of their rulings to political outlooks extrinsic to the legal issues
they are considering.176 The statutory interpretation rulings of the judges in
our study were not responsive to such outlooks—just as the evidentiary
rulings of the judges in the Redding–Reppucci study were not. That the legal
reasoning of law students, in our study and theirs, were not immune to this
bias furnishes reason to view both studies as evidence that professional
judgment contributes to neutralizing identity-protective cognition on judicial
decisionmaking.
It is not particularly surprising that this topic has not received scholarly
attention. The impact of identity-protective reasoning is not nearly so
significant for understanding and perfecting professional judgment as are the
effects of the various other cognitive dynamics featured in the decision
science literature. Recognizing that auditors, say, have a stake in forming
perceptions of societal risk that protect their status in their cultural groups is
unlikely to contribute much to assessing the role that prototypical reasoning
plays in their detection of irregularities in financial records.181 The prospect
that doctors will misdiagnose a disease because of base-rate neglect or
coherence-based reasoning, likewise, is more critical to assessing the
proficiency of physicians than is figuring out if they are likely to be
unconsciously motivated to selectively credit and dismiss data on climate
change in a manner that reflects their worldviews. The relevance of identity-
protective cognition to professional judgment is much more conspicuous in
law, where assuring that facts are determined, and legal rules administered,
in a manner that is neutral as between competing cultural understandings of
the best way to live is the very form of expertise that a judge is required to
exercise.
But there is at least one other group of experts whose vulnerability to
identity-protective cognition has become an issue of speculation: scientists
who investigate risks and related facts that excite cultural polarization.
Sensibly, citizens tend to treat “scientific consensus” on environmental risk
and other highly technical matters as a reliable normative guide for
decisionmaking, collective and individual.182 But what makes it sensible for
them to do so is that the method of inquiry that scientists themselves use
does not afford existing “scientific consensus” any particular weight. On the
contrary, the entitlement of any previously supported proposition to
continued assent is, for science, conditional on its permanent amenability to
reexamination and revision in light of new evidence.183
If, then, there were reason to believe that scientists themselves were being
unconsciously motivated to discount evidence challenging “consensus”
positions on issues like climate change, say, or nuclear power or genetically
modified foods, by their cultural outlooks, that would be a reason for treating
181 Cf. Bedard & Biggs, supra note 88 (showing that hypothesis creation in auditor analysis is
not related to an auditor’s particular worldview).
182 See generally Dan M. Kahan, Hank Jenkins-Smith, & Donald Braman, Cultural Cognition of
Scientific Consensus, 14 J. RISK RES. 147, 147 (2011) (noting that both sides in culturally polarizing
debates over societal risks understand their group’s position to be consistent with “scientific
consensus”—but disagree about what “scientific consensus” is as a result of identity-protective
cognition).
183 See KARL P OPPER , T HE LOGIC OF SCIENTIFIC D ISCOVERY 40 (1959).
416 University of Pennsylvania Law Review [Vol. 164: 349
184 See, e.g., INTERGOVERNMENTAL PANEL ON CLIMATE CHANGE, CLIMATE CHANGE 2013:
THE PHYSICAL SCIENCE BASIS: WORKING GROUP I CONTRIBUTION TO THE FIFTH ASSESSMENT
REPORT OF THE INTERGOVERNMENTAL PANEL ON CLIMATE CHANGE (2013).
185 See, e.g., N ATIONAL RESEARCH C OUNCIL, E COLOGICAL I MPACTS OF C LIMATE
CHANGE (2008).
186 See, e.g., ROYAL SOCIETY , P REVENTING D ANGEROUS C LIMATE C HANGE (2009).
187 See, e.g., Judith Curry, Scientists and Motivated Reasoning, CLIMATE ETC. (Aug. 20, 2013),
http://judithcurry.com/2013/08/20/scientists-and-motivated-reasoning [http://perma.cc/MMA7-KVKF].
188 See supra notes 88–99 and accompanying text.
2016] “Ideology” or “Situation Sense”? 417
sources of disruption create fissures between expert and lay judgment and
resulting forms of public conflict over environmental risk.
Identity-protective cognition can be understood to be a disrupting
influence of this character. When a fact subject to expert judgment (Is the
earth heating up and are humans causing that? Does permitting citizens to
carry handguns in public make crime rates go up or down? Does the HPV
vaccine protect adolescent girls from a cancer-causing disease—or lull them
into sexual promiscuity that increases their risk of pregnancy and other
STDs?) becomes entangled in antagonistic cultural meanings, positions on
that fact can become transformed into badges of membership in and loyalty
to opposing groups. At that point the stake people have in protecting their
status in their group will compete with, and likely overwhelm, the one they
have in forming perceptions that align with expert judgments.189
As we have noted,190 there is a striking affinity between the account
Margolis gives of pattern recognition in expert judgment among scientists
and other professionals and Karl Llewellyn’s account of “situation sense” as a
professionalized recognition capacity that enables lawyers and judges to
converge on appropriate legal outcomes despite the indeterminacy of formal
legal rules. We would surmise, based on this study and previous ones,191 a
parallel account of public conflict over judicial decisions.
Lacking lawyers’ “situation sense,” members of the public will not reliably
be able to make sense of the application of legal rules. But members of the
public will presumably have acquired lay prototypes that enable them, most
of the time anyway, to recognize the validity of legal decisions despite their
own inability to verify their correctness or comprehend their relationship to
relevant sources of legal authority.
But just like their capacity to recognize the validity of scientific expert
judgments, the public’s capacity to recognize the validity of expert legal
determinations will be vulnerable to conditions that excite identity-protective
reasoning. When that happens, culturally diverse citizens will experience
disagreement and conflict over legal determinations that do not generate such
disagreement among legal decisionmakers.192
This was the basic theoretical account that informed our study. It was the
basis for our prediction that judges, as experts possessing professional
189 See Dan Kahan, Why We Are Poles Apart on Climate Change, 488 NATURE 255 (2012).
190 See supra notes 88–99 and accompanying text.
191 See generally Kahan, Hoffman & Braman, supra note 46 (studying perceptions of a police
chase, and finding that differences in cultural and social groups can have significant effects on
individuals’ perceptions); Kahan, Hoffman, Braman, Evans & Rachlinski, supra note 46 (showing
evidence for cultural cognition affecting individual perceptions of the speech–conduct distinction).
192 See generally Kahan, Neutral Principles, Motivated Cognition, supra note 9.
418 University of Pennsylvania Law Review [Vol. 164: 349
193 Cf. Toby Bolsen, James N. Druckman & Fay Lomax Cook, Citizens’, Scientists’, and Policy
Advisors’ Beliefs About Global Warming, ANNALS AM. ACAD. POL. & SOCIAL SCI., March 2015, at
271 (reporting data suggesting that scientists are less culturally polarized than members of public on
climate change); J. S. Carlton, Rebecca Perry-Hill, Matthew Huber & Linda S. Prokopy, The
Climate Change Consensus Extends Beyond Climate Scientists, ENVTL. RES. LETTERS (Sept. 24, 2015),
http://iopscience.iop.org/article/10.1088/1748-9326/10/9/ 094025/ pdf [http://perma.cc/L246-
CBEE] (same).
2016] “Ideology” or “Situation Sense”? 419
194 See supra Section I.A. This study also underscores the unreliability of treating imaginative
extrapolation from decision-science research involving the general public as a valid method for
determining the vulnerability of judges to other biases that might constrain their effectiveness in
performing tasks such as applying rules of evidence. This method is useful for generating hypotheses
worthy of testing. See, e.g., Frederick Schauer, On the Supposed Jury-Dependence of Evidence Law, 155
U. PA. L. REV. 16 (2006) (discussing whether judges should limit themselves to the same evidence
as juries, and arguing that testing should be done to determine whether judges fall prey to the same
problems of bias that juries do). But when held forth as an “explanation” supported by “scientific
evidence,” this form of exposition confuses story-telling plausibility with empirical proof.
420 University of Pennsylvania Law Review [Vol. 164: 349
predictably divide the public. Their job is to decide those sorts of cases
neutrally, and our evidence supports the inference that they have both the
capacity and disposition to carry it out.
That such a result defies public perceptions should not come as any sort
of surprise. Numerous studies have found that members of the general public
themselves can be expected to conform their assessments of evidence and
their interpretation of rules to the stake they have in legal outcomes that
affirm the status of their groups and their own standing within them.195 These
studies, we have emphasized, are not a reliable basis for drawing inferences
about the in-domain reasoning processes of judges. But the one sort of
inference that they do support is that members of the public can be expected
to perceive judges to be biased in cases the outcomes of which are invested
with antagonistic cultural meanings even when the outcomes of those cases
reflect neutral decisionmaking.196
That conclusion is, in fact, the bad news associated with our study results:
the reliable convergence of culturally diverse judges on genuinely neutral
outcomes has no connection at all to how untrained members of the public
perceive the neutrality of those judges’ decisions. Again, because citizens lack
the elements of professional judgment—the “situation sense”—that lawyers
and judges acquire through their training and experience, citizens do not have
the capacity to discern those aspects of the case and the governing legal rules
pertinent to assessing the neutrality or validity of judicial resolutions of them.
On the contrary, in precisely those cases in which public anxiety about the
cultural neutrality of the law is likely to be highest, identity-protective
cognition will predictably disable members of the public from using their
usually reliable lay prototypes of valid decisionmaking to assess cases
outcomes. In that circumstance, no matter how expertly and impartially
judges decide, the sense of the public—or at least those who belong to the
cultural group whose identity is denigrated by the decision—will be disposed
to see judges’ decisions as “politically biased.”197
This problem is exactly parallel to the one that scientists face when
empirical issues on which they possess expertise become entangled in
culturally contested meanings. Obviously, doing valid science does not in
itself communicate the validity of scientific research: people lack the expertise
to see validity for themselves; they must rely on cues and processes that help
them to reliably recognize who knows what about what.198 The capacity of
members of the public to interpret those cues is compromised when
propositions of risk or fact become symbols of the status of competing
cultural groups.199 In that sort of “polluted science-communication
environment,”200 just doing valid science—including the part of valid science
that consists of communicating validity to other scientists—will do nothing to
silence public confusion and agitation.
Fixing this science communication problem is the aim of a new science of
science communication.201 This subdivision of decision science uses empirical
methods to identify the various dynamics that enable people to recognize as
valid scientific insights that they could never verify for themselves. It also
aims to understand, empirically, how those processes can be disrupted, and
how society can effectively preempt such disruptions and counteract them
when strategies of prevention fail.
Exploiting the benefits of the science of science communication will
demand appropriate adjustments to myriad institutional practices. The sorts
of conscious interventions necessary to protect the science communication
environment from contamination are not self-executing. An integral part of
the science of science communication, then, is to identify programs of
implementation that appropriately reconfigure the processes for science-
informed policymaking, the norms of science-generating and -consuming
professions, and the structure of university training of scientists and public-
policymaking professionals.
The law has a similar communication problem. Doing and communicating
neutral decisionmaking are as different from one another as doing and
communicating valid science. Just as solving the science communication
problem demands scientific knowledge and appropriate institutional reforms,
so solving the law’s neutrality communication problem will require
appropriate acquisition and use of empirical knowledge of a sort aimed at
expanding understanding of how people come to recognize the neutrality of
the law and what law should do to make its neutrality fully recognizable.
There is one critical difference, however, between the science
communication problem and the neutrality communication problem. Unlike
scientists, judges are expected both to make valid decisions and communicate
the validity of their work to the public. It is widely recognized that the
CONCLUSION
The motivations for conducting the study described in this Article were
two. The first, narrower and more immediate one was to examine whether
judicial decisionmaking is “ideologically biased.” The results of the study,
which was designed to remedy methodological defects that prevent drawing
valid inferences from existing studies, supply reason to discount the pervasive
claim that judges are “politicians in robes.”
The second, more general, and much more generally important, aim was
to demonstrate the need for making judging an evidence-based profession.
Like other experts, judges are endowed with expert professional judgment—
“situation sense,” in Llewellyn’s terms. In law no more than in any other
profession, the use of empirical methods cannot plausibly be viewed as an
alternative to either the role of shared experience in generating professional
judgment or the successful acquisition and proficient use of such judgment
by individual practitioners.
But also like the professional judgment of all other manner of experts, the
suitability of lawyers’ “situation sense” for the decisionmaking task that they
must perform inevitably depends on its being informed by empirical facts, the
nature and significance of which will evade confident detection by casual
reflection. The disciplined methods of observation, measurement, and
inference that are distinctive of science furnish the most reliable basis—the
only reliable basis—for determining what those facts are. The legal profession
comprises norms of collective self-reflection and -assessment that can be
expected to instill in their members the habits of mind and the dispositions
necessary to solve the neutrality communication problem and like challenges,
but only if those processes are informed by valid understandings of how the
world actually works.
2016] “Ideology” or “Situation Sense”? 423
202 See Kahan, Cognition of Law, supra note 9; Kahan, Neutral Principles, Motivated Cognition,
supra note 17, at 58-71.
424 University of Pennsylvania Law Review [Vol. 164: 349
1. Littering Problem
Models 1 and 2 of Table A1 add predictors for assignment to the
“immigrant aid” version of Littering and for cultural worldviews and
corresponding cross-product interactions, respectively. The positive sign of
the coefficient for “imm” (b = 0.33, p < 0.01) indicates that for the subjects
considered as a whole the likelihood of finding a violation was higher in the
“immigrant aid” than in the “construction worker” version of the problem.
The positive coefficients for the two cross-product interaction terms in Model
2—“hfac_x_imm” (b = -0.50, p < 0.01) and “ifac_x_imm” (b = 0.26,
p < 0.05)—indicate that the relative probability of finding a violation in the
“immigrant aid” version as opposed to the “construction worker” version
increased as subjects became more hierarchical and individualistic;
correspondingly, the relative probability of finding a violation in that version
of the problem as opposed to the “construction worker” version decreased as
subjects became more egalitarian and communitarian. Because Models 1 and
2 do not include predictors relating to the types of subjects, they should be
interpreted as effects for subjects “on average.”
The predictors added to Model 3 account for differences between the re-
sponses of the various subject types. The substantial improvement in the fit
of the Model (ΔLR χ2 = 143.3(18), p < 0.01) supports the inference that in
2016] “Ideology” or “Situation Sense”? 425
Model 1 2 3
Imm 0.33 (3.18) 0.34 (3.26) 0.27 (1.69)
Hfac -0.08 (-1.12) -0.34 (-3.41)
Ifac 0.1 (0.09) -0.29 (-2.81)
hfac_x_imm 0.50 (4.65) 0.77 (5.32)
ifac_x_imm 0.26 (2.42) 0.46 (3.02)
student -0.66 (-2.95)
lawyer -1.32 (-4.24)
Judge -1.59 (-5.52)
student_x_imm 0.10 (0.31)
lawyer_x_imm -0.01 (-0.01)
judge_x_imm -0.01 (-0.04)
hfac_x_student 0.55 (2.30)
hfac_x_lawyer -0.02 (-0.05)
hfac_x_judge 0.54 (1.94)
ifac_x_student 0.19 (0.93)
ifac_x_judge 0.49 (1.83)
ifac_x_lawyer 0.36 (1.17)
student_x_hfac_x_imm -0.45 (-1.32)
student_x_ifac_x_imm -0.42 (-1.32)
lawyer_x_hfac_x_imm -0.74 (-1.59)
lawer_ifac_x_imm -0.47 (-1.21)
judge_x_hfac_x_imm -0.91 (-2.45)
judge_x_ifac_x_imm -0.63 (-1.59)
LR χ2 10.1(1) 53.2(5) 196.5(23)
ΔLR χ2 43.0(4) 143.3(18)
Note: N = 1502. Outcome variable is ruling (1 = violation, 0 = no violation). Predictor estimates are
logit coefficients with z-test statistic indicated parenthetically. Bolded typeface indicates predictor
coefficient, model LR χ2, or incremental change in model LR χ2 is significant at p < 0.05. Listwise
deletion for missing data.
426 University of Pennsylvania Law Review [Vol. 164: 349
It is readily apparent from Model 3 that members of the public were more
inclined to find a violation in the “construction worker” version than were
other types of subjects. The predictors for being a student (b = -.66, p < 0.01),
a lawyer (b = -1.32, p < 0.01), and a judge (b = -1.59, p < 0.01) reflect how being
the indicated type of subject as opposed to being a member of the public
influences the probability of finding a violation in the “construction worker”
version of the problem, when cultural worldviews are “controlled for” (i.e.,
when the predictor values for hfac and ifac are set at their mean value, 0).
The coefficients for each of those predictors are negative, indicating that
members of the public, on average, are more inclined to find a violation in
the “construction worker” version of the problem than the other subject
types.
The coefficient for “imm” now reflects how much more likely a member
of the public is to find a violation in “immigrant aid” than in “construction
worker” when that member of the public is “average” in cultural worldview.
The sign of that predictor is positive (b = 0.27), implying that a member of
the public is still more inclined to find a violation in the “immigrant aid”
version, although this propensity is, for what it is worth, only “marginally
significant” (p = 0.09).
Also readily apparent from the output of Model 3 is the impact that
cultural outlooks have on members of the public. The coefficients for both
“hfac” (b = -0.34, p < 0.01) and “ifac” (b = -0.29, p < 0.01), are negative, and
those for “hfac_x_imm” (b = 0.77, p < 0.01), and “ifac_x_imm” (b = 0.46,
p < 0.01), are both positive. These results reflect the strong impact—visible
in the summary data (Figure 5)—that members of the public were inclined
to polarize along the lines consistent with the predicted impact of identity-
protective cognition: that is, as they became more hierarchical and
individualistic, members of the public became more inclined to find a
violation in the “immigration-aid” version and less so in the “construction
worker” version; as they became more egalitarian and communitarian, they
displayed the opposite decisionmaking tendency.
The remaining information in the regression is less readily accessible.
Figuring out how much more or less a judge is affected by his cultural
outlooks in either the “immigrant aid” or “construction worker” version of
the problem than is a member of the public or a law student, or a law student
is than a lawyer, or a lawyer than a member of the public or a judge requires
adding appropriate combinations of Model 3 predictors. It is the sign and
magnitude of those sums and not the sign and magnitude of individual
predictor coefficients that must thus be examined to assess the competing
study hypotheses.
2016] “Ideology” or “Situation Sense”? 427
EC
EC HI imm rts HI
HI
HI constr imm rts constr
imm rts
constr EC
EC
imm rts
constr
Lawyers Judges
EC EC
constr EC constr
imm rts EC
HI imm rts
constr HI HI
imm rts imm rts
HI
constr
203 See G ELMAN & HILL, supra note 146, at 137-51; King, Tomz & Wittenberg, supra note 148,
at 351-53 (explaining that simulation is preferable to analytical methods because, among other things,
it “can provide accurate answers even when no analytical solutions exist”).
428 University of Pennsylvania Law Review [Vol. 164: 349
the entire multivariate normal distribution for the regression model estimates
of the predicted probabilities associated with relevant combinations of
predictors.204
Based on the simulated values, the predicted probability that a member
of the public will find that the defendants committed a violation in the
“immigrant aid” version of the problem is 28% higher (± 14%) if that member
of the public has moderately hierarchical and individualistic values than if
that individual has moderately egalitarian and communitarian ones. By the
same token, the predicted probability that a member of the public will find
that the defendants committed a violation in the “construction worker”
version of the problem is 30% higher (± 12%) if that member of the public has
moderately egalitarian and communitarian values than if that individual has
moderately hierarchical and individualistic ones.
The interaction of the experimental assignment with the cultural
worldviews of student decisionmakers is less dramatic but still evinces modest
effects consistent with identity-protective cognition. The biggest impact is
on a Hierarchical Individualist student, whose predicted probability of
finding a violation is 17% higher (± 28%) in the “construction worker” version
than in the “immigrant aid” version. The difference is not statistically
significant, however, at the conventional p < 0.05 level.
For lawyer and judge decisionmakers, in contrast, the impacts are
inconsistent with identity-protective cognition. Indeed, whether an
Egalitarian Communitarian decisionmaker is a lawyer or a judge, the
predicted probability that he or she will find a violation in the “immigrant
aid” version is higher than it is in the “construction worker” version. If that
decisionmaker is a lawyer, the probability that he or she will find a violation
in the “immigrant aid” worker version is lower if the decisionmaker is a
Hierarchical Individualist than if the decisionmaker is an Egalitarian
Communitarian. These patterns are contrary to the ones associated with
identity-protective cognition, although the magnitudes of all these effects
differ by an amount that is neither practically nor statistically significant.
204 See King, Tomz & Wittenberg, supra note 148, at 349.
2016] “Ideology” or “Situation Sense”? 429
2. Disclosure Problem
Table A2 presents the regression analyses for subject responses to the
Disclosure problem. The coefficient for “pro-life” in Model 1 (b = 0.08,
p = 0.80) is only trivially different from zero, implying that, overall, subjects
were not meaningfully more likely to find a violation in the “pro-life center”
version than the “pro-choice center” version.
Lawyers Judges
HC
EI pro-life
pro-life
EI EI
pro-choice EI pro-choice
HC pro-life
pro-choice
HC
HC pro-choice
pro-life
Note: The figure is derived from a Monte Carlo simulation based on Regression Model 3, Table A2.
Predictors for cultural worldviews set at +1 for hierarchy and -1 for individualism and -1 for hierarchy
and +1 for individualism in the case of Hierarchical Communitarian (“HC”) and Egalitarian
Individualist (“EI”) decisionmakers, respectively. The curves reflect the probability density
distribution for the predicted probability that the indicated subject type will find a violation in the
indicated version of the problem. The most likely predicted outcome is the probability
corresponding to the apex of the curve; probabilities higher or lower become progressively smaller
as one approaches the values at the extreme tails of the curve.
2016] “Ideology” or “Situation Sense”? 431
B. Risk Perceptions
Table A3 sets forth separate regression models for each of the societal-
risk perception scales.205 Predictors include the continuous cultural
relatively low degree of variance with respect to the social deviancy risks in
the general population, combined with the modest reliability of the two-item
abbreviated version of the individualism–communitarianism scale. It is clear
from the cross-product interaction terms, however, that differences in the
communitarian–individualism predictor did contribute to variance for
members of the student sample (b = -0.10, p < 0.05). It is also clear that being
a student (b = -0.50, p < 0.01) and being a lawyer (b = -0.47, p < 0.01), but not
being a judge (b = -0.04, p = 0.40), as opposed to being a member of the public
predicted less concern with social deviancy risks when worldviews were held
constant at their mean.
shapes their processing of information about the relevant classes of risk. The
impacts vary in size. For example, the estimated impact on members of the
public is larger than that of the other subject types for environmental risks
and smaller than the estimated impact on the other subject types for social
deviancy risks.
But unlike the results observed in relation to the legal problems, the
impact of cultural worldviews is manifest and substantial for all subject types.
The results do not furnish support, then, for the hypothesis that either the
professional judgment characteristic of lawyers and judges or the experience
of being a judge meaningfully counteracts identity-protective reasoning for
out-of-domain judgments.
EC HC EI
HI EC HI HC EI
-1 0 1 -1 0 1 -1 0 1 -1 0 1
Perceived risk Perceived risk Perceived risk Perceived risk
-1 0 1 -1 0 1 -1 0 1 -1 0 1
Perceived risk Perceived risk Perceived risk Perceived risk
Note: The curves reflect the probability density distribution for the estimated score on the indicated
societal risk scale. Scores on the scales are normalized, with the mean set at 0 and with units in
standard deviations. Predictors for cultural worldviews set at +1 for both hierarchy and individualism
for “Hierarchical Individualist” (“HI”); at -1 for both for “Egalitarian Communitarian” (“EC”); at
+1 for hierarchy and -1 for individualism in the case of Hierarchical Communitarian (“HC”); and at
-1 for hierarchy and +1 for individualism for Egalitarian Individualist (“EI”), respectively. The most
likely estimated score is the one corresponding to the apex of the curve; probabilities higher or lower
become progressively smaller as one approaches the values at the extreme tails of the curve.
2016] “Ideology” or “Situation Sense”? 435
1. Littering
You are a trial court judge presiding over a “civil penalty” action filed by
the U.S. government under a law known as the Wildlife Environment
Protection Act (“the Act” or “WEPA”). WEPA prohibits “littering,
disposing, or depositing any form of garbage, refuse, junk, or other debris”
on land designated as a national wildlife preserve. The Government has
charged the defendants, [a group of construction workers,/members of a
immigrants’ aid group,] with 400 separate WEPA violations (each subject to
a $500 fine) for dispersing, and thereafter leaving unattended, 400 ten-gallon
reusable plastic dispensers of drinking water in a wildlife refuge located in
the desert along the United States–Mexico border. [The defendants placed the
dispensers along a fifty-mile stretch in which they had been hired to do work on the
construction of a “border fence” to keep out illegal aliens. The defendants anticipated
drinking the water as they completed their work over a three-month
period./Defendants placed the dispensers along a fifty-mile stretch known to
be traversed by undocumented migrant farm workers. The defendants’
expected the water to be found and consumed by the migrant workers, who
face a high risk of death from dehydration during attempts to cross the
border.]
The issue raised by the defendants’ motion is how to interpret WEPA.
The defendants argue that they had not permanently discarded the plastic
water dispensers but instead temporarily placed them in the desert with an
expectation that they would be used and reused. Such behavior, they argue,
does not count as “littering, disposing, or depositing any form of garbage,
refuse, junk, or other debris” under WEPA.
The Government focuses on the terms “depositing,” “junk,” and “other
debris.” On its reading, the defendants “deposited” the water dispensers in
the desert by placing them there and then leaving them unattended. The
436 University of Pennsylvania Law Review [Vol. 164: 349
terms “junk” and “debris,” the government argues, are by design very broad
and cover all manmade materials, including reusable plastic water dispensers,
foreign to the habitat of wildlife in the preserve.
LITTERING_ruling. We are interested in knowing how you might
decide the defendant’s motion to dismiss. That motion should be granted if
the defendant’s interpretation of WEPA is correct but denied if the
Government’s competing interpretation is correct. Of course, if you were
really a judge in the case, you’d do more legal research, and hear arguments
from the parties. But at this point, based on the materials you’ve read, which
of these two rulings do you think you would make, and how confident do you
think you’d be in your decision?
Select one:
( ) Based on my analysis of the statue, I would conclude that the
defendants DID violate WEPA.
( ) Based on my analysis of the statue, I would conclude that the
defendants did NOT violate WEPA.
2. Disclosure
You are one judge of three on an appellate court. D, a former state police
officer, is appealing his conviction under a state law known as the
Government Information Disclosure Act (“Act” or “GIDA”). Evidence at
trial shows that D told his sister, the owner of [a noncommercial “family
planning” center that provides free information on birth control and abortion
services/a religious “family planning” center that counsels women on
alternatives to abortion], that C, an applicant for a job at the center, belonged
to a local [anti-abortion/abortion-rights] group.
The relevant language of GIDA states:
No government employee shall disclose to anyone outside the
government confidential information relating to an identifiable private
citizen unless such disclosure is subject to a statutory exception. Any
government employee who knowingly violates this Act shall be guilty of a class
C Felony.
The information about C’s membership in the [anti-abortion/abortion-
rights] group came from a police investigatory file. There is no dispute that
the information was “confidential” under GIDA and not subject to any of the
law’s exceptions (for example, the one allowing disclosure “when necessary
to avert an imminent threat to public health or safety”).
Testifying on his own behalf, D admitted that he “knew” that disclosure
“was against departmental policy on confidential investigatory files,” but
stated that he “had no idea” GIDA existed, and hence didn’t “know”
2016] “Ideology” or “Situation Sense”? 437
disclosure violated that or any other statute. He testified that he released the
information about C’s group membership because he was “outraged” that C,
in light of his [anti-abortion/abortion-rights] activities, would seek
employment at the family-planning service operated by D’s sister.
The issue before the appeals court is a narrow one: under GIDA, what
exactly does the prosecution have to prove for a jury to find a defendant
“knowingly violated this Act”?
At trial, the defense requested the court to issue an instruction to the jury
stating, “Because a person commits a crime under GIDA only when he
‘knowingly violates’ the Act, you should return a verdict of guilty only if you
find beyond a reasonable doubt that D knew his conduct violated GIDA.”
The court rejected this request. Instead it instructed the jury, “Because a
person commits a crime under GIDA only when he ‘knowingly violates’ the
Act, you should return a verdict of ‘guilty’ only if you find beyond a reasonable
doubt that D (1) knew he was disclosing government information that (2) he
knew was ‘confidential’ and that (3) he knew ‘related to a private citizen.’” The
court told the jury that it “need not find, however, that D (4) knew his conduct
violated GIDA or any other criminal law.”
If the court of appeals finds that the trial court’s instruction reflected a
correct reading of the Act, the court of appeals will uphold the conviction. If
the court of appeals finds the trial court’s instruction was incorrect, the court
of appeals will reverse the conviction and order a new trial at which the trial
court will be required to give the instruction requested by the defense.
DISCLOSURE_ruling. We are interested in knowing how you might
decide this case if you were on the court of appeals. Of course, if you were
really a judge in the case, you would do more legal research, hear arguments
from the parties, and consult with your fellow judges. But at this point, based
on the materials you’ve read, which of these two rulings do you think you
would make, and how confident do you think you’d be in your decision?
Select one:
( ) Based on my analysis of the statue, I would conclude that the only facts
the prosecution has to prove are (1) that the defendant employee knew he or
she was disclosing information; (2) that the defendant employee knew that
the information was “confidential”; and (3) that the defendant employee
knew the information “related to” an “identifiable private citizen.”
( ) Based on my analysis of the statute, I would conclude that in a criminal
case under GIDA the prosecution must prove not only that the defendant
knew that he or she was disclosing information that he or she knew was
confidential and knew related to an identifiable private citizen; it must also
438 University of Pennsylvania Law Review [Vol. 164: 349
prove that the defendant employee knew that disclosure of such information
would violate GIDA.
B. Risk Perceptions
Introduction. Now we would like to ask you about your views about risks.
As individuals and as a society, we face a number of possible hazards. Some
threaten people’s health, safety, or prosperity directly. Others threaten
health, safety, or prosperity indirectly through the damage they can impose
on the environment or the economy. How much risk do you believe each of
the following poses to human health, safety, or prosperity? [0 “no risk at all”;
1 “Very low risk”; 2 “Low risk”; 3 “Between low and moderate risk”; 4
“Moderate risk”; 5 “Between moderate and high risk”; 6 “High risk”; 7 “Very
high risk.”]
[RANDOMIZE ORDER]
C. Cultural Worldviews
Introduction. Finally, we’d like to ask you about some moral and social
issues.
A. Dimension 1 (Individualism–Communitarianism)
People in our society often disagree about how far to let individuals go in
making decisions for themselves. How strongly do you agree or disagree with
each of these statements? [1 “Strongly disagree”; 2 “Moderately disagree”; 3
“Slightly disagree”; 4 “Slightly agree”; 5 “Moderately agree”; 6 “Strongly
agree.”]
2016] “Ideology” or “Situation Sense”? 439