JUDGES in The LAB. No Common-Civil Law Differences
JUDGES in The LAB. No Common-Civil Law Differences
HARVARD
JOHN M. OLIN CENTER FOR LAW, ECONOMICS, AND BUSINESS
Holger Spamann
Lars Klöhn
Christophe Jamin
Vikramaditya Khanna
John Zhuang Liu
Pavan Mamidi
Alexander Morell
Ivan Reidel
11/2020
*
Lawrence R. Grove Professor of Law, Harvard Law School. [email protected] (corresponding author).
#
Professor of Law, Humboldt Universität zu Berlin. [email protected].
♠
Dean and Professor of Law, Sciences Po Law School. [email protected].
♥
William W. Cook Professor of Law, University of Michigan Law School. [email protected].
♦
Asst. Professor of Law and Economics, The Chinese University of Hong Kong, Shenzhen. [email protected].
♣
Principal Investigator for Law, Sociology, and Management and Director of International Research Collaborations
for the Centre for Experimental Social Sciences (CESS) Nuffield – FLAME University. [email protected].
⊕
Professor of Law, Universität Mannheim. [email protected].
§
CEO of Sinfonik Inc., formerly Professor of Law, Universidad Torcuato di Tella. [email protected].
∞
We thank FLAME University, Harvard Law School, Harvard Law School’s Summer Research program, Humboldt
Universität zu Berlin, Ludwig-Maximilians-Universität, the Max Planck Institute for Collective Goods, Sciences Po,
and Universidad Torcuato di Tella for financial support; the Max Planck Institute for Collective Goods for use of their
mobile lab; Roland Ramthun of Docustorm for programming the experimental interface; Brendan Halpin for help
with the Stata package SADI (and for programming it in the first place); Alex Whiting for suggesting the aiding and
abetting controversy; Damien Charlotin, Leandro Dias, and Máximo Langer for information about national laws;
Gabriela Haymes, Henrique Duchini, Allana Pereira, Andrioli Soares, Chen Gu, Linda Hao (Langscale Translation),
Aurelien Bernard, Damien Charlotin, Jakob Jochmann, and Lola Witt for translating the experimental materials into
local languages; Marcelo Moreno Bonassa, Etiene Coelho Martins, Linda Hao (Langscale Translation), Damien
Charlotin, Mareike Grundmann, and Andrew Hammel for translating the judgment reasons from local languages into
English; James Foy, Andres Constantin, Victoria Arredondo Civico, Amanda Bernztein, Nicole Varela, Julian Franke,
Oliver Schön, Martin Sternberg, Philipp Schmidt, Andrés Parrado, Shardul Vaidya, Howard Duan, Molly Eskridge,
Tyler Good-Cohn, and JohneHenre Rzodkiewicz for excellent research assistance and administrative support; Judge
Marta Capalbo, Judge Duilio Campora, and Mariano Sigman (Argentina), Judge Bruno Bodart, Estevao Gomes, Judge
Isabela Isa, and HLSBSA (Brazil), Xueyao Li (China), Justice Dominique Lottin, Judge Pénélope Postel-Vinay, and the
ENM (France), Deutsche Richterakademie (Germany), Ranita Nagar, Kalpesh Kumar, Gujarat National Law University,
and the Chief Justice of the Gujarat High Court (India), Judge Nancy Gertner, John Manning, Denise Neary, and the
FJC (U.S.), for accommodating and/or supporting the respective sessions at which we collected our data; and above
all the judges for their participation. For comments, we thank participants and commentators at the American
Society of Comparative Law’s Special Committee on Comparative Law and the Social Sciences, the Max Planck
Institute for Collective Good, the Max Planck Institute for Tax Law and Public Economics, Academica Sinica, City
University Hong Kong, ETH, Freie Universität Berlin, Harvard Law School, Hong Kong University, Humboldt Universität
zu Berlin, Sciences Po, Sichuan University, University of Hamburg, Wissenschaftszentrum Berlin, the Annual
Meetings of the American Law and Economics Association (2019) and the American Political Science Association
(2018), the 14th Annual Conference on Empirical Legal Studies, the 2019 Experimental Methods in Legal Scholarship
Conference, and the Workshop on Methoden Quantitativer Textanalyse at Humboldt Universität and
Wissenschaftskolleg zu Berlin, and Katerina Linos. We have no conflicting interests with respect to this research. This
research was approved by Harvard University’s Committee on the Use of Human Subjects as IRB15-0206 (US), IRB15-
3608 (AR), IRB15-3803 (CN), IRB17-1263 (DE), IRB17-1430 (FR), IRB18-0113 (IN), and IRB18-0304 (BR). All data and
code are available at doi.org/10.7910/DVN/TTMUZK with the exception of raw U.S. covariate data, which pursuant
to IRB conditions can only be obtained from Spamann upon signing a confidentiality agreement. All experimental
materials are included as appendices to the supplementary materials.
10/11/2020
Judicial opinions—the official reasons judges write for their decisions—would suggest the opposite
answers to both questions. The style of judicial opinions differs systematically between countries, ranging
from the narrowly technical, terse, and syllogistic style of the French Cour de Cassation to the more liberal,
discursive, and expansive style of the U.S. Supreme Court (e.g., Lasser 2004), and many have seen
systematic differences between common and civil law (e.g., Wetter 1960; Kötz 1982). Similarly, most
courts extensively cite precedent, including horizontal precedent, even in jurisdictions that do not have
an explicit norm of binding precedent like the common law (stare decisis) (Summers & Taruffo 1991 §VI).
As evidence of judicial thinking, however, judicial opinions are at least problematic. To maintain their
legitimacy, judges must project a certain image, which may differ by country, but what really drives judges’
decisions is a different matter (e.g., Gény 1899; Kantorowicz 1906; Frank 1930; Llewellyn 1930; Kennedy
1998; Simon 1998; Lasser 2004; Epstein et al. 2013). 1
To answer the first question—differences between common and civil law thinking—without relying on
judicial opinions, we generate a novel type of evidence: judges’ document use while they work on deciding
a case. This “document path” process imaging technique reveals judicial thought structures in a similar
way as MRIs and other brain imagining techniques reveal brain structures: the image is partial and
requires statistical inference, but it is nevertheless highly informative. We analyze similarities between
document paths using sequence analysis (edit distance), a method that we show to be very powerful in
detecting even small differences between groups of judges. Nevertheless, we detect only differences
between individual countries but not between common and civil law judges. Common and civil lawyers
do not think differently after all, at least those in our sample at the resolution of our document path
image.
1
At a minimum, judges have more discretion than their opinions admit: A lower bound on legal indeterminacy and
judicial error is set by inconsistency between judges deciding the same cases on a panel (e.g., justices on the U.S.
Supreme Court) or deciding random draws from the same distribution of cases (e.g., judges on U.S. immigration
courts) (Fischman 2013). How judges use this discretion correlates with their background and ideology (e.g.,
Rachlinski & Wistrich 2017; Harris & Sen 2019).
We create a unique setting to study both questions: 299 real judges from seven key jurisdictions spend
up to 55 minutes deciding—with written reasons—a fully briefed appeals case. From the perspective of
comparative law, this setting is unique simply because comparative law has never used laboratory or
experimental methods—we are not aware of any other study that has observed lawyers from multiple
jurisdictions under controlled conditions. We recruited our participants at comparable professional
education seminars in jurisdictions representing half the world population and playing key roles in the
common/civil law taxonomy. France (n=43) and Germany (n=74) are the historical and, by most accounts,
current center of the civil law world (e.g., Zweigert & Kötz 1998). Others see the current center of the civil
law in Latin America (Merryman & Pérez-Perdomo 2007), of which Brazil (n=33) is the largest and
Argentina (n=31) the historically dominant jurisdiction. The United States (n=29) is one of the two current
centers of the common law world (our attempts to recruit participants in the other, historical center of
the common law, England, were unsuccessful). China (n=47) and India (n=42) are the world’s largest
countries and belong to the wider civil and common law, respectively (on China, see Liu et al., in press).
Informed consent was obtained. See SI S1 for more information on venues and recruitment.
From the perspective of experimental studies of judicial decision-making, our setting offers an unusually
high degree of realism. Most other studies do not have judges as research subjects, and those that do
provide only a vignette summary of the decision context and do not ask for reasons. 3 These other studies
may fail to capture key, outcome-determinative features of judicial decision-making in the real world.
Judges are highly trained and selected professionals. They operate in an environment designed to
elucidate all sides of the case through adversarial argument, emphasize the importance of objectivity
through decorum, provide time for reflection, and create accountability, principally by the obligation to
provide reasons for the decision (Llewellyn 1940; Spellman and Schauer 2012; Kahan 2015; Spamann and
Klöhn 2016). Our design includes all of these features.
Our study casts participants in the role of a judge on the Appeals Chamber of the International Criminal
Tribunal for the Former Yugoslavia (ICTY) (see SI S2 for more details). Participants’ task is to decide a
2
Spamann & Klöhn (2016) and Liu et al. (in press) report partial results of the present study for the USA and China,
respectively. A parallel study by one of us, Klerman and Spamann 2019, is discussed below.
3
The closest other studies we are aware of in this respect are Englich and Mussweiler 2001 and Englich et al. 2006,
who provide a 4-page description of a rape case to German judges spending about 15 minutes on a sentencing task;
however, in Germany, a judge would never sentence a defendant without having first determined guilt during a full
trial. One of us uses a similarly realistic design in a subsequent study discussed below (Klerman and Spamann 2019).
Unbeknownst to participants, each participant is randomly assigned one of two defendants, one of three
precedents, and—in the sentencing part—one of three anchors in a 2×3×3 factorial experiment (see SI S3
for more details). The defendant is either sympathetic—a regretful, conciliatory Croat—or
unsympathetic—a hateful, nationalist Serb—; these attributes are strictly legally irrelevant for purposes
of determining whether a crime has been committed even though they may be taken into account for
sentencing, i.e., for setting the appropriate penalty (e.g., Nadler & McDonnell 2012). The precedent either
supports affirmance (Affirm) or supports reversal weakly (reverse) or strongly (REVERSE). REVERSE is a
disguised version of the actual decision in Prosecutor v. Perišić, i.e., it concerns the exact same facts and
is thus the strongest precedent imaginable. 4 Affirm is the explicit, lengthy refutation of REVERSE by
another panel of the Appeals Chamber and hence very strong precedent in the opposite direction even
while dealing with somewhat different facts (we redacted Affirm to hide the existence of REVERSE; in our
redaction, Affirm addresses a position advanced by the defense). reverse defines aiding and abetting in
terms that would exclude our defendant’s behavior; however, this definition is neither discussed at length
nor outcome-determinative in reverse, limiting its precedential weight. Finally, the sentencing task is
explained using an example of 10, 25, or 40 years (anchor). Prior research on judicial biases has shown
judges to be affected by anchors, specifically in sentencing (e.g., Englich and Mussweiler 2001; Guthrie et
al. 2001; Englich et al. 2006; Rachlinski et al. 2015), and by party sympathies (Rachlinski et al. 2009;
Wistrich et al. 2014). See SI S3 for more information on the experimental treatments.
As in our study, many domestic cases are decided by a single judge without a hearing on a limited record
and in about one hour (Spamann & Klöhn 2016, §5.2). Our reason to use an ICTY case is that it is neutral
between participants’ national backgrounds, while its criminal law subject matter is familiar to
participants from their national law. The ecological validity of our study hinges on whether participants
understand the case and approach it with their judicial mindset. Participants’ written reasons and
behavior in the lab suggest they do. With very few exceptions (the exclusion of which does not affect
results), participants’ reasons are coherent and mostly adopt a specifically judicial diction (see SI S1). In
the lab, participants approached the task with the utmost concentration and seriousness. Four actual or
potential participants even expressed reservations about “deciding” a case of this gravity in such a short
time. Most participants finished early, but some had to be asked repeatedly to finish writing their reasons.
Subsequent work by one of us (Klerman and Spamann 2019) replicates the absence of a (strong) law effect
4
We did not use REVERSE in the first country where we ran the study, the United States, because we had expected
that reverse would be strong enough to generate a precedent effect. We then added REVERSE in the other six
countries.
Results
Comparative Judicial Thinking
Figure 1 plots our main comparative evidence, the judges’ document view paths by country. Each judge’s
time scale is normalized by that judge’s total time, so that all paths are of the same length. This
normalization suppresses variation in length between individual judges (mean 35 min., s.d. 10 min.) and
between countries (Argentina 37, Brazil 32, China 28, France 37, Germany 36, India 40, USA 36) that we
believe to be orthogonal to our question of interest, which is how judges think, i.e., how they form their
view of a case, not how long it takes them to form that view.
At a high level and relative to the range of possible paths, the distribution of judges’ paths looks similar in
all countries. Judges everywhere tend to begin with the facts and the briefs (which the study instructions
recommended reading in full) before examining the trial judgment, with brief glances at the statute
throughout and longer examinations of the precedent later. This is only a tendency, though: individual
judges do not all start and end with the same documents nor move through the documents at the same
rhythm; there is even more heterogeneity at the paragraph level, which we have found too variable for
insight and thus do not report. To the naked eye, the heterogeneity is mostly within country; there are no
obvious country patterns.
Civil Law
31 33 47
start finish start finish start finish
France Germany
1 1
Judge
43 74
start finish start finish
India USA
facts 1 1
Common Law
briefs
trial judgment
statute
precedent
42 29
start finish start finish
Normalized Time
By country (plot), the graph shows for each judge (vertical axis) the documents live on that judge's screen from start to finish (horizontal axis). Orange indicates instruction views.
For a more nuanced, formal analysis, the first question we need to address is the choice of analysis
method. Since our data are of a novel type, there is no established method for analyzing them in
comparative law and related fields. Nor is the widespread belief that common and civil lawyers think
differently articulated in specific testable hypotheses. 5 Simple multivariate ANOVA with the frequency of
different documents’ views as dependent variables rejects equality of countries at p<0.001 but not
equality of common and civil law (p=.78). However, ANOVA does not take into account the ordering of
document views, which one might argue is the most characteristic aspect of possibly different ways of
thinking. To capture this important dimension, we introduce sequence analysis to comparative law: we
discretize all document paths to 500 steps of equal normalized length, calculate the Levenshtein distance
between any two such sequences, aggregate the within-group distances normalized by total distances,
and then compare this normalized aggregate to its empirical distribution under random permutation of
group labels. In more detail: Levenshtein distance is a standard metric for distance between sequences of
categorical data in computer science, linguistics, and other fields. It is the minimum number of edit
operations (delete, insert, or substitute) required to transform one sequence into the other. (We obtain
identical results with substitution cost equal to 1.5 times the cost of insert or delete. We use Halpin’s
(2017) implementation.) Then letting SST be the total sum of all pairwise Levenshtein distances divided by
5
Zweigert & Kötz 1998 §5.III.2, Legrand 1996, 1999, and others claim that civil law thinking is abstract, systematizing,
and institution-focused while common law thinking is more concrete, casuistic, and fact-based. None of these
differences seems sufficiently specifically articulated, however, for rigorous testing.
When the groups are countries, the pseudo-R2 equals .04. This is low, but much higher than random: in
100,000 random re-assignments of all country labels to judges, we not once obtained a pseudo-R2 this
high, i.e., p≤10-5. In other words, there are detectable country differences in the document view paths.
When the groups are common and civil law, the pseudo-R2 is much lower still (<.01). To assess its statistical
significance, we need to permute common/civil law labels of countries, not of individuals, because the
common/civil law hypothesis and corresponding null hypothesis is about the similarity of entire countries
and hence groups of judges, not merely of judges within countries. 6 With two group members and seven
countries, we reject equality of common and civil law at p≤.05 only if the two common law countries India
7
and USA generate the most extreme pseudo-R2 of any of the � � = 21 possible country pairs. While this
2
seems to be a demanding test, the extreme test statistic is arguably exactly what one should expect if
common and civil lawyers really thought differently. In any event, we have found empirically that the test
is not so demanding after all, i.e., the test has high power even for relatively small differences. In simulated
data drawn with replacement from the entire population of judges, even a 1% “half-time” sliver (the
middle 5 out of the 500 steps) set to precedent for (arbitrarily designated) “common law judges” and to
statute for the other, “civil law” judges is detected 72% of the time, and a 2% sliver is detected 99% of the
time. When we also simulate within-country similarities by first drawing countries and then drawing
judges only within countries, in each case with replacement, we still estimate 93% power for detecting a
5% sliver. In words, if there are meaningful differences between common and civil law, our permutation
test has high power to detect them at p<.05. However, in actuality, our test does not come close to
rejecting equality of common and civil law (p=.29, i.e., 5 other country pairs generate a higher pseudo-R2).
The underlying reason is that judges from the two common law countries India and USA are very different.
For example, US and Indian judges spend respectively the lowest and second-highest average fraction of
time with the statute, and the second-highest and second-lowest with the precedent. We return in the
conclusion to the question of whether other common law countries might be more similar.
6
Intuitively, it is easy to see that individual permutation would be wrong: if all within-country distances were zero
and all countries were equidistant (i.e., no group of countries is special), then with probability approaching one,
permuting individual labels would yield greater “within-group” distances and hence a lower pseudo-R2 than any
actual group of countries simply because the permutation leads to more mixing of judges from different countries.
7
In the raw text data, average length of reasons measured in English words (mean 172, s.d. 140) differs considerably
between countries. Chinese (mean 74) and US (mean 94) judges tend to write short reasons, French (mean 262) and
German (mean 227) judges long reasons, with Argentinian (mean 168), Brazilian (mean 165), and Indian (mean 155)
AR BR CN FR DE IN US AR BR CN FR DE IN US AR BR CN FR DE IN US
Civil Law Common L. Civil Law Common L. Civil Law Common L.
Country means and 95% bootstrap confidence intervals.
In the upper panel, the mean is taken over indicators whether an individual judge mentioned the feature in the
reasons (scored as the number of 0-4 coders who so coded this judge, divided by four). In the lower panel, the
mean is taken over the indicators divided by the average number of words in that country's translated reasons.
We again see considerable variation within countries and across countries (now even with the naked eye),
but not between common and civil law. In fact, the biggest country differences are between the two
common law countries India and USA, who occupy opposite extremes for the prevalence of precedent
and, when scaled by average words, policy and whose differences are larger than those between the
common and civil law means on all 6 dimensions. As with document paths, permutation tests strongly
reject equality of the three prevalences—scaled or not—in the seven countries (p<10-4) but not in
common and civil law (p=.57); see SI S5 for the technical details. To the extent written reasons are
probative about judges’ thinking (see the general caveat in the Introduction), they thus confirm our finding
from document paths: judges “think differently” in different countries, but there is no evidence that these
differences are related to the common/civil law distinction.
Notice that our last finding concerns the arguments in judges’ reasons and thus is not inconsistent with
the common view reported in the Introduction that there are systematic differences in the style of judicial
opinions. Expert readers can arguably detect such differences also in our judges’ written reasons. We have
verified that natural language processing algorithms (NLP) can detect these differences too. NLP also
show, however, how superficial such differences can be: NLP can detect source language differences in
translations (Rabinovich & Wintner 2015), but that alone would hardly be considered evidence that the
judges in between. The differences in length might themselves be a difference in style, or simply reflect experimental
artefacts such as the greater ease of writing on a laptop (which we used in France and Germany) compared to on an
iPad (which we used in the United States and partly in China).
By Defendant
sympathetic 40
unsympathetic
Spikes indicate exact 95% c.i.s. Graphs by anchor value. Exclude 8 sentences >50 years.
All estimated effects go in the expected direction. The defendant’s conviction is affirmed more frequently
under Affirm than under reverse, and more frequently under reverse than under REVERSE. 8 The
unsympathetic defendant’s conviction is affirmed more often than the sympathetic defendant’s. The
distribution of sentence lengths shifts right as the anchor increases from 10 to 25 to 40.
8
REVERSE’s confidence interval is larger than the other two precedents’ because we underweighted REVERSE in our
randomization, see SI S3.
The precedent effect estimate is not meaningfully larger in subgroups that one might have expected to
exhibit a stronger precedent effect. Judges who mention the precedent in their reasons exhibit twice the
difference in affirmance rates between Affirm and REVERSE and between Affirm and reverse, but of course
even the double differences are still small: eleven (p=.14) and ten (p=.17) percentage points, respectively.
Judges from common law countries (which, recall, have stare decisis) do not exhibit stronger precedent
effects; in fact, their observed behavior is counter to precedent: US judges had higher affirmance rates
under reverse than Affirm (REVERSE was not used in the USA, see note 4), and all Indian judges affirmed
under REVERSE but not under Affirm and reverse. 11
Discussion
Our results provide negative answers to both our questions: Common and civil law judges do not appear
to think differently, and horizontal precedent does not affect judicial decisions, at least not much more
than sympathy and anchoring biases. Naturally, we cannot exclude the existence of effects that are
smaller than what we have power to detect, or that manifest only in different contexts. Still, our findings
are important because the effects we study were supposed to be large and not restricted to particular
contexts, and our 299 observations would have given us good power to detect even modest main effect
sizes. We emphasize the comparison of precedent effect to biases for this reason: whether or not
horizontal precedent ultimately has some effect, our troublesome finding is that the size of this effect is
bounded to be not much larger than that of biases.
Specifically, regarding the common/civil law distinction, our finding is limited to our sample of countries,
which for the common law includes only India and the United States. There might be more commonalities
between common law countries of a similar level of development or in the Commonwealth. Still, it is
important to know that any common law family resemblance does not extend to India and the United
States. Moreover, our data also show that similarity cannot simply be assumed even between similarly
developed, geographically adjacent countries of the same family: our document path and reason data
show considerable differences between France and Germany (they only rank 10th in proximity among the
21 country pairs).
9
The Fisher tests are stratified on country and the respective other factor (as in Jung 2014). This and the other results
for precedent and defendant reported in this and the next paragraph are similar in simple Fisher tests, ANOVA,
country-conditional logit, or country-conditional exact logit.
10
We obtain similar results in various regressions reported in SI S5.
11
By contrast, Chinese judges analyzed separately do exhibit precedent effects (Liu et al., in press).
To be sure, our design addresses only one of many constellations in which precedent might matter. In
addition, the realism of our design and that of Klerman & Spamann (2019) is imperfect. Most importantly,
our judges decide anonymously, whereas real-world judgments are signed and hence implicate
reputational consequences, including career concerns for lower judges. We believe that the most pressing
agenda for future research on judicial decision-making is to investigate this and similar ways and areas in
which law might matter for judicial decisions, and how this may differ across jurisdictions. Our findings
and those of Klerman & Spamann (2019) show law to be less determinative than previously thought (cf.
Spamann & Klöhn 2016, §3). But counterexamples such as Obergefell’s reception show that legal nihilism
is equally inapposite. Tracing the boundaries between these and similar situations, and identifying cross-
country differences in these boundaries, will help us understand and improve law’s important role in
society.
Similarly, our finding of considerable country differences coupled with the inability of the common/civil
law distinction to explain any of them suggests a fruitful search for alternative classifications of legal
systems, which is already under way (e.g., Glenn 2014). We hope that the document path evidence we
introduce in this paper can be of great help in this search. These novel data can presumably be analyzed
in innovative ways that we have not even thought of.
10
Englich, Birte, and Thomas Mussweiler. 2001. Sentencing Under Uncertainty: Anchoring Effects in the
Courtroom. Journal of Applied Social Psychology 31:1535-1551.
Englich, Birte, Thomas Mussweiler, and Fritz Strack. 2006. Playing Dice With Criminal Sentences: The
Influence of Irrelevant Anchors on Experts’ Judicial Decision Making. Personality and Social
Psychology Bulletin 32:188-200.
Epstein, Lee, William M. Landes, and Richard A. Posner. 2013. The Behavior of Federal Judges: A
Theoretical and Empirical Study of Rational Choice. Cambridge, MA: Harvard University Press.
Fischman, Joshua B. 2014. Measuring Inconsistency, Indeterminacy, and Error in Adjudication. American
Law and Economics Review 16:40-85.
Frank, Jerome. 1930 [2009]. Law & The Modern Mind. New Brunswick, NJ: Transaction Publishers.
Gény, François. 1899. Méthode d’Interprétation et Sources en Droit Privé Positif. Paris: L.G.D.J.
Glenn, H. Patrick. 2014. Legal Traditions of the World: Sustainable Diversity in Law (5th ed.). Oxford: Oxford
University Press.
Guthrie, Chris, Jeffrey J. Rachlinski, and Andrew J. Wistrich. 2001. Inside the Judicial Mind. Cornell Law
Review 86:777-830.
Guthrie, Chris, Jeffrey J. Rachlinski, and Andrew J. Wistrich. 2007. Blinking on the Bench. Cornell Law
Review 93:1-43.
Halpin, Brendan. 2017. SADI: Sequence analysis tools for Stata. The Stata Journal 17:546-572.
Harris, Allison P., and Maya Sen. Bias and Judging. Annual Review of Political Science 22:241-259.
Henderson, M. Todd, and William H. J. Hubbard. 2015. Judicial Noncompliance with Mandatory
Procedural Rules under the Private Securities Litigation Reform Act. Journal of Legal Studies
44:S87-S105.
Jung, Sin-Ho. 2014. Stratified Fisher’s Exact Test and Its Sample Size Calculation. Biometrical Journal
56:129–140.
Kahan, Dan M. 2015. Laws of Cognition and the Cognition of Law. Cognition 135:56-60.
Kahan, Dan M., David Hoffman, Danieli Evans, Neal Devins, Eugene Lucci, and Katherine Cheng. 2016.
“Ideology” or “Situation Sense”? An Experimental Investigation of Motivated Reasoning and
Professional Judgment. University of Pennsylvania Law Review 164:349–439.
Kantorowicz, Hermann [Gnaeus Flavius]. 1906. Der Kampf um die Rechtswissenschaft. Heidelberg: Carl
Winter’s Universitätsbuchhandlung.
11
Klerman, Daniel, and Holger Spamann. 2019. Law Matters – Less Than We Thought. SSRN Working Paper,
available at https://ssrn.com/abstract=3439526.
Kötz, Hein. 1982. Die Begründung höchstrichterlicher Urteile. In Preadviezen uitgebracht voor de
Nederlandse Vereniging voor Rechtsvergelijking no. 32, 5-21. Deventer: Kluwer.
La Porta, Rafael, Florencio Lopez-de-Silanes, and Andrei Shleifer. 2008. The Economic Consequences of
Legal Origins. Journal of Economic Literature 46:285–332.
La Porta, Rafael, Florencio Lopez-de-Silanes, and Andrei Shleifer. 2013. Law and Finance After a Decade
of Research. In Handbook of the Economics of Finance, ed. G Constantinides, M Harris, RM Stulz,
2A:425–91. Amsterdam: Elsevier.
Lasser, Mitchel de S.-O.-L’E. 2004. Judicial Deliberations: A Comparative Analysis of Judicial Transparency
and Legitimacy. New York: Oxford University Press.
Legrand, Pierre. 1996. European Legal Systems are Not Converging. International and Comparative Law
Quarterly 45:52-81.
Lehmann, Erich L., and Joseph P. Romano. 2005. Testing Statistical Hypotheses. Springer-Verlag New York.
Liu, John Zhuang, Lars Klöhn, and Holger Spamann. In Press. Precedent and Chinese Judges: An
Experiment. American Journal of Comparative Law.
Llewellyn, Karl N. 1930. A Realistic Jurisprudence—The Next Step. Columbia Law Review 30:431-465.
Merryman, John, and Rogelio Pérez-Perdomo. 2007. The Civil Law Tradition (3rd ed.). Stanford: Stanford
University Press.
Nadler, Janice, and Mary-Hunter McDonnell. 2012. Moral Character, Motive, and the Psychology of Blame.
Cornell Law Review 97:255–304.
Rabinovich, Ella, and Shuly Wintner. 2015. Unsupervised Identification of Translationese. Transactions of
the Association for Computational Linguistics 3:419–432.
Rachlinski, Jeffrey J., and Andrew J. Wistrich. 2017. Judging the Judiciary by the Numbers: Empirical
Research on Judges. Annual Review of Law and Social Science 13:203-229.
Rachlinski, Jeffrey J., Andrew J. Wistrich, and Chris Guthrie. 2015. Can Judges Make Reliable Numeric
Judgments? Distorted Damages and Skewed Sentences. 90 Indiana Law Journal 695-739.
Rachlinski, Jeffrey J., Sheri Johnson, Andrew J. Wistrich, and Chris Guthrie. 2009. Does Unconscious Racial
Bias Affect Trial Judges? Notre Dame Law Review 84:1195-1246.
Schauer, Frederick. 2018. Stare Decisis—Rhetoric and Reality in the Supreme Court. Supreme Court
Review 2018:121-143.
12
Spamann, Holger. 2015. Empirical Comparative Law. Annual Review of Law and Social Science 11:131–53.
Spamann, Holger, and Lars Klöhn. 2016. Justice is Less Blind, and Less Legalistic, Than We Thought:
Evidence From an Experiment with Real Judges. Journal of Legal Studies 45:255-280.
Spellman, Barbara, and Frederick Schauer. 2012. Legal Reasoning, in K. J. Holyoak & R. G. Morrison ed.,
The Oxford Handbook of Thinking and Reasoning, 2nd ed. Oxford University Press.
Studer, Matthias, Gilbert Ritschard, Alexis Gabadinho, and Nicolas S. Müller. 2011. Discrepancy Analysis
of State Sequences. Sociological Methods & Research 40:471-510.
Summers, Robert S., and Michele Taruffo. 1991. Interpretation and Comparative Analysis. In Interpreting
Statutes: A Comparative Study, ed. D. Neil MacCormick & Robert S. Summers. Brookfield, VT, USA:
Dartmouth.
Wetter, J. Gillis. 1960. The styles of appellate judicial opinions: a case study in comparative law. Leyden,
A. W. Sythoff.
Wistrich, Andrew J., Jeffrey J. Rachlinski, and Chris Guthrie. 2014. Heart Versus Head: Do Judges Follow
the Law or Follow Their Feelings? Texas Law Review 93:855-923.
Zweigert, Konrad, and Hein Kötz. 1998. An Introduction to Comparative Law (Tony Weir transl., 3rd ed.).
New York: Oxford University Press.
13
The following describes in more detail our subject pool (S1), the study task and materials (S2), our
experimental treatments (S3), the data collected including excluded observations (S4), and our statistical
methods (S5). The description is exhaustive; we did not perform other interventions, collect other
variables, or exclude other observations.
Judges who were inclined to participate were pointed to a URL on the computers we provided, or that
they brought themselves. The study was hosted online and began with the consent form, which described
the study’s purpose as “to learn about legal reasoning and the role of various factors therein” and
participants’ task as “judg[ing] a fictitious yet highly realistic international law case.” Invitation letters or
other materials distributed to judges ahead of the event (as described in Appendix A) also used only this
language to describe the study.
A total of 371 judges completed the initial consent, of which 367 proceeded to the main materials and
312 proceeded to submit a judgment; attrition rates are almost identical across treatment groups. 13 Four
Chinese participants withdrew their participation after the debriefing (which our IRB required us to allow
in Argentina and China). 14
12
In four jurisdictions (Argentina, Brazil, France, Germany), the seminar was also open to prosecutors, whom we
exclude from the analysis.
13
Of the 59 judges who completed initial consent but did not submit a judgment, 11 are U.S. judges. These U.S.
judges might have failed to complete the study due to a computer (iPad) freeze, in which case they might also have
been given a replacement iPad with a new participation number. To the extent this happened, they would have been
double-counted in the number of completed consents and read instructions. When we did replace an iPad, we could
not control the treatment assignment on the new iPad, nor did we log which was a replacement iPad. To exclude
the possibility that a participant might have been exposed to different treatments on different iPads, we exclude the
two U.S. participations that began more than 55 seconds after the first unfinished U.S. participation was exposed to
treatment-specific information beyond the defendant’s name (when they proceeded to the main study materials).
The same problem occurred with laptops used in one of the four rounds of German data collection, leading to one
exclusion.
14
Our IRB considered “judicial decision-making” incomplete disclosure of our research goals for the Argentinian and
Chinese judges. Our IRB therefore demanded that we grant participants the right to withdraw their data from the
study upon completion and full debriefing about the details of the study.
S-1
To guarantee anonymity and save time, we collected very little demographic information and asked few
additional questions. Some questions were not asked in all countries because of country-specific
sensitivities, in particular about identifiability. Table S1 summarizes all the information we have beyond
the outcome data discussed in the main text. Different countries’ participants differed along various
characteristics including age, gender, and career path. To the extent these differences are systematic,
however, they more likely reflect differences between the respective national judiciaries than differences
in our sampling scheme. For example, our U.S. participants are considerably older than other participants,
but this is a feature of recruitment into the U.S. federal judiciary (lifetime appointment after a successful
career elsewhere), not of recruitment of U.S. federal judges into our sample. Such differences entail that
one cannot pin down why judges from different countries differ (demographics, training, institutional
embedding, etc.), but they do not impede the study of the existence of differences, which is our focus.
15
In the USA and one German round, some iPads or laptops, respectively, froze due to network issues. When this
happened, the assistants present usually offered the participant a replacement machine. In the USA, we did not have
the technical ability to transfer frozen participations to new machines; in Germany, we did have it but the assistants
might not have used it. Hence the new machine would only have registered the participants’ document views from
then on. Worse, it is likely that the participant was transferred to a different treatment group. To eliminate both
possibilities, we exclude any participant in those sessions who commenced their participation after the first
unfinished participation in that session—among those who got at least to the document view stage, and hence were
exposed to the treatment—stopped recording data.
16
The latter only wrote “Not sufficient time to form a judgment.” The former’s reasons began “Having completed
this study, it is impossible for me to adjudicate adequately, for the following reasons: …”
S-2
Once a participant submitted the affirm/reverse decision and reasons, we also asked what sentence
length the participant would deem appropriate (if the participant had decided to reverse, we explained
that the participant should imagine having been outvoted on a panel).
We chose an international criminal case to ensure that the task would be comparable yet accessible for
judges from different jurisdictions. We chose an appeals case to focus on legal reasoning. We chose a
17
The U.S. instructions asked the participants "to judge whether the defendant is or is not guilty." This language is
technically imprecise because the direct determination of guilt is generally considered a question for the trial court,
whereas appeals courts decide whether to reverse or affirm.
S-3
We derived our case from a real case at the International Criminal Tribunal for the former Yugoslavia
(ICTY), Prosecutor v. Perišić. The main question in Perišić – in our setup, the only question – was whether
a conviction for aiding and abetting under Article 7(1) of the ICTY Statute requires that the aid be
“specifically directed” at the war crime, or whether any substantial contribution is sufficient. Defendant
Momčilo Perišić had been the highest ranking general of Yugoslavia for much of the Bosnian war. In this
capacity, he had been responsible for organizing various types of Yugoslavian support for the Army of the
Republika Srpska (VRS). The VRS was the main armed group of ethnic Serbs in the Bosnian war and
committed various war crimes in Bosnia, including the notorious Srebrenica massacre. Yugoslavian
support for the VRS included personnel and arms. In 2011, the ICTY’s Trial Chamber convicted Perišić as
an aider and abettor to the VRS crimes. 19 In a controversial decision from 2013, the ICTY Appeals Chamber
reversed, holding that aiding and abetting required the aid to be “specifically directed” at the crimes. 20
Perišić had had knowledge of the VRS war crimes when providing substantial support to the VRS. But the
Appeals Chamber found that Perišić’s support was directed merely towards the general war effort of the
VRS, not specifically towards its war crimes.
We provided the original Perišić trial judgment of the ICTY Trial Chamber in the materials, except that we
changed the names and some biographical information as described below. 21 We also provided the
original ICTY statute and one redacted original precedent from the ICTY Appeals Chamber, as described
below. We wrote the “Agreed Facts” and the briefs reproduced in Appendix B.2-B.4 with the goal of
focusing the judges on only one legal issue, namely the reach of aiding and abetting liability under Article
7(1) explained above. 22 The original materials were in English; they were professionally translated into the
respective local languages for all but the US and Indian participants. The original statute, trial judgment,
and precedent, as well as our briefs, were presented with the official seal of the tribunal and other
trappings of official legal documents at the ICTY (in the case of the originals, simply in the form found on
the ICTY’s website). While the briefs and statement of facts, which the instructions advised to read first,
were short (under 1,000 words each), the trial judgment (more than 16,000 words), precedent (more than
18
Participants’ average affirmance rates cannot be explained by the position of their respective national laws. To
our knowledge, there is good authority favoring our defendant only in Germany (BGH NJW 2001, 2409 at III.2) and
Argentina (cf. CFCP, Register 371/15, FSA 44000195/2009/19/1/CFC2, 'Blaquier, Carlos Pedro Tadeo y otro s/recurso
de casación', 13 March 2015, [concurrence of Justice Gemignani]), yet German and Argentinian judges’ affirmance
rates were above average (81% and 94%, respectively). The question was expressly left open by the U.S. Supreme
Court in Rosemond v. United States, 572 U.S. 65, 77 n.8 (2014). Most likely, our participants were impressed by the
gravity of the war crimes in question, which probably far exceeds anything they may encounter in their domestic
courtrooms.
19
Prosecutor v. Momčilo Perišić, Case No. IT-04-81-T, judgment of the Trial Chamber of 9/6/2011.
20
Prosecutor v. Momčilo Perišić, Case No. IT-04-81-A, judgment of the Appeals Chamber of 2/28/2013.
21
We changed the date to January 2014 to make it a live issue, and we omitted the parts relating to Zagreb because
it proved too difficult to find a credible mirror city targeted by ethnic Croats.
22
Towards this goal, the statement of facts was entitled “Agreed Facts” and began with the sentence: “The parties
have agreed that the following key facts are not in dispute.” Similarly, the brief for the appellant began with the
words: “This appeal concerns a single point of law: whether or not aiding and abetting under Article 7(1) of the
Statute governing this Tribunal requires that the assistance be specifically directed to the commission of a crime.”
Both briefs focused on this issue alone. They discussed the precedent and the policy issues. They cited specific
passages of the precedent that could be accessed directly using hyperlinks.
S-4
Precedents
To help the reader, we label the three precedents in this paper by the direction of the decision that they
would support: Affirm, reverse, and REVERSE (capital letters indicating greater strength). All three were
actual prior decisions of the ICTY Appeals Chamber, but any participant’s materials were written as if the
respective other two precedents did not exist.
The two precedents that we used with all study groups were Affirm and reverse. reverse was helpful to
the appeal’s legal argument because it defined aiding and abetting as “specifically directed to assist … the
perpetration of a certain specific crime.” 23 By contrast, Affirm was harmful to the appeal’s legal argument
because it held “[t]hat ʻspecific directionʼ is not an element of aiding and abetting liability under customary
international law.” 24 Under two well-established legal principles, however, neither of these was
technically binding precedent even for a judge who generally adhered to the principle of stare decisis (i.e.,
binding precedent). First, both precedents were “distinguishable” because they involved materially
different facts, as neither of them dealt with the issue of support provided from a separate military
organization based in a different country. Second, the legal pronouncements on “specific direction” in
reverse and arguably also in Affirm were “obiter dictum”— i.e., not necessary to reach a judgment in the
case—because the accused’s acts in those cases had in fact been specifically directed at the crime;
reverse’s engagement with “specific direction” was also concomitantly short and limited to the quoted
definition. Nevertheless, both precedents were persuasive authority for their points, particularly Affirm,
which proclaimed that its treatment of “specific direction” was not dictum and dedicated over twenty
pages to it. 25 In the real world, legal argument in general and judges’ written reasons in particular appeal
to such persuasive authority all the time. In the absence of any better authority, we expected these
precedents to have a sizeable effect, and a survey of law professors shared our view (Spamann & Klöhn
2016).
23
Prosecutor v. Vasiljević, Case No. IT-98-32-A, at para. 102 (emphasis added).
24
Prosecutor v. Sainović, Case No. IT-05-87-A, at para. 1649.
25
Cf. id. at 646, note 5320 (“the issue of specific direction cannot be circumvented in determining the outcome of
the present case”), and generally 643-668 (discussing specific direction).
S-5
We made some minor modifications to the precedents to fit them into our case, as described in Spamann
& Klöhn (2016).
Defendants
Our two fictitious defendants differed only in their nationality, biography, and attitude. We chose these
attributes and their depiction to be clearly irrelevant from a strictly legal perspective, as further discussed
in Spamann & Klöhn (2016). We named these defendants Borislav Vuković (a fictitious unsympathetic
Serb) and Ante Horvat (a fictitious sympathetic Croat), but in this paper refer to them simply as the
sympathetic and unsympathetic defendant to help the reader.
The unsympathetic defendant's facts and trial judgment were identical to the original Perišić facts and
trial judgment except for some locations and names, as described in Spamann & Klöhn (2016). We made
only two substantive, fictitious changes to the Perišić original. First, we added the following biographical
sentence to the statement of facts and the trial judgment facts: “He held this position [as the army’s chief
of staff] until his mandatory retirement from the [army] in 2004, when he became advisor to the
[Yugoslavian] government for ‘the rehabilitation of Serb victims of Albanian persecution’ and chairman of
the United Serbia Party.” Second, the fictitious brief for the prosecution noted in its closing passage that
the defendant “has publicly mocked this tribunal and repeatedly inflamed lingering tensions with
inflammatory public statements showing absolutely no regrets about the horrors of the war in general,
and the war crimes he supported in particular.” This statement was supported by a footnote. See the full
documents in Appendix B for details.
The sympathetic defendant’s facts and trial judgment were identical to the unsympathetic defendant’s,
with the following three exceptions. First, we changed all names of Yugoslav and Bosnian-Serb persons,
institutions, and places to their Croat and Bosnian-Croat equivalents, as described in Spamann & Klöhn
(2016). Second, our (fictitious) facts noted in passing that the sympathetic defendant had contacts with
26
In this connection, it is worth mentioning that the Affirm variant of the prosecution's response brief in the U.S.
experiment contained an unintentional typo ("required" instead of "rejected") in its second out of three references
to the precedent's holding (see Appendix A.1.4, paragraph 4). According to their written reasons, however, none of
the U.S. judges were mislead by this. We corrected the mistake before administering the study in the other
jurisdictions.
27
We changed the defendant’s name to a fictitious Emir Besić instead of Momčilo Perišić and made him the deputy
chief of staff of the Army of the Republic of Bosnia and Herzegovina (ArBiH) instead of Yugoslavia. We slightly
rewrote the rest of the facts to fit this setup. In particular, our fictitious facts described Bešić as responsible for
coordinating the ArBiH’s activities with those of the (fictitious) Zeleni Sokoli (“ZS”), a paramilitary group consisting
of Bosniak and foreign fighters. In that role, however, our fictitious Bešić did exactly the same for the ZS as the real
Perišić did for the VRS, and the fictitious ZS committed the same crimes as the real VRS.
S-6
Anchor
After the participants submitted their affirm/reverse decisions and associated reasons, we asked them to
determine the appropriate sentence length. For calibration, we first provided two examples of sentences
(measured in years) given to real convicted war criminals. In the US version of the experiment, we then
simply provided an input line demanding a number and prepopulated that line with 10, 25, or 40, which
were randomly assigned to participants. We expected participants to enter their judgment in years, as in
our examples. To eliminate any possible confusion about the time unit, however, we slightly changed the
setup in the other countries, adding the following language after the calibration examples: “To avoid
misunderstandings, please write the time units (years or months) explicitly. For example, if you thought
that [10/25/40] years were an appropriate penalty, you should write either ‘[10/25/40] years’ or
`[120/300/480] months,’” again randomizing the number inputs.
The process data consist of separate series of document requests and document visibility. The former
records, with exact time stamps, judges’ clicks requesting to view documents in the materials. The latter
records which individual paragraphs are live on a judge’s screen at roughly 10 second intervals. 29 As
mentioned in the main text, our analysis focus on the click data because document-level information is
sufficiently fine-grained for our purposes (the paragraph-level data is available in our open access
repository). In fact, we even omit some further detail that seems irrelevant for our inquiry: we collapse
the two briefs into one document category; we omit all table of content views (which is every judge’s
starting point, but occurs only eleven times after the start with a mean of 13s and a maximum of 26s
duration); and we chop off instruction views at the beginning of every judge’s participation (the remaining
instruction views are shown in figure 1 but not mentioned in its legend because they seem unimportant
to understand judicial though processes and only comprise 0.81% of judges’ time). We then normalize the
28
Our idea was that NATO would trigger an in-group bias with judges from NATO countries, i.e., France, Germany,
and the United States. But informal conversations suggest that even Chinese today are at least neutral towards
NATO, NATO’s bombing of the Chinese embassy during the Kosovo conflict notwithstanding.
29
The spacing is not always exact because the experiment was hosted online and connections could trigger
somewhat irregular recording.
S-7
Text data (judgment reasons) was submitted by all but nine of the 299 participants. Expert translators
translated into English all text written in another language—that is, all text from civil law participants. The
English texts are reproduced in Appendix C, along with the names of the translators, where applicable.
Two US-American and two German law students (who were blind to the research questions)
independently coded these texts using the coding protocol included as Appendix D. In the paper, we focus
on the three features that we consider to be the most basic and thus of first-order importance for judicial
style: whether a judge mentions the statute, the precedent, and/or policy reasons. (We would get very
similar results if we considered only references to a specific precedent or even distinguishing of precedent
instead of any generic mention of precedent.) The intercoder reliability is almost perfect for the first two
(κ=0.84 and κ=0.91) and good for the third (κ=0.59); to the extent there is disagreement, we average the
scores of the four coders.
The outcome data are straightforward and require only four further explanations. First, as explained supra
section S1, we corrected three judgments in function of their associated reasons without any impact on
our results. Second, we have only 268 sentences because the sentencing task was part of the exit survey
that participants only reached after submitting a judgment and could then skip. Third, the US sentences
did not have explicit units. In the US, we had not asked judges to specify the unit when submitting their
sentence. Nevertheless, we believe there is no confusion that they were entered in years, as we expected,
because the calibrating examples we gave to judges in the US and elsewhere (see Appendix B.6) use years
as the unit, and none of the sentences submitted by US judges is divisible by 12 and only three are divisible
by 6. Fourth, five US sentences seem “fat finger” data entry errors: 209, 210, 1040, 1540, 11019. Our
reason to think they are errors are their extreme size (the remaining sentences’ maximum, mean, and
median are 110, 25, and 20, respectively), indivisibility by 10 or 12 or, except one case, even 6 (if judges
had entered the sentence in months), and our use of iPads without a physical keyboard only in the U.S.
and partially in Argentina and China, which facilitated typographical mistakes. Excluding these outliers
makes no meaningful difference for the rank-correlation reported in the main text or for the quantile
regressions or log regressions reported below, but it does matter for simple regressions without the log
transformation as reported in S5 below. Finally, for the sake of transparency, figures S1 and S2 show the
outcome data by country:
30
The mean time spent with the documents is 35 min. with a s.d. of 10 min. Country means are Argentina 37, Brazil
32, China 28, France 37, Germany 36, India 40, USA 36. Chinese judges are by far the quickest. Higher reading speed
for logographic relative to phonetic script might be an explanation, but we have not found any research backing up
this conjecture.
S-8
0 .2 .4 .6 .8 1
Rate
Defendant: both unsympathetic sympathetic
10
Brazil 25
40
10
China 25
40
10
France 25
40
10
Germany 25
40
10
India 25
40
10
USA 25
40
0 10 20 30 40 50
Sentence (years)
Medians (red marker), 25th and 75th percentiles (box), and adjacent values (whiskers).
Five U.S. outlier sentences (11019, 1540, 1040, 210, and 209 years) omitted.
S-9
Reasons: Rank Correlations; Permutation Tests of Pillai’s Trace and Modified Hotelling’s T2
A complication in working with the reasons data arises from coder disagreement, even though there is
not much of it (supra section S4). For lack of a better alternative, we average over our four coders. This
transforms what is in principle a binary variable into a quinary variable (with values 0, ¼, ½, ¾, and 1). For
univariate tests, this only requires small technical adjustments, specifically the use of the Spearman rank
correlation and associated test instead of a Fisher or other exact test to assess associations between
document views and mentions in reason. For multivariate tests, however, the theoretical consequences
are graver even though, ultimately, their practical import turns out to be minor. We need multivariate
tests because we have three outcome variables: statute, policy, and precedent.
Multivariate tests generally rely on asymptotics or very restrictive distributional assumptions that are not
met by our data. If our data were binary, we could easily construct exact non-parametric tests using
permutations because the binomial distribution only has one parameter, such that any null hypothesis
concerning this one parameter entails the randomization null hypothesis, whereas permutation tests are
exact against the randomization null (Lehmann and Romano 2005, Theorem 15.2.2). However, for more
general distributions such as our multinomial distribution, the randomization null is more restrictive than
a null concerning only one parameter, such that the permutation test is no longer necessarily exact. We
believe that the relevant null hypothesis is equality of means, not equality of distributions, because the
latter also depends on the distribution of coder disagreement, which could well differ between countries,
for example because of writing styles or the mere fact that reasons for some but not all countries had to
be translated.
Our solution is to conduct permutation tests of statistics that confer some robustness against deviations
from the randomization null hypothesis. To test for differences between common and civil law, we use
the “modified Hotelling’s T2” statistic m’V-1m, where m is the 3×1 column-vector of mean differences
between the two groups and V is the weighted sum of the group-specific covariance matrices, with
weights equal to the inverse of the respective group sizes. We compare the realized modified Hotelling’s
7
T2 against its permutation distribution from the � � = 21 permutations of country-clusters, i.e., of
2
country membership in the common law group, not of individual judges’ nationalities (the justification for
this approach is the same as for common/civil law tests of document view paths explained in the previous
subsection). Chung & Romano (2016) show that this test is asymptotically valid against the less restrictive
null hypothesis—compared to the randomization hypothesis—that the parameters of interest (in our
case, the means) are equal. In any event, in our data the test fails to reject even the stricter randomization
null hypothesis, against which, recall, the permutation test is exact.
For tests of country differences, the modified Hotelling’s T2 is unsuitable because it only works for two
groups, not seven. Instead, we conduct the permutation test using Pillai’s trace and 10,000 draws from
S-10
To estimate absolute and relative effect sizes of defendant and precedent and their respective confidence
intervals, we need a more parametric model. To allow for country differences in baseline affirmance rates
without having to estimate them consistently, we condition on the number of affirmances in each country
in a conditional logit model (also known as fixed-effects logit or conditional logistic regression for matched
case–control groups). Let subscripts c and i denote countries and participants, respectively, and define
ℎ𝑖𝑖𝑖𝑖 ≡ 𝛽𝛽𝐷𝐷 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠ℎ𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑖𝑖𝑖𝑖 + 𝛽𝛽𝑃𝑃1 reverse𝑖𝑖𝑖𝑖 + 𝛽𝛽𝑃𝑃2 REVERSE𝑖𝑖𝑖𝑖 , where sympatheticic etc. are dummy
variables indicating whether participant i in country c was assigned the sympathetic defendant etc. (the
baselines being the unsympathetic defendant and the Affirm precedent, without loss of generality).
Further let k be the number of affirmances in country c, 𝒮𝒮𝑘𝑘𝑐𝑐 the set of size-k subsets of participants from
country c, and 𝑓𝑓𝑐𝑐 ≡ ∑𝑆𝑆∈𝒮𝒮𝑘𝑘𝑐𝑐 exp{∑𝑖𝑖∈𝑆𝑆 ℎ𝑖𝑖𝑖𝑖 }. The log-likelihood to maximize over 𝛽𝛽𝐷𝐷 , 𝛽𝛽𝑃𝑃1 ,𝛽𝛽𝑃𝑃2 is
The reported odds ratios and associated confidence intervals are the exponentials of the maximum
likelihood estimates of 𝛽𝛽𝐷𝐷 , 𝛽𝛽𝑃𝑃1 ,𝛽𝛽𝑃𝑃2 and of their normal-based confidence interval boundaries.
Sentences
Unlike our other outcome data, sentences are a continuous variable. We can still use the Spearman rank
correlation for a univariate non-parametric test of simple association, and report this in the main text.
Table S.2 additionally provides regression estimates, which are more meaningful representations of effect
sizes. The regression is also helpful to control for country differences and other treatments, which reduces
estimation error even though it is not necessary for consistency (the treatments are orthogonal to one
another and to countries by construction, and our sample is nearly perfectly balanced).
S-11
The first three models (1)-(3) use a continuous measure of the anchor, whereas their variants (1’)-(3’) use
dummy variables for the three anchor variables (one of them, 15, being the omitted baseline), with very
similar results.
Using years (not log years) as the outcome variable requires a technique to dampen the impact of the
extreme outliers 11019, 1540, 1040, 210, and 209 years. They would otherwise overwhelm any other
patterns in the data in a simple linear regression given that the other data points’ maximum, median,
mean, and standard deviation are only 110, 25, 23, and 14, respectively. As explained in section S4 above,
we have every reason to think that these extreme outliers are data entry errors and thus feel comfortable
excluding them in models (1) and (1’). In any event, the results of the other regression models do not
depend on this exclusion.
S-12
Argentina. In Argentina, we recruited judges from civil, commercial, family and criminal courts both at the
trial and appellate levels, attending one of two seminars in “Law and Neuroscience.” The first of these
seminars was a full-day event organized by one of the authors (IR) at the Universidad Torcuato di Tella in
Buenos Aires in November 2015, and all invitees were informed weeks in advance about the opportunity
to participate in an experiment and that such participation would be voluntary and independent of the
seminar. Upon arrival, those wishing to participate in the experiment were randomly assigned to one of
two groups: group A completed the experiment in the university’s computer lab from 12:30 pm until 1:30
pm while group B was offered lunch, and group A was offered lunch while group B participated in the
experiment in the lab from 1:30 pm until 2:30 pm. The authors were not present, but student assistants
remained in the experiment room to provide technical support regarding the use of lab computers and
did not otherwise interact with the participants. Besides the student assistants, everyone else in the room
was an active participant in the experiment. Prosecutors were also invited to participate in the
experiment, but their submissions have not been included in the results presented in this paper.
The second seminar took place in March 2016 from 2:30 pm until 6:15 pm, and was organized by the
Magistrate’s Association of San Isidro, a district in the northern part of the state of Buenos Aires. All
invitees were informed weeks in advance about the opportunity to participate in an experiment and that
such participation would be voluntary and independent from the seminar. The experiment was conducted
during a coffee break at 3:15 pm on laptops that participants brought themselves, or, for a limited
number, on iPads that we provided.
Brazil. For Brazil, we organized a “Workshop: How judges reason?” as part of the Harvard Law Brazilian
Association Legal Symposium held in April 2018. The Symposium was organized by current students and
alumni of Harvard Law School from Brazil with the support, among others, of the Association of Brazilian
Federal Judges of the 2nd Regional Court. It brought to campus dozens of Brazilian judges who travelled at
their own expense. Our 80-minute workshop was the first session of the day and open only to judges, who
had been specifically invited to come and told to bring a laptop if they had one (most did). We provided a
laptop computer to all participants who did not have their own. Following an extremely brief and generic
welcome and invitation to participate by one of us (HS, who stayed in the room), judges worked on the
study for up to an hour. We then debriefed.
China. In China, we recruited judges from local and intermediate courts in Zhejiang Province attending
academic writing classes at Judges College in Hangzhou in April 2016. 31 By way of background, the Chinese
judiciary encourages judges to write academic articles. Every court above the intermediate level publishes
its own journal with academic work of judges. Most articles are closely related to judicial practice, such as
new problems or hard cases that the judges encountered in their practice. But the journals also publish
more theoretical and sometimes even descriptively statistical work. Judges write in these journals for
fame and reputation, which can relate to promotion. The papers’ quality is also seen as an indicator of a
court’s performance. The College offers two-day courses in such writing. Journal editors and judges with
good publication records lecture about their writing technique and research experience. Applicants for
31
We had previously attempted to recruit judges by email invitation forwarded by chief judges of certain courts. This
proved unsuccessful, however, as only 24 judges completed the experiment and gave us permission to use the data.
A-1
France. In France, we conducted the study as part of a management course (CADEJ) for experienced
judges and other justice professionals (prosecutors, senior clerks, etc.) wishing to assume administrative
responsibilities in the French judicial system. CADEJ is organized under the auspices of the French national
judge school (Ecole Nationale de la Magistrature, ENM). Participating judges come to Paris for three days
of classes once a month from September 2017 through June 2018. We were given access to the morning
session of the third day of the November meeting, which was dedicated to digitalization. Our particular
session was exceptionally held in the rooms of Sciences Po law school. The ENM organizers had informed
the participants that one of us [HS] would be offering participation in an “experiment,” followed by a
presentation by another one of us [CJ] on “The Act of Judging, the Motivations of the Judge, and the
Unpredictability of Decisions.” When the judges arrived, they found every seat equipped with a laptop
computer. After brief words of welcome from one of the ENM organizers and one of us (CJ, who is the
dean of the venue), another one of us (HS) invited the participants to participate in a study aiming “to
collect scientific information about the act of judging.” CJ and HS remained in the room during the
experiment but did not answer questions; technical questions were answered by technicians in the room.
Of the 65 participants in the room, 63 initially participated, but 5 cancelled before completion (at least
one of them by accident).
Germany. In Germany, we offered the study as a spare time activity to judges participating in three
training programs at the German Judge Academy (Deutsche Richterakademie) in Trier in September,
October, and November 2017, respectively, and one training event for young judges organized by the
Ministry of Justice of the state of Lower Saxony at Bad Nenndorf in June 2018.
In Trier, on one day of the training program, one of us (AM) would briefly address the judges at the lunch
break, which gathered all judges in one room, to invite them to participate in “a study by researchers from
Harvard University, Humboldt University, and the MPI for Research on Collective Goods. The purpose of
the study is to learn about legal reasoning and the role of various factors therein.” The judges were also
given a flyer with this information and promised that there would be a debriefing with wine and crackers
following the actual study. In the evening after their regular sessions interested judges gathered in a
dedicated room where a mobile computer lab had been set up. Judges were seated in front of the laptops.
One of the authors [AM] welcomed them, reminded them of the information provided over lunch, pointed
out that participation was voluntary and referred them to the written instructions on their screens. One
(first two session) or two (last session) student assistants were available for questions arising during the
session. They were familiar with the hardware and had instructions to answer foreseeable questions in a
standardized way. However they were not informed about what the study was about. AM stayed in the
room and was available for the students in case they were approached with questions they could not
answer.
A-2
India. We used the good offices of Gujarat National Law University in Gandhinagar and the Chief Justice
of the Gujarat High Court to create and offer a one day workshop on “Behavioural Law and Justice: Aspects
of Negotiations and Decision-making,” where several of us (VK, PM, HS) taught. Invitations were sent to
Chief Justices in all Indian states with a request to nominate district judges from their state for the
workshop. The Gujarat High Court covered the judges’ travel expenses. Upon arrival at the workshop,
judges received an invitation letter that explained that “[t]he linchpin of the conference will be a
simulation of an international law judgement” and encouraged them to participate (while emphasizing
that participation was not mandatory). We provided computers to all participants.
USA. We conducted the U.S. session at an annual three-day workshop for U.S. federal judges organized
jointly by Harvard Law School and the Federal Judicial Center in April 2015. The workshop is open to all
U.S. federal judges including circuit judges, district judges, bankruptcy judges, and magistrates. 32 The
experiment was part of a session on “Behavioral Research on Judicial Decision-Making” in the middle of
the second morning. Several weeks earlier, the judges had received an invitation to the experiment with
all consent-relevant information and a reading “assignment”: Guthrie, Chris, Jeffrey J. Rachlinski, and
Andrew J. Wistrich, Blinking on the Bench, Cornell Law Review 93:1-43 (2007), which discusses biases in
judicial fact-finding. The experiment was administered on iPads we provided to the judges.
One of the authors (HS) welcomed the judges, reminded them of the experiment as described in the
invitation letter, and pointed them to the iPads. Three student assistants distributed and collected the
iPads and were available for help with technical questions; they did not know what the experiment was
about. HS stayed in the room but did not interact with the subjects. Participation was voluntary but all the
judges present in the room participated in the experiment. 33
32
At the time of the experiment, no circuit judge may have been present. We refrained from collecting this
information out of concern for preserving anonymity.
33
We lost some small number due to technical problems, see Spamann & Klöhn 2016 for details.
A-3
1. Instructions
Please imagine you are an appeals judge in the case Prosecutor v. [NAME] pending at the
International Criminal Tribunal for the Former Yugoslavia (ICTY). This case is fictitious but very
closely resembles an actual case recently decided by the ICTY. The ICTY is an international tribunal
with the power to prosecute persons responsible for serious violations of international
humanitarian law committed in the territory of the former Yugoslavia since 1991 in accordance
with the provisions of the ICTY Statute. [As an international tribunal, the procedure of the ICTY
combines elements from common law and from civil law systems, some of which may seem
unfamiliar to you. – VISIBLE TO ONLY HALF THE SUBJECTS]
You have already presided over several hearings. The prosecution and the defence have now
submitted their final appeals briefs and agreed on a list of agreed facts.
Your task is to judge whether to affirm or reverse the trial court's conviction of the defendant for
aiding and abetting 34 various war crimes by the [RELEVANT MILITARY GROUP] on the territory of
Bosnia-Herzegovina in the years 1993-1995. 35
In reaching your judgment, you will be able to peruse the aforementioned briefs and the list of
agreed facts. I recommend you read these in full. The briefs link to other documents, namely the
decision of the trial court below, a recent decision by the Appeals Chamber in another case, and
the statute establishing the ICTY. These other documents are very long. You will not have time to
read them in full, but you may pursue a handful of further passages that you deem particularly
relevant.
Please do NOT access any information on another device such as your smart phone, and please
do NOT talk to your neighbors until the study is completed.
You have 50 minutes to reach a decision and submit a brief summary of your reasoning, either on
this computer or on a separate piece of paper marked with your participant number, which will
be randomly generated and shown on the screen. 36 To help you keep track of time, a clock on the
screen will count down the 50 minutes.
By clicking on the button below, you will proceed to an index page with all the documents
provided. You can at any time return to this introduction or to the index page by clicking the
34
[The U.S. version up to here read: “Your task is to judge whether the defendant is or is not guilty of aiding and
abetting …”]
35
[In the U.S., Argentinian, and Chinese versions, the years “1993-1995” were erroneously given as “1992-1994.”]
36
[The Argentinian, Chinese, French, German, and U.S. versions also included the following words at the end of this
sentence: “, either on this computer or on a separate piece of paper marked with your participant number, which
will be randomly generated and shown on the screen.”]
B-1
B-2
The parties have agreed that the following key facts are not in dispute.
[Borislav Vuković / Ante Horvat] was born on 22 May 1944 in [Koštunići, Serbia, in the Socialist
Federal Republic of Yugoslavia / Skradin, Croatia]. After joining the Yugoslav People’s Army, he
graduated from the Ground Forces Military Academy in 1966 and became an officer. Shortly after
the conflict in the former Yugoslavia began, [Vuković / Horvat] became the Chief of Staff and then
Commander of the 3rd Army within the [Yugoslav / newly formed Croatian] Army (“[VJ/HV]”)
based in [Niš, Serbia / Knin, Croatia]. On 26 August 1993, the President of [the Federal Republic
of Yugoslavia (“FRY”) / Croatia] appointed [Vuković / Horvat] as Chief of the [VJ / HV] General
Staff, a position which made him the most senior officer in the [VJ / HV]. He held this position
until his mandatory retirement from the [VJ / HV] in 2004, when he became [advisor to the FRY
government for “the rehabilitation of Serb victims of Albanian persecution” and chairman of
the United Serbia Party / Croatian vice-chairman of the Croatian-Bosnian Reconciliation
Commission].
THE INDICTMENT
[Vuković / Horvat] was charged on the basis of individual criminal responsibility (Article 7(1) of
the Statute of the International Criminal Tribunal for the former Yugoslavia, hereinafter “the
Statute”) with aiding and abetting crimes in the Bosnian towns of [Sarajevo / Mostar] and
[Vlasenica / Ahmići] by facilitating the provision of military and logistical assistance from the [VJ /
HV] to the [Army of the Republika Srpska (“VRS”) / Defence Council of the Hrvatska Republika
Herceg-Bosna (“HVO”)]. The [VRS / HVO] was an armed group of ethnic [Serbs / Croats] in the
Bosnian civil war. [Vuković / Horvat] was also charged with superior responsibility (Article 7(3) of
the Statute); since he was acquitted of these charges and this part of the decision is not on appeal,
however, no further mention will be made of the facts underlying this part of the indictment.
The underlying events took place in the territory of Bosnia and Herzegovina in the period between
August 1993 and November 1995.
[SARAJEVO / MOSTAR] From September 1992 to November 1995, the [VRS / HVO] conducted a
lengthy campaign of shelling and sniping in [Sarajevo / Mostar] which resulted in the deaths of
hundreds of civilians and the wounding of thousands of others.
[VLASENICA / AHMIĆI] In the summer of 1995, the [VRS / HVO] invaded the town of [Vlasenica /
Ahmići], which the United Nations Security Council had previously established as a safe area for
civilians. After taking over [Vlasenica / Ahmići], the [VRS / HVO] proceeded to forcibly remove and
massacre hundreds of Muslim civilians and persons not taking an active part in hostilities.
B-3
Logistic assistance notably included vast quantities of infantry and artillery ammunition, fuel,
spare parts, training and technical assistance. The Supreme Defence Council of [the Federal
Republic of Yugoslavia / Croatia] granted [Vuković / Horvat] and the [Yugoslav / Croatian] Army
the authority to provide logistic assistance to the [VRS / HVO]. Even though [Vuković / Horvat]
was not officially a member of the Supreme Defence Council, he participated in the Council’s
meetings, along with its members, notably [Slobodan Milošević and Zoran Lilić / Franjo Tudjman],
who at the time held the title[s] of President of [Serbia and President of the Federal Republic of
Yugoslavia, respectively / Croatia]. [Vuković / Horvat] regularly urged the Council to continue
providing logistic assistance to the [VRS / HVO], insisting that they could not wage war without
significant military support.
A large number of [VRS / HVO] officers were drawn from the ranks of the [Yugoslav / Croatian]
Army. They officially remained members of the [Yugoslav / Croatian] Army even as they were
fighting in Bosnia under the banners of the [VRS / HVO]. [Vuković / Horvat] proposed and carefully
implemented the idea of creating “Personnel Centres” to regularise the status of these officers
and allow them to lawfully remain part of the [Yugoslav / Croatian] Army. [VRS / HVO] officers
retained their salaries and benefits as [Yugoslav / Croatian] Army members through what was
known as the 30th Personnel Centre. [Vuković / Horvat] was well aware that the payment of
salaries was, in his own words, of “great help” to the [VRS / HVO].
[Vuković / Horvat] knew that the [VRS / HVO]’s operations encompassed grave crimes against
civilians. [Vuković / Horvat] received information from a variety of sources concerning the [VRS /
HVO]’s criminal behaviour and discriminatory intent against Muslims. Under [Vuković / Horvat]’s
direction, the [Yugoslav / Croatian] Army’s intelligence and security organs monitored the views
of the international community and international media concerning the conflict in Bosnia and
Herzegovina. [The Yugoslav Army General Staff also received diplomatic reports about
proceedings at the United Nations Security Council / During meetings with NATO to coordinate
enforcement of the UN’s no-fly zone against Serbian violations, Horvat also received briefings
on NATO intelligence] concerning grave abuses against civilians by [VRS / HVO] forces in [Sarajevo
/ Mostar] and other parts of Bosnia and Herzegovina. In particular, [Vuković / Horvat] was alerted
to the fact that the [VRS / HVO] was conducting a campaign of sniping and shelling against civilians
during its siege of [Sarajevo / Mostar]. These regular attacks were well documented and widely
reported for a period of three years.
On 7 January 2014, the Trial Chamber found defendant-appellant [Vuković / Horvat] guilty of
aiding and abetting the following crimes committed by members of the [VRS / HVO] in [Sarajevo
/ Mostar] and [Vlasenica / Ahmići]: murder, inhumane acts (injuring and wounding civilians,
inflicting serious injuries, wounding, forcible transfer), and persecutions as crimes against
humanity; and murder and attacks on civilians as violations of the laws or customs of war. The
B-4
B-5
I. Introduction
1. This appeal concerns a single point of law: whether or not aiding and abetting under Article 7(1)
of the Statute governing this Tribunal requires that the assistance be specifically directed to the
commission of a crime.
2. The prosecution agrees that the defendant must be acquitted if specific direction was required,
as the defendant merely provided unspecific support to the forces committing the crimes on the
ground.
3. The Trial Chamber, however, convicted the defendant because it did not consider specific
direction an essential element of aiding and abetting. We urge the Appeals Chamber to overturn
this decision because it is inconsistent with prior decisions of the Appeals Chamber, the most
fundamental principles of the Statute, and sound policy.
II. “Specific direction” is a component of the actus reus of aiding and abetting.
[Precedent: Affirm]
4. The Trial Chamber failed to take into account that aiding and abetting must be specifically
directed to assist the commission of crimes. Neutral acts providing general logistic and military
assistance to an army engaged in legitimate military operations do not qualify as aiding and
abetting.
5. It is true that not all prior decisions of the Appeals Chamber concerning aiding and abetting
explicitly mention the element of specific direction. We submit, however, that specific direction
has always been implicit in the finding that the accused provided practical assistance to the
principal perpetrator that had a substantial effect on the commission of the crime. Specific
direction was never in doubt in previous cases because the accused was at or proximate to the
crime scene.
6. In contrast, in cases where the conduct of the accused is remote in relation to the commission
of the crimes, the requirement of specific direction as an explicit element of aiding and abetting
is manifest. This is especially important in this case, as [Vuković / Horvat] is not accused of
providing assistance to the commission of crimes committed by the [VJ / HV]. Rather, he is
accused of facilitating the commission of crimes committed by the [VRS / HVO], a separate military
organization not under his personal command and not even part of the same command hierarchy.
7. The prosecution relies on the recent Appeals Chamber’s decision in Šainović37 to argue that
specific direction is not a component of the actus reus of aiding and abetting. However, as Judge
Tuzmukhamedov stated in his dissent, this issue was not actually relevant for deciding the case in
Šainović. 38 Furthermore the facts in Šainović are completely different from the facts at hand. In
37
Prosecutor v. Šainović, Case No. IT-05-87-A, Judgement 23.1.2014.
38
Prosecutor v. Šainović, Case No. IT-05-87-A, Judgement 23.1.2014, Dissenting Opinion of Judge Tuzmukhamedov,
at para. 40.
B-6
[Precedent: reverse]
4. If there was ever any doubt about the requirement of “specific direction,” it was laid to rest by
the Appeals Chamber’s Vasiljević decision handed down two weeks after the Trial judgment in the
present case. The Vasiljević decision expressly required “specific direction” as part of the actus
reus of aiding and abetting. Distinguishing aiding and abetting from joint criminal enterprise (JCE),
the Appeals Chamber stated in Vasiljević:
“The aider and abettor carries out acts specifically directed to assist, encourage or lend moral
support to the perpetration of a certain specific crime (murder, extermination, rape, torture,
wanton destruction of civilian property, etc.), and this support has a substantial effect upon the
perpetration of the crime. By contrast, it is sufficient for a participant in a joint criminal enterprise
to perform acts that in some way are directed to the furtherance of the common design.” 40
(emphasis added)
5. The Appeals Chamber thus explicitly acknowledged that criminal liability does not attach to the
mere assistance to a military group, even if that assistance had a substantial effect on the
commission of the crime. Rather, the actus reus of aiding and abetting is assistance specifically
directed at the commission of a crime.
6. [= Šainović para. 5]
7. [= Šainović para. 6]
8. [= Šainović para. 8]
[Precedent: REVERSE]
4. If there was ever any doubt about the requirement of “specific direction,” it was laid to rest by
the Appeals Chamber’s Bešić decision handed down two weeks after the Trial judgment in the
present case. Bešić expressly held that the actus reus of aiding and abetting requires “specific
direction” in a case with very similar facts.
39
Prosecutor v. Šainović, Case No. IT-05-87-A, Judgement 23.1.2014, Dissenting Opinion of Judge Tuzmukhamedov,
at para. 43.
40
Prosecutor v. Vasiljević, Case No. IT-98-32-A, Judgement 23 January 2014, at para. 102.
B-7
“that specific direction is an element of the actus reus of aiding and abetting liability, and that in
cases like this one, where an accused individual’s assistance is remote from the actions of principal
perpetrators, specific direction must be explicitly established.“ 41 (emphasis added)
6. The facts of Bešić closely resembled the present case, and the Appeals Chamber explicitly held
that they are not sufficient for aiding and abetting liability under the Statute. The accused in Bešić
was a high-ranking officer in the Army of the Republic of Bosnia and Herzegovina (ArBiH). He was
accused of aiding and abetting war crimes of another armed group, the Zeleni Sokoli (ZS), because
he managed the supply of weapons and personnel from the ArBiH to the ZS. The Appeals Chamber
entered an acquittal, holding that
“assistance from one army to another army’s war efforts is insufficient, in itself, to trigger
individual criminal liability for individual aid providers absent proof that the relevant assistance
was specifically directed towards criminal activities.” 42 (emphasis added)
7. It is hard to imagine a clearer rejection of the legal analysis in the Trial judgment in the present
case. Like Bešić, [Vuković /Horvat] is not accused of providing assistance to the commission of
crimes committed by the [VJ / HV]. Rather, he is accused of facilitating the commission of crimes
committed by the [VRS / HVO], a separate military organization not under his personal command
and not even part of the same command hierarchy. Bešić demands that the Trial judgment be
overturned and [Vuković /Horvat] be acquitted. To hold otherwise would not only misapply the
law but also do gross injustice by unequal treatment of [Vuković /Horvat] and Bešić.
8. Bešić ought to settle this case. It may be worth adding, however, that Bešić merely implements
the general principles governing the ICTY Statute. One of those principles is, as is generally
accepted, that the Statute does not criminalise the waging of war per se. States provide military
and technical assistance to one another with varying strategic objectives in a number of regions
around the world. However, this aid in itself does not render the leaders of the assisting states
individually criminally responsible for aiding and abetting crimes committed during such wars. To
be held individually criminally responsible, the leaders must be shown to have committed or aided
and abetted the commission of some crimes during the war, an act which is distinct, and apart,
from the mere provision of military assistance. To conclude otherwise, as the Trial Chamber has
done, is to criminalise the waging of war, which is not a crime according to the Statute of the
Tribunal. Any provision of weapons would result in the individual criminal responsibility of the
provider, approaching a form of strict liability.
9. Moreover, dispensing with the requirement of specific direction, as the Trial Chamber did, leads
to absurd consequences. It would potentially ensnare all military and political leaders who
approve logistical assistance to a foreign army without the power to control every decision of that
41
Prosecutor v. Bešić, Case No. IT-03-56-A, Judgement 23 January 2014, at para 73.
42
Id., at para. 72.
B-8
III. Conclusion
[HORVAT ONLY: 10. From the very beginning of this case, the defendant has expressed his deep
regret at all bloodshed in this tragic war, and in particular at the inexcusable crimes of certain
soldiers and officers in the field. He categorically denies, however, that he is personally
responsible for those crimes. We urge the Appeals Chamber to affirm that the law is on the side
of the defendant and others forced by history to make difficult decisions in times of war, and
to overturn the conviction by the Trial Chamber.]
[10 / 11]. For these reasons, [Vuković /Horvat] respectfully requests that the Appeals Chamber:
(i) hold that the actus reus of aiding and abetting in international law requires specific direction;
(ii) apply that standard; and (iii) reverse the Trial Chamber’s judgment and enter an acquittal.
B-9
I. Introduction
1. The defense requests that the Appeals Chamber reverse the Trial Chamber’s judgment on the
grounds that the assistance provided by [Vuković / Horvat] was not specifically directed at the
war crimes committed by the [VRS / HVO 43].
2. The defense’s argument is without merit. The Trial Chamber correctly found [Vuković / Horvat]
guilty of aiding and abetting under Article 7 of the Statute governing this Tribunal because
[Vuković / Horvat] provided logistical and personnel assistance to the [VRS / HVO] in full
knowledge that the [VRS / HVO] committed atrocious war crimes. 44
[Precedent: Affirm]
3. It is irrelevant that none of [Vuković’s / Horvat’s] acts were specifically directed toward the
commission of the war crimes by the [VRS / HVO]. Specific direction is not an element of aiding
and abetting. The defense’s argument to the contrary is a transparent attempt to introduce a
novel, restrictive element to the actus reus of aiding and abetting that has no basis in previous
cases and that would make it more difficult to convict those who knowingly facilitate the most
grievous crimes. We urge the Appeals Chamber to reject this attempt to undermine the very
purpose of this Tribunal to hold to account those responsible for the horrors of the Yugoslav wars.
4. If there was ever any doubt about the requirement of “specific direction,” it was laid to rest by
the Appeals Chamber’s Sainović decision handed down two weeks after the Trial Chamber’s
judgment in the present case. The Sainović decision expressly rejected 45 “specific direction” as
part of the actus reus of aiding and abetting. After an exhaustive discussion of the national and
international case law, including all relevant decisions by the Appeals Chamber, the majority,
Judge Tuzmukhamedov dissenting, came to the conclusion:
“[t]hat ʻspecific directionʼ is not an element of aiding and abetting liability under customary
international law.” 46
5. The prosecution agrees that the defendant’s physical distance from the crime scene may be
relevant for aiding and abetting liability. However, this follows from the simple fact that in these
situations the assistance is less likely to have a substantial effect on the main act. The prosecution
submits that the proximity of an alleged aider and abettor to crimes committed by the principal
perpetrators is one factor that a trial chamber may consider in determining whether substantial
contribution is established. If such effect is established, however, there can be no doubt that the
43
[In the U.S., Argentinian, and Chinese versions, “HVO” was erroneously replaced by “VRS.”]
44
[This was followed by the redundant words ", as [Vuković / Horvat] knew fully well" in the Argentinian, Chinese,
German, and U.S. versions.]
45
[The U.S. version read “required” instead of “rejected.”]
46
Prosecutor v. Šainović, Case No. IT-05-87-A, Judgement 23.1.2014, at para. 1649.
B-10
4. The defense’s only legal argument is a quotation taken out of context from the Appeals
Chamber’s recent decision in the Vasiljević case. 47 The discussion in Vasiljević, however, was not
concerned with systematically defining aiding and abetting liability. Vasiljević merely mentioned
aiding and abetting in the context of defining a different basis for criminal liability, namely joint
criminal enterprise. To better define the latter, the Vasiljević decision drew comparisons to the
former. Nothing in that discussion suggests that specific direction is a stand-alone element of
aiding and abetting. Indeed, the defense implicitly concedes that no prior decision of this Tribunal
has ever denied aiding and abetting liability merely because the assistance was not specifically
directed at the crime.
[Precedent: REVERSE]
3. It is irrelevant that none of [Vuković's /Horvat's] acts were specifically directed toward the
commission of the war crimes by the [VRS / HVO]. Specific direction is not an element of aiding
and abetting. The text of the Statute is clear on this point. Rewriting the Statute is not proper for
this Tribunal, nor would it be wise from a policy perspective. The rewriting urged by the defense
would make it more difficult to convict those who knowingly facilitate the most grievous crimes,
contrary to the Statute’s stated objectives. The prosecution urges the Appeals Chamber to
reconsider certain passages of its recent Bešić decision 48 that conflict with the Statute’s text and
purpose.
4. Article 7(1) of the Statute clearly extends criminal responsibility to any support knowingly given
to an eligible crime. The text does not distinguish support that is either “specifically directed” or
not, direct or indirect, etc. It simply reads:
“A person who planned, instigated, ordered, committed or otherwise aided and abetted in the
planning, preparation or execution of a crime referred to in articles 2 to 5 of the present Statute,
shall be individually responsible for the crime.” (emphasis added)
5. The broad sweep of Article 7(1) is intentional. As the subsequent two paragraphs (Article 7(2)
and 7(3)) demonstrate, the Statute makes a point of persecuting not only the foot soldiers but the
masterminds who enable their horrible crimes. The [VRS / HVO] could not have committed its
crimes without [Vuković's /Horvat's] support. [Vuković /Horvat] knew this and yet went ahead.
Justice demands that he be convicted, and so does the Statute.
47
Prosecutor v. Vasiljević, Case No. IT-98-32-A, Judgement 23 January 2014.
48
Prosecutor v. Bešić, Case No. IT-03-56-A, Judgement 23 January 2014.
B-11
7. The defense submits that under the approach favoured by the Trial Chamber, any provision of
weapons would result in individual criminal responsibility of the provider, approaching a form of
strict liability. However, this fear is unfounded. In order for aiding and abetting liability to arise, a
number of additional elements need to be present: one or more crimes must have been actually
perpetrated; the weapons provided must have substantially contributed to the perpetration; and
the weapons-provider must have been aware of their likely use. The fact that they could have
theoretically been used in lawful activities would not be decisive in this assessment.
8. The defense’s only legal argument is Bešić. It is true that that decision did acquit a defendant
of aiding and abetting liability on the grounds that specific direction was lacking. The prosecution
recalls, however, that the clear text of the Statute must always prevail over language in prior
decisions of the Tribunal. Moreover, Bešić is far from the only decision that has addressed the
issue of specific direction, and the long discussion in Bešić itself makes clear, many other decisions
of the Appeals Chamber reached opposite conclusions. Finally, Bešić concerned different facts.
Unlike the accused in the present case, the accused In Bešić was not the highest-ranking officer
and thus did not have final authority to decide the assistance rendered to the other army.
III. Conclusion
[VUKOVIĆ ONLY: [8./9.]] For too long, the defendant has been able to walk free. He has publicly
mocked this tribunal and repeatedly inflamed lingering tensions with inflammatory public
statements showing absolutely no regrets about the horrors of the war in general, and the war
crimes he supported in particular. 49 The Appeals Chamber should make a strong statement that
generals in the headquarters can be as guilty as, or more guilty than, the soldiers on the ground
when heinous crimes are committed, and that this Tribunal will prosecute both.]
[8, 9, or 10]. For the reasons stated above, the Prosecution respectfully requests that the Appeals
Chamber: (i) hold that the actus reus of aiding and abetting in international law does not require
specific direction; (ii) and uphold the Trial Chamber’s judgment.
49
The prosecution recalls, for example, the defendant’s opening statement before the Trial Chamber, where he
stated, among other things, that his “only regret about the war is that too few of Serbia's enemies died." Defence
Opening Statement, 22 February 2010, Trial Hearing Transcript 9904.
B-12
YOUR JUDGMENT
YOUR REASONS
Please provide your reasons for this judgment in the form below 52: 53 [text entry box]
50
[US merely read: Guilty]
51
[US merely read: Not guilty]
52
[Brazil, France, Germany addition: or on a piece of paper marked with your random participation ID (shown at the
bottom of this screen)]
53
[Argentina, USA addition: Alternatively use the provided sheet of paper. Please mark your sheet with the
anonymous ID XYZ so we can match the sheet and your judgment later]
B-13
1. What proportion of your colleagues do you think decided the case as you did?
2. [FOR THOSE WHO ACQUITTED: Assume you were outvoted and the appeals chamber upheld
the conviction.] What sentence would you find appropriate?
a. For comparison, Charles Taylor, the former president of Liberia, was sentenced to
50 year in prison by an international tribunal for aiding and abetting widespread
brutality in Sierra Leone that included murder, rape, the use of child soldiers, the
mutilation of thousands of civilians and the mining of diamonds to pay for guns and
ammunition. On the other hand, Razim Delic, the chief of staff of the Army of Bosnia
and Herzegovina during the war, was sentenced to only three years by the ICTY for
his failure to prevent members of his army from committing crimes against captured
civilians and enemy combatants (murder, rape, torture).
b. To avoid misunderstandings, please write the time units (years or months) explicitly.
For example, if you thought that [10/25/40] years were an appropriate penalty, you
should write either “[10/25/40] years” or “[120/300/480] months.” [numbers of
years and corresponding months to be randomized]
3. Did you have any previous knowledge of international criminal law? Y/N 54
4. Did you recognize any of the names or places in the case? Y/N
5. [Country specific questions on background] 55
6. What is your age group? Under 30 / 30-40 / 40-50 / over 50 56
7. [US only] What is your gender? Female Male
54
[India: How well did you know international criminal law before this exercise? Very well / Well / Not well / Not too
well]
55
[Argentina: Are you a judge or prosecutor? Judge/prosecutor; [if judge] What level of court are you currently
serving on? Trial/Appeal; [if judge] What subject matter court are you currently serving on?
Civil/Commercial/Criminal/Other]
[Brazil: Are you a federal or state judge? Federal/state/not a judge; Are you now, or have you ever been, responsible
for criminal matters as a judge? Y/N/NA; Have you ever been a prosecutor? Y/N]
[China: What level of court are you currently serving on? Local/Intermediate/Provincial/Supreme; What division of
the court are you currently serving on? Civil/Criminal/Administrative]
[France: What is your current position? Judge/Prosecutor/Other; [For those who selected judge:] Are you sitting on
an appeals court? Y/N, Are you responsible for civil or criminal matters, or both? Civil(including
commercial)/Criminal/both, Have you ever been a prosecutor? Y/N; [For those who selected prosecutor or other:]
Have you ever been a judge? Y/N, [if yes] As a judge, did you ever sit on an appeals court?, As a judge, were you
responsible for civil or criminal matters, or both? Civil(including commercial)/Criminal/both]
[Germany: Are you a judge or a prosecutor? Judge/Prosecutor; [For those who selected: judge:] What court are you
currently serving on? Amtsgericht/Landgericht/Oberlandesgericht/Verwaltungsgericht/Finanzgericht/
Arbeitsgericht/Sozialgericht; [For those who selected Amtsgericht, Landgericht, or Oberlandesgericht:] Are your
responsible for civil or criminal matters, or both? Civil(including commercial)/Criminal/both]
[India: What type of court are you currently serving on? High Court/District Court/Civil Court]
[US: Were you ever a prosecutor? Y/N; Were you ever a criminal defense attorney? Y/N]
56
[Argentina: [not asked]] [US: What is your age? Under 45, 45-55, 55-65, 65-75, over 75]
B-14
Decision: Affirmed?
Judgment Reasons
Argentina 57
A s 1 Subject to the temporal restrictions on the time necessary to properly study the present case, I argue that the decision of the court of first 0 0 0
instance must be affirmed given that the arguments brought forth by the Defense are not sufficient to demonstrate the
unreasonableness or arbitrariness of this ruling under the applicable law. Concretely, if the discussion turns on the requirement of
knowledge and specific direction on the part of Horvat that his conduct contributed to the commission of illicit acts by the material
perpetrator of the crimes committed against the Bosnian Population, I consider that it is enough for the accused to suspect that his
activities could have collaborated directly or indirectly in the perpetration of crimes and for him to have paid no regard to any outcomes,
for assigning to him concrete responsibility as an accomplice of the conduct attributed by the prosecutor. The position in which the
accused found himself can be likened to one of a guarantor, and therefore his role in administrating sources of risks, far from releasing
him of any responsibility, demands greater efforts at the time of making decisions. Without this implying an objective responsibility
typical of civil law, but rather a subjective element necessary to maintain that he knew but did not care if his conduct would contribute to
the commission of illicit acts as the ones accredited. Therefore, there was the sufficient knowledge required to consider him an
accomplice without having to require knowledge of specific illicit situations created by the material perpetrators of these events.
57
Translated by one of the authors (IR).
C-1
C-2
C-3
C-4
C-5
C-6
C-7
58
Translated by Marcelo Moreno Bonassa with input from Etiene Coelho Martins.
C-8
C-10
C-11
C-12
C-13
59
Translated by Linda Hao (Langscale Translation), reviewed by one of the authors (JZL).
C-17
C-18
C-20
C-21
60
Translated by Damien Charlotin with edits by Mareike Grundmann and one of the authors (HS).
C-22
C-25
The court also notes that a causal link is unnecessary as between Ante Horvat’s course of action as an accomplice by way of aiding and
abetting and the perpetration of crimes, nor that his acts constituted a necessary condition to the perpetration of crimes, nor that they
have be a condition sine qua non.
Whereas to summarize, the court concludes beyond a reasonable doubt that, whether considered individually or in their totality, the
logistical support and the manpower supplied by Ante Horvat significantly contributed to the crimes committed by the HVO at Mostar
and at Ahmići as alleged in the indictment.
Whereas consequently the court affirms the judgment of the tribunal of first instance as to both the guilty verdict and the sentence
imposed.
r s 1 Ante HORVAT claims that the actus reus of the offence of aiding and abetting requires that specific direction be established to qualify as 0 0 0
a war crime, which was not the case for him; however the actus reus may involve different forms of intervention which do not require
neither the physical presence of the accused, as occurred in the case at bar, nor the commission of any physical acts ; in other words, the
fact of otherwise making logistical means available, of permitting members of the UVO army to spend their salaries in order to undertake
their criminal actions, characterizes the actus reus of aiding and abetting, without needing to determine whether the accused acted with
the specific objective of exterminating the Muslim members of local communities. The proof of the existence of this actus reus is
established in the case at bar, the accused having himself admitted that, as Chief of the Defence Staff of the UV, he had access to high
quality intelligence in relation to the means available to the government in addition to the power to make decisions, both of which
allowed him to aid and abet the UVO.
Consequently, the first instance judgement shall be confirmed concerning the guilt and the quantum of the sentence.
r s 1 It has been proven that the accused provided very significant logistical support, over a significant period of time, while being full aware of 4 3 4
the crimes committed by the HVO. This logistical assistance played an essential role in the perpetration of these crimes (weapons,
ammunition, manpower).
On appeal, the accused contends that these elements are insufficient to constitute the offence of aiding and abetting on the grounds
C-32
C-34
C-39
61
Translated by Andrew Hammel.
C-40
C-41
C-44
C-45
C-47
C-48
C-53
C-54
C-60
C-65
C-66
C-67
C-68
C-70
C-71
C-73
C-74
C-75
C-76
C-77
C-78
1. This includes any judge who explicitly notes that they did not want to give judgment but clicked
through by mistake, to express their disapproval, etc.
2. If not: did the participant express explicit reservations about submitting a judgment (in
particular because of the short time available)?
ARE THERE CLEAR INDICATIONS THAT THE PARTICIPANT MISUNDERSTOOD THE TASK (OF JUDGING THE CASE ON APPEAL)?
We are NOT judging the quality of their written work product but merely looking for indicators that they
did not do what they were supposed to be doing in this study. In particular, an answer that is merely
cryptic, inconclusive, or even inconsistent is not evidence of a misunderstanding. Bear in mind that the
judges were working under time pressure, and time may have run out before they finished their
sentence or before they could delete obsolete words they had written earlier. Also, judges may
purposefully use conclusory language, distort the facts, overstep their role, etc. -- happens in real life all
the time.
Similarly, participants need not couch their reasons explicitly in the language of reviewing the lower
court’s findings. In other words, it is not evidence of a misunderstanding that a participant did not
explicitly frame their argument as a review of the lower court’s findings. First, most appeals courts
around the world review questions of law de novo, so discussing the legal evaluation of facts established
by the trial court is consistent with the appeal’s judge role whether or not it is framed as a “review” of
the trial court. Second, some appeals courts in the world are allowed to review the entire case de novo,
see next note. Third, we must again make allowance for the time pressure and the rhetorical short cuts
that it may have inspired.
1. NB: Making factual findings is not per se evidence of a misunderstanding. First, the ICTY Appeals
Chamber is technically allowed to review “error of fact” under Art. 25(1)(b) of the ICTY statute
(provided the factual error “has occasioned a miscarriage of justice”). Second, many appeals
courts in the (civil law) world are allowed to review facts de novo, and we purposefully did not
specify a standard of review. Also see the next question.
2. If you answer yes to, or had doubts about, this question 2., please include a note explaining why
Did the judge make factual findings rather than accept the conclusions of the trial court and the
statement of facts, in particular with respect to whether the accused provided “substantial help” to the
armed group, and whether this help was “specifically directed” at the crimes?
D-1
We count the judge as making factual findings only if the judge states facts that go beyond or against
what is in the statement of facts and/or the trial court judgment. In particular, we accept that the
participant in the experiment may simply be referring to factual findings of the trial court and/or the
statement of facts even if the participant simply states “X” rather than writing "the trial court found that
X" or "the parties agreed that X.”
An example of an actual factual finding would be for the judge to say "We can infer from the fact that
the defendant knew about the crimes and did not disapprove that the defendant intended the war
crimes to be committed" -- because nowhere in the statement of facts or the trial court's factual
findings does it say that the defendant had such intent, so here a factual inference is being drawn that
goes beyond the SoF and the TC judgment.
If yes: do those factual findings disturb those of the trial court, i.e., are they in conflict with the trial
court’s findings?
Findings regarding specific direction and substantial help have a factual basis, but they are primarily legal
findings rather than factual ones. When a reason makes a contrary finding regarding one of these issues,
it is probably reinterpreting the same set of underlying facts / applying a different legal label to them,
rather than finding new facts that change the legal conclusion.
Example:
“In my view, the Appellant knew that the weapons etc. would not be used only for permissible
military action because he continued to advocate for and take measures to ensure support, and
thus a specific direction is actually present in this case (meaning that the legal question need not
be addressed at all).”
Here the judge makes no factual finding. The judge merely states that specific direction is
present because of the fact that the appellant knew how the weapons would be used, and this
fact was already present in the statement of facts. In other words, the judge does not add any
new facts, but instead reinterprets an old fact to mean specific direction. (The trial court had
considered this same fact but determined that, legally, it did not indicate specific direction.)
Findings regarding specific direction and substantial help that are contrary to those of the trial court
should be coded in a column separate from the columns for factual findings and contrary factual
findings. There is no relationship of logical necessity between this legal findings column and the columns
about factual findings. (E.g, the judge can contradict the trial court’s finding regarding specific direction
without making any contrary factual findings, and vice versa.) Record a contrary factual finding alongside
D-2
Intent
Intent is similarly both a legal and a factual issue, but the legal side or the factual side can predominate
depending on how the judge uses the word. If the judge appears to refer to the defendant’s will or
goals, then he may be making a factual finding about the defendant’s mental state. If instead the judge
appears to refer to intent as a legal construct (similar to specific direction, above), then he is most likely
making a legal finding.
Examples:
(1)
“In the final analysis, the question whether specific direction is required can remain open, since
the Defendant’s acts in support indeed show a “specific direction” toward the commission of
the underlying offenses. (whereby it is assumed that the Defendant provided the assistance
with knowledge of the consequences and the will to achieve them)”
(2)
“His logistical support therefore assisted the perpetration of that crime and any support given
after he had knowledge of the sniping can be deemed to have been done with the specific
intent to commit the crime.”
In the examples (1) and (2) the judge makes factual findings about the defendant’s mental state.
In the phrases shown in bold the judge assumes a will to achieve the consequences or an intent.
However, the trial court only found that the defendant knew about the crimes going to be
committed but the trial court did not make a finding regarding the will of the defendant.
If the judge states that the defendant knew about the crimes and, therefore, must have accepted or
approved the crimes, then no factual finding is made. You can tell that no factual finding is made in such
a case because this conclusion is based upon the facts already given in the statement of facts and the
trial court judgment; no new fact is added in reaching this conclusion. As mentioned above, the situation
is different if the judge speaks of an intent or the will to achieve the crimes.
1. Precedent:
a. Was there any mention of precedent, even in generic form (“courts have generally …”,
“it is well established in the practice of the courts that …”)?
b. If yes: can the mention be interpreted as a reference to a specific precedent available in
the case?
c. If yes: Was the name(s) of the precedent cited?
d. Was the precedent distinguished, i.e., differences in facts pointed out?
D-3
Record an irrelevant fact when the judge mentions a fact that should not have influenced the
legal outcome, yet appears to have done so. Most irrelevant facts are things that make the
defendant appear sympathetic or unsympathetic. However, this “sympathy rule” should be
interpreted very broadly, and the following are always irrelevant facts: remorse or
reconciliation, being nationalist party leader, insulting the court, and having any contacts with
NATO.
D-4