Louise Amoore - Cloud Ethics - Algorithms and The Attributes of Ourselves and Others-Duke University Press Books (2020)
Louise Amoore - Cloud Ethics - Algorithms and The Attributes of Ourselves and Others-Duke University Press Books (2020)
Ethics
Cloud
Louise Amoore
Ethics
Algorithms
and the Attributes
of Ourselves
and Others
Cover art: James Turrell, Skyspace Seldom Seen, 2004. Houghton Hall,
King’s Lynn, Norfolk, England. Courtesy of Houghton Hall Archives.
Photograph by Louise Amoore.
For my family, with love
&
For my students, with thanks for their curiosity
This page intentionally left blank
Any true scientific knowing is always, like feeling, only partial.
— John Fowles
xi
Acknowledgments
1 Introduction
Politics and Ethics in the Age of Algorithms
Part 1. Condensation
Part 2. Attribution
Part 3. Ethics
173
Notes
197
Bibliography
212
Index
This page intentionally left blank
ACK NOW LED GM EN T S
The writing of this book coincided with what has felt like the most difficult
and dangerous of political times I have experienced. The UK stands on the
brink of leaving the community of European peoples that it joined in my first
year of life, and far-right political forces are gathering at every turn. So many
of the things in the world that I hold dear seem to be under threat: the right
of all young people to have an education funded by those who have already
benefited in the same way; the obligation on all people and governments to
welcome and support those who flee from violence and hardship; the capacity
of a democratic society to challenge hate and prejudice; the rights of women,
black and ethnic minorities, and lgbtq+ people to make political claims in
the world without risk to their lives and well-being. Following the rising pow-
ers of algorithms, as they have begun to dominate economic and political life,
my fears for the future are amplified. Let me be clear, machine learning algo-
rithms that anticipate our future propensities are seriously threatening the
chances that we have to make possible alternative political futures. In this con-
text, the writing of this book has been challenging for its constant reminder
of the limiting of potential political futures. But it has also been intensely up-
lifting for the incredible generosity of others in encouraging my work. One of
the overriding feelings I have as I complete this book is that young people have
the full measure of what is at stake ethicopolitically in an age of algorithms.
After every lecture I have given, over every discussion in corridors and cafes,
I have been struck by the depth and force of the understanding and innova-
tive thought of the next generation of young scholars and students. There is
much to be hopeful for, and I dedicate this book to this sense of hopefulness
and curiosity.
I have worked in the Department of Geography at Durham University for
fourteen years. It is not easy to describe how a group of people comes to matter
so greatly to whether it will be possible to conduct the work one most wishes
to. I can only say that the mood of experimentation, the quiet support of cu-
rious juxtapositions of theory and method, the standing together on a snowy
union picket line, and the quite breathtaking generosity in reading work has
been vital to me. Thank you all — you are pretty incredible: Ben Anderson, An-
drew Baldwin, Oliver Belcher, Mike Bentley, Harriet Bulkeley, Sydney Calkin,
Kate Coddington, Rachel Colls, Mike Crang, Niall Cunningham, Jonny Dar-
ling, Alex Densmore, Nicky Gregson, Paul Harrison, Sarah Hughes, Elizabeth
Johnson, Sarah Knuth, Paul Langley, Lauren Martin, Cheryl McEwan, Colin
McFarlane, Joe Painter, and Jeremy Schmidt. The past few years have also been
extraordinary for the PhD students I have been so lucky to supervise: Alice
Cree, Arely Cruz Santiago, Andrei Dinca, Pete Forman, Sarah Hughes, Olivia
Mason, Anna Okada, and Vanessa Schofield. I have never known any period
so exciting for the discussions you have provoked and the experimental meth-
ods you are pioneering. You all have so much courage, and you make me feel
xii more courageous.
When I reflect on the many forks in the path of writing this book, I can
Acknowledgments
Acknowledgments
the Vasari Institute, for conversations on cloud ethics; Susan Schuppli, Luci-
ana Parisi, Jussi Parrika, and Eyal Weizman, for memorable discussions of the
architecture of algorithms; N. Katherine Hayles and all the participants in
the Thinking with Algorithms conference in Durham; Jamie Woodward, Maja
Zefuss, Noel Castree, Martin Coward, Veronique Pin-Fat, and those in the au-
dience at the Manchester University 125 Years Anniversary lecture. You have
no idea how much your generosity has supported my work.
I am grateful for the patience and time afforded to me by many people whose
daily working lives bring them close to the lifeworlds of algorithms. Without
the insights of computer scientists, doctors, software developers, lawyers —
and many others who were willing to share their accounts — it would not have
been possible to pursue my questions. It can be difficult to engage with new
vocabularies and practices, but this kind of difficulty has been so productive
in following threads in the dark and finding insight in the multiple meanings
of bias, attribute, decision, weight, and judgment. It seems to me that the basis
for ethics in an age of algorithms is not strictly a shared language and common
understanding but precisely the persistence of difference and the inexplicable.
The writing of this book was funded by the award of a Leverhulme Major
Research Fellowship, “Ethics of Algorithms” (2016 – 2018). I have been extraor-
dinarily fortunate to be afforded the time and resources to do the work that
I so wanted to be able to do. It has become much more difficult in the UK to
fund the aspects of scholarship that involve reading, thinking, presenting and
discussing work, and writing. Thank you, Leverhulme Trust, and may you
long continue to act against the grain of prevailing logics of research funds
in our times. During my fellowship, two extraordinary women have under-
taken my teaching duties: Sarah Hughes and Sydney Calkin. They are a tour
de force, and I thank them for everything they have done. Much of the field-
work research for the book was funded by the Economic and Social Research
Council (esrc) for research on “Securing against Future Events” (2012 – 16).
Throughout the project I worked in collaboration with Volha Piotukh — a post-
doctoral researcher of simply astounding talent. Volha, my thanks will always
fall short. You simply never gave up — you got us into places for interviews and
observations that I would have said were impossible. You asked questions that
challenged, exposed, and opened new pathways. You are the human embodi-
ment of a whole new methodology for the social sciences and humanities to
engage science. Hanging out with you (long-haul flights, broken trains) has
been inspirational.
xiv I owe an immeasurable debt to Duke University Press editor Courtney
Berger, who has once more offered encouragement and thoughtful critique in
Acknowledgments
equal measure. Thank you to Jenny Tan and Sandra Korn for guiding me pa-
tiently through the editorial and production process, and Lalitree Darnielle
for working on the art and illustrations with such care. I am especially grate-
ful to the three anonymous reviewers for the incredible investment of their
time and insight in making this a better book. A version of chapter 1 appeared
previously as “Cloud Geographies: Computing, Data, Sovereignty,” Progress in
Human Geography 42, no. 1 (2018): 4 – 2 4.
Finally, friends and family. You know who you are, and you know how you
tolerated the tunnel vision that can be necessary to see something through.
Marieke de Goede, we have walked and talked in countless places from Turin
to San Diego over twenty-two years, and every conversation has left its mark.
My gratitude to you is distilled to a dinner in Boston in April 2016 when you
put the whole show back on the road — thank you. Jonathan, you listened pa-
tiently to all the nonsense and supplied a different kind of distillation, craft
gin — a finer brother there cannot be. Grace and Tom — you are older now, and
though the words “mum is finishing a book” bring necessary shrugs of the
shoulders (and some giggling), you are so very kind about it. Paul, I only real-
ized in these past few weeks that we never work on completing manuscripts at
the same time. Like everything in our lives, they are shared across time, with
love and support. Thank you for making it possible, and it is your turn again
now.
IN T RODUC T ION
It is March 2016, and I am seated in a London auditorium, the gray curve of the
river Thames visible from the windows. A tech start-up business, specializing
in developing machine learning algorithms for anomaly detection, is present-
ing its latest algorithmic innovations to the assembled government and corpo-
rate clients. The projection screen displays a “protest monitoring dashboard”
as it outputs risk scores of “upcoming threats of civil unrest” in cities around
the globe, their names scrolling: Chicago, London, Paris, Cairo, Lahore, Is-
lamabad, Karachi. The score that the analyst reads from the dashboard is the
singular output of deep neural network algorithms that have been trained to
recognize the attributes of urban public life, the norms and anomalies latent
in the data streams extracted from multiple sources, from Twitter and Face-
book to government databases. As the presenter explains to the audience of
national security, policing, and border officials, “We train our algorithm to
understand what a protest is and is not,” and “it gets better,” “adapting day by
day,” as it iteratively learns with humans and other algorithms.1 The process of
learning “what a protest is” from the clustered attributes in data and modify-
ing the model continues when the algorithm is later deployed in the city or at
the border: “We give you the code,” he pledges, “so that you can edit it.” How
does an algorithm learn to recognize what a protest is? What does it mean to
cluster data according to the attributes and propensities of humans to gather
in protest or in solidarity? At the London event, as the presenter displays a still
2
Introduction
from Minority Report (figure I.1), my thoughts turn to the protests that took
place one year earlier, in the US city of Baltimore.
On April 12, 2015, Freddie Gray, a twenty-five-year-old African American
man, sustained fatal injuries in the custody of the Baltimore Police Depart-
ment. The profound violence of Gray’s murder is an all-too-familiar event in
the racialized architectures of our contemporary cities. During the days that
followed his death, however, as people gathered on Baltimore’s streets to pro-
test the killing, the violence of the act extended into the plural actions of a set
of machine learning algorithms that had been supplied to the Baltimore Po-
lice Department and the US Department of Homeland Security by the tech
company Geofeedia. With the use of techniques similar to those described in
the London protest-monitoring software, the Geofeedia algorithms had been
trained on social media data, analyzing the inputs of Twitter, Facebook, You-
Tube, Flickr, and Instagram and producing scored output of the incipient pro-
pensities of the assembled people protesting Gray’s murder. “Several known
sovereign citizens have begun to post social media attempting to rally per-
sons to demonstrate,” recorded the Baltimore Police Department in a memo
that promised to “continue to evaluate threat streams and follow all actionable
leads.”2 Indeed, Geofeedia went on to market its algorithms to other states on
the basis of a Baltimore “case study” (figure I.2) in which Freddie Gray is said
to have “passed away,” the city to have “braced itself for imminent protests,”
and the police to have seized “opportunities” to analyze “increased chatter
from high school kids who planned to walk out of class.”3
During those days in April, terabytes of images, video, audio, text, and bio- 3
metric and geospatial data from the protests of the people of Baltimore were
At first sight, the potential for violent harm precipitated by algorithms that
learn to recognize human propensities appears to be a self-evident matter for
critique. Surely, one could say, the ethical terrain of the algorithm resides in
the broader political landscape of rights and wrongs, good and evil. After all,
one could readily identify a set of rights, already apparently registered as be-
longing to rights-bearing subjects, that has been contravened by algorithms 5
that generate targets, adjudicating which people may peaceably assemble, or
trovertible axis, exempted from doubt, on which all social, political, and eco-
nomic life turns. “The hinge point,” as Foucault differently identifies, can also
be the point of “ethical concerns and political struggle,” as well as the point of
“critical thought against abusive techniques of government.”28
So, a principal ethicopolitical problem lies in the algorithm’s promise to
render all agonistic political difficulty as tractable and resolvable. Where poli-
tics expresses the fallibility of the world and the irresolvability of all claims,
the algorithm expresses optimized outcomes and the resolvability of the claim
in the reduction to a single output. In the following pages, I specify how this
book approaches what an algorithm is and what it does. Among the many
problems of studying algorithms is the matter of specifying which type of al-
gorithm one is addressing. Though my primary focus in this book is on ma-
chine learning algorithms, and predominantly deep neural networks, in most
of the instances I discuss, these algorithms are used in conjunction with some
much older and less fashionable rules-based and decision tree algorithms. The
form of the algorithm is not delimited by its name but by its coming into be-
ing, its use in the wild. As Nick Seaver has argued, “Rather than offering a
‘correct’ definition,” a critical study of algorithms could begin from “their em-
pirical profusion and practical existence in the wild.”29 For example, an ad-
vanced deep neural network for object recognition is intimately connected to
some much older classifiers that are used in the preparation of the data, and
it meaningfully comes into being as it is modified through its deployment in
the world. It is not possible to identify a secure boundary around one specific
named algorithm because, as a calculative device, it is a composite creature.
I propose three routes into understanding what algorithms are in the world,
each with its own distinctive implications for what is at stake ethicopolitically:
algorithms as arrangements of propositions; algorithms as aperture instruments; al-
gorithms as giving accounts of themselves.
“Generate All the Rules and Debate Which Is Interesting”:
Algorithms as Arrangements of Propositions
Critical accounts of the rise of algorithms have placed great emphasis on the
power of algorithms to visualize, to reprogram vision, or indeed even to “see”
that which is not otherwise available to human regimes of visuality. Similar to
the spatial arrangement, this primacy of the visual register has also annexed
what could count as the ethics and politics of algorithms. There are two cu-
riously twinned accounts of contemporary algorithms in relation to regimes 15
of sight and vision. The first is that algorithms operate on a plane in excess of
Perhaps the most widespread concern in public and scholarly discourse on the
operation of algorithms in society is that they are unaccountable or that they
18 cannot meaningfully be held to account for harmful actions. Indeed, a kind of
proxy form of accountability is emerging, in which the designers of algorithms
are made the locus of responsibility for the onward life of their algorithms.
Introduction
Virginia Eubanks proposes a “Hippocratic oath for data science,” in which the
designers of algorithms would be accountable to human beings and not “data
points, probabilities, or patterns.”58 Similarly, in a 2016 Nature editorial call-
ing for “more accountability for big data algorithms,” the editors propose that
“greater transparency” could be achieved if the designers of algorithms “made
public the source of the data sets they use to train and feed them.”59 They urge
greater disclosure of the design of algorithms and an “opening up to scrutiny”
of their workings. Indeed, so widespread is this notion that the accountability
of algorithms can be grounded in their design or source code that technical so-
lutions for “explainability” are being developed to trace an apparent bias back
to a design problem. For example, automated systems for the assessment of
creditworthiness are thought to be rendered transparent by a tool that traces
the specific credit score output back to a data element, such as an unpaid bill.60
Such techniques are thought to anchor the accountability of the algorithm
precisely in an intelligible knowledge of its workings.
Within these demands for algorithmic accountability lies a specific form of
giving an account. The locus of a truthful account is in the apparent “source”
of the algorithm, in its origins, whether in the source code or in the algorithm
designer as an author. This locus of original account, I suggest, is profoundly
limiting the capacity to demand that algorithms give accounts of themselves.
It imagines a secret source or origin to which all potential future harms could
be traced. As Foucault writes on the disappearance of the author, “the task
of criticism is not to bring out the work’s relationship with the author” but
rather to “analyse the work through its structure, its architecture, its intrinsic
form, and the play of its internal relationships.”61 For our contemporary times,
the call for accountability of algorithms has precisely targeted the work’s re-
lationship with the author, seeking to render transparent the intent and the
workings of the design. At one level there are clear limits to identifying the
source of the algorithm, not least that each apparent “one” contains multiple
elements from multiple sources, with much of this aggregation concealed even
from the designer. My point, however, is that the problem of anchoring ac-
countability in a source or origin is not confined to algorithms but is a persis-
tent and irresolvable ethicopolitical problem.
The problem of the unidentifiable origin of the algorithm extends to all no-
tions of an authoring subject I who can give a clear-sighted account of herself.
If one’s account of oneself can never be fully secured, then the full disclosure
of one’s grounds for action is impossible. To give an account is always to give an 19
uncertain narrative that risks falling short or failing to be recognized. Let us
Chapter Outlines
In sum, my cloud ethics has a principal concern with the algorithm’s double
political foreclosure: the condensing of multiple potentials to a single output
that appears as a resolution of political duress; and the actual preemptive clo-
sure of political claims based on data attributes that seek recognizability in
advance. Confronted with this double foreclosure, each chapter of this book
elaborates a speculative strategy for reinstating the partial, contingent, and in-
complete character of all algorithmic forms of calculation. In their partial and
incomplete way of generating worlds, we can locate their ethicopolitics. Part 1,
“Condensation,” comprises two chapters that detail how algorithms condense
and reduce the teeming multiplicity of the world to a precise output. Chapter 1,
“The Cloud Chambers,” examines how algorithms are acting through cloud
data architectures to produce a new paradigmatic alliance between sovereign 21
authority and scientific knowledge. Developing an analogy with the scientific
which algorithms write themselves into being in the world. Taking seriously
the attribute as it is understood in computer science, the chapters connect
this attribution back to genealogies of writing and ethics. In chapter 3, “The
Uncertain Author,” I am concerned with how the search for ethical codes to
govern algorithms has located an author function in the “source code” of al-
gorithms. When an algorithm appears to have precipitated a crisis or to have
caused a harm, often the reflex response is to seek out its origin: Who designed
the model? Who wrote the code? Who or what labeled the training data? Was
it biased? Who is the author of the algorithm? In this chapter I explain the lim-
its of locating an ethicopolitical response in the authorship of source code. Fo-
cusing on the algorithmic techniques for natural language processing — where
a corpus of literary texts are used to train an algorithm to recognize style and
sensibility — I suggest that the algorithm’s ways of being in the world are not
all present in the source code and, indeed, substantially exceed the design of
an authoring subject. The authorship of the algorithm is multiple, continually
edited, modified, and rewritten through the algorithm’s engagement with the
world. Juxtaposing novelists’ accounts of their own uncertain authorship with
computer scientists’ reflections on how the fragments of an algorithm come
together in acts of writing, I suggest the impossibility of identifying a defini-
tive author.
To invoke the call for attributing authorship is not only insufficient as cri-
tique, but it also risks amplifying the ability of the algorithm to bind together
a unity of incompatible, fraught elements as though the difficulties and dif-
ferences could be resolved. In place of the call for securing responsibility via
authorship, I propose that there is a possibility of ethicopolitics in the act of
writing, in the bringing together of scattered elements, and in the opening of
a space of uncertainty. A cloud ethics must begin to reopen the act of writ-
ing as a site of ethical significance. The profound uncertainty that is brought
to the writing of an algorithm — “I do not know how adjusting this weight is
changing the output”; “I cannot be sure how my training data has produced
these clusters”; “I am curious whether shifting this threshold will reduce the
false positives” — invites an iterative process of writing that is never completed.
Here is an ethicopolitical tension that is worth holding on to — the algorithm
promises to complete everything, to condense to a single optimized output
and action, and yet it enacts a process of writing that opens on to an indeter-
minate future.
Chapter 4, “The Madness of Algorithms,” addresses the moments when it
appears that an algorithm has acted in a state of frenzy or has departed from 23
its otherwise rational logic. From the lethal accidents of autonomous vehi-
Condensation
This page intentionally left blank
1
The Cloud Chambers
Condensed Data and Correlative Reason
What I wouldn’t give now for a map of the ever constant ineffable?
To possess, as it were, an atlas of clouds.
— David Mitchell, Cloud Atlas
A Beautiful Sight
were in some exceptional condition.” Though Wilson had set out to reproduce
mimetically the formation of clouds in nature, in fact his experiments gener-
ated traces of the electrically charged atoms and ions previously unavailable
to the sciences.3
What Wilson’s cloud chamber ultimately made possible for the twentieth
century’s study of particle physics was the ability to photograph and to per-
ceive the movement of particles in an exceptional state (figures 1.1 and 1.2). As
historian of science Peter Galison writes in his account of Wilson’s work, “Af-
ter the cloud chamber the subatomic world was suddenly rendered visualiz-
able.”4 Charged or ionized particles could not be observed directly with optic
devices, as with the instruments of microscopy or telescopy, but their traces
and trajectories of motion appeared indirectly via the cloud tracks condens-
ing on the nuclei. As Wilson reflects on his 1911 experiments, “I was delighted
to see the cloud chamber filled with little wisps and threads of clouds” so that
“the very beautiful sight of the clouds condensed along the tracks of the alpha
particles was seen for the first time.”5 The cloud chamber apparatus, conceived
for the human observation of processes of formation in nature, had become a
technique for rendering perceptible the movement of objects beyond the thresh-
olds of human observation.
Almost exactly one century on from the publication of Wilson’s first cloud
chamber images, the idea of the cloud is once more describing the advent of
processes at scales that appear to transcend the observational paradigm and
exceed our human capacities to see, to know, and to understand. Indeed, the
cloud in cloud computing is widely held to derive from the mapping of infra-
structures of computer networks, where the visualization of a figurative cloud
stands in for the vast complexity of the internet.6 In the twenty-first century,
cloud computing promises to have effects on the very episteme of our world,
analogous to the effects of the cloud chamber on what could be rendered
knowable in twentieth-century physics.
More precisely, the advent of cloud computing opens space for a renewed
Figure 1.1
Charles Thomson
Rees Wilson’s cloud
chamber expansion
apparatus (1912 – 13).
Science Museum
Group Collection,
Board of Trustees of
the Science Museum,
31
London.
Cloud I is concerned with the identification and spatial location of the data
centers where the cloud is thought to materialize. Indeed, as computer science
began to document the emergence of cloud computing, the idea of geography
came to have a specific meaning defined by where data and programs are spa-
tially located. In a 2008 Association of Computing Machines (acm) forum
devoted to the advent of cloud computing, a transformation is described “in
the geography of computation,” with “data and programs” being “liberated” as
they are “swept up from desktop pcs and corporate servers and installed in the
compute cloud.”12 Such accounts of the cloud appeal to a geography of “scal-
able” computation, which is thought to change radically with the rise of big
data and the concomitant need for flexible storage and computational power.13
There is need for some caution, however, in understanding the geography of
the cloud primarily in relation to the rise of twenty-first-century volumes of
digital data. Indeed, the emergence of cloud computing has important origins
in grid computing, distributed scientific data, and, perhaps most significantly,
the notion of computing as a public utility. “Computing may someday be or-
ganized as a public utility,” speculated computer scientist John McCarthy in
his mit lecture of 1961, so that “the computer utility could become a new and 35
important industry.”14
designs a portable data center inside a standard shipping container, the matter
of geography is thought to reside in the spatial location of data storage.19 Of
course, these materializations of the cloud have ethicopolitical significance,
so that when Google patented the floating data center (figure 1.4), the offshore
potentials of cloud techniques beyond jurisdiction were actualized in these
vast data ships.
Let us capture this geography of the spaces of cloud computing as Cloud
I. This imagination of the cloud as a dispersed yet spatially located array of
data centers is present in computer science, and that has extended into geo-
graphic and even political and geopolitical debate. So, for example, following
the Snowden disclosures of the extent of US authorities’ access to European
citizens’ data via US data centers, the EU sought initially to develop a “Euro-
pean cloud,” in which they might store safely European data under European
jurisdiction.20 Similarly, following the US subpoena and mining of European
financial transactions, the major Society for Worldwide Interbank Financial
Telecommunications moved its cloud-computing provision to an underground
data center in Switzerland, and the Canadian government has legislated for
what it calls “data sovereignty,” meaning domestic public data traffic must
not leave Canadian territory. Understood as a spatial and territorial question
of the jurisdiction of data technologies, then, the political response has been
to seek to render the cloud intelligible and governable.
The unintelligibility of the cloud as a form of governing or a locus of geo-
political power has profoundly political implications. Following the exposure
of the nsa’s prism program in 2013, for example, the UK Intelligence and Se-
curity Committee (isc) of Parliament — the sole body responsible for public
oversight of security and intelligence powers in the UK — called the then for-
eign secretary, Philip Hammond, to give evidence before the committee. At
the time of his evidence, Hammond was the final signatory on all warrants
authorizing the interception and analysis of external communications data, 37
conventionally understood as where one end of the communication is located
gue, the new predictive “approaches to policing and intelligence may be both
qualitatively and quantitatively different from surveillance approaches,” and
thus enable “discriminatory practices that circumvent current regulations.”22
Yet, Crawford and Schultz go on to suggest that an alternative space for demo-
cratic oversight might lie in what they call “a right to procedural data due pro-
cess,” where constraints and oversight mechanisms are placed on the algorith-
mic processes of data analysis.23 Even as the cloud overflows and exceeds the
categories and practices of bureaucracy and law, what has come to be at stake
ethicopolitically has become a struggle to wrest the cloud back into a form
over which one can have oversight, to expose its “bias” and demand neutrality,
to make it comprehensible and accountable in democratic fora, and to render
the cloud bureaucratically and juridically intelligible.
Among the critical geographic and artistic accounts of cloud computing,
the desire to wrest the cloud into an intelligible form similarly finds expression
in methods of visualization. The geographer and artist Trevor Paglen seeks to
“make the invisible visible,” reflecting that “the cloud is a metaphor that ob-
fuscates and obscures” the material geographies of the “surveillance state.”24
Paglen’s work is concerned with bringing the geopolitics of cloud computing
back into a human line of sight through visualization. His methods deploy op-
tical devices of many kinds to bring back into human vision that which would
otherwise exceed the limits of observation. His ghostly images of the nsa’s
data centers are photographs taken at night with a long-focus lens from a he-
licopter (figure 1.5); and his photographs of the secret installations of military
and drone bases in the Nevada desert are taken with adapted telescopic instru-
ments of astronomy.25
The optical instruments deployed by Paglen belong to a paradigm of ob-
servation in which, as Peter Galison describes, one is offered “a direct view”
of things otherwise “subvisible.”26 As Paglen accounts for his own work: “My
intention is to expand the visual vocabulary we use to see the US intelligence
community. Although the organizing logic of our nation’s surveillance appa-
39
ratus is invisibility and secrecy, its operations occupy the physical world. Digi-
tal surveillance programs require concrete data centers; intelligence agencies
are based in real buildings. . . . [I]f we look in the right places at the right times,
we can begin to glimpse the vast intelligence infrastructure.”27 So, for Paglen
the challenge is to “expand the visual vocabulary” to see more clearly the ac-
tions of algorithms, and to bring into vision the things that would otherwise
be obfuscated by the cloud. Similarly, for the artist James Bridle, “one way of
interrogating the cloud is to look where its shadow falls,” to “investigate the
sites of data centres” and to “see what they tell us about the real disposition
of power.”28
Yet, what are the “right places” and “right times” to look and to observe
an apparatus? How does one investigate something describable as the real dis-
position of power? Indeed, what ways of seeing would be appropriate to what
art historian Jonathan Crary calls a “relocation of vision” taking place with
computation, or the visual vocabulary appropriate to the digital mediation
of cultural objects and urban scenes identified by Gillian Rose and Shannon
Mattern?29 If the cloud is to be observed and revealed in the secret glimmer-
ing buildings of the nsa’s data centers in Paglen’s images, then could his “real
buildings” also be located in other places? Could they be observed, for exam-
ple, in the rented North London offices where a small team of physics gradu-
ates write algorithms for risk-based security?30 Must the material geography of
cloud computing be found in the buildings or territories where it is thought to
actualize? Could the “right place” to look also be in the spatial arrangements
of an experimental clustering algorithm used in anomaly detection, or in the
generative logics of emergent machine learning algorithms?31
My point is that the desire to “open the black box” of cloud computing
40 and to expand the visual vocabulary of the cloud, to envision the cloud and
its properties in geographic space, dwells within and alongside the paradigm
Chapter One
What I call Cloud II, or the cloud as analytical gathering of algorithms with
data, displaces the question, Where is the cloud?, replacing it with, How does
the cloud render the world perceptible and analyzable? In this way, Cloud II
42 witnesses the extension of algorithmic modes of reason that surface something
for perception and action amid an otherwise profoundly uncertain environ-
Chapter One
ment. As historian of science Paul Erickson and his colleagues have traced me-
ticulously in the historical emergence of algorithmic rationality, the profound
uncertainties of the Cold War nurtured a desire to extend mathematical logic
into the realm of decision. The decision procedures and cybernetic methods of
algorithm appeared to extend the faculties of human reason so that they “no
longer discriminated among humans, animals, and machines” in the capacity
to analyze, to decide, and to act.40 What we see here is the entwining of hu-
man and machine modes of reasoning such that what Henri Bergson calls the
“organs of perception” of the world are composite beings formed through the
relations among humans, algorithms, data, and other forms of life.41
Understood in terms of the intertwined faculties of human and machine,
the contemporary spaces of cloud computing exceed the territorial geogra-
phies of the location of data centers, becoming instead a novel political space
of calculative reasoning. Returning to the site of the icite program, what
kinds of perceptions and calculations of the world become possible with the
algorithmic instruments that gather in cloud space? When the seventeen US
intelligence agencies upload or analyze data in icite, they access software as
a service, so that they are not merely “joining the dots of their data,” as the
post-9/11 measures claimed, but are in fact combining their modes of analysis.
Among the available icite platforms is Digital Reasoning software, a set of
machine learning algorithms for analyzing and deriving meaning from data
across government intelligence databases and open source social media data
streams. Describing the Synthesys application available to analysts via icite,
Digital Reasoning claims that its algorithms are able to “read,” “resolve,” and
“reason” from the correlations in cloud data. The algorithms are said to “ex-
tract value from complex and opaque data” and to “determine what’s impor-
tant” among the “people, places, organizations, events and facts being dis-
cussed,” ultimately “figuring out what the final picture means” by comparing
it with “the opportunities and anomalies you are looking for.”42 What this
means in terms of the practices of the algorithms is that they identify clusters
within the data (to read), derive attributes from those clusters to make them
recognizable into the future (to resolve), and compare the output of the algo-
rithms with the target output of the analyst (to reason). This is an iterative and
experimental process in which the humans and machines feel their way toward
solutions and resolutions to otherwise indelibly political situations and events.
Though Digital Reasoning’s machine learning algorithms are now the
mainstay of US government defense analysis, they were developed initially for
anomaly detection in the wholesale financial industry. The frenzy of the 2008 43
financial crisis included trading strategies that were in breach of regulations,
gest why this form of algorithmic reason has significance for our ethicopoliti-
cal present.
Condensing Traces
The algorithmic techniques of Cloud II involve condensing traces, practices not
primarily concerned with seeing or bringing into vision, but rather focused on
engaging a subvisible world by inferring from the traces and trajectories that
condense at indeterminate points. Returning to my analogy with the appara-
tus of the cloud chamber, by the mid-t wentieth century, when Charles Wil-
son’s cloud chamber was being used in subatomic physics, the “purpose” of the
instrument was described as being “to study the motion of ionizing particles
from records of the drops condensed on ions formed along the trajectories fol-
lowed by these particles.”48 The motion of particles could not be observed di-
rectly, but their trajectory could be perceived obliquely, via the visible drops
condensed on the ions — the cloud “tracks.” Figure 1.7 shows one of the best-
known cloud chamber photographs, C. T. R. Wilson’s image of alpha-emitting
thorium, with the cloud originating from an alpha ray passing through the
chamber, its “trajectory disturbed in two places,” as recorded in the atlas of
cloud chamber images.49 The newly available images of radioactivity made the
object perceptible via the records of condensed drops on the ions, observing
the motion of the particles obliquely. In the compelling images from the cloud
chamber, one can locate a capacity to perceive the movement of otherwise sub-
visible entities. The chamber of the cloud chamber is akin to an apparatus, in
Michel Foucault’s terms, in that it “inserts the phenomena in question” within
a “series of probable events.”50 In this sense, to condense a trace is not to show
or to reveal something existing in the past, but rather to establish a condensed
series of possible correlations between entities and events. The cloud cham-
ber apparatus is experimental to the extent that it is concerned with probable
tendencies and trajectories, condensing a larger volume down to the probable
event.
Figure 1.7 Cloud chamber tracks of alpha-emitting thorium. C. T. R. Wilson,
c. 1923, Proceedings of the Cambridge Philosophical Society.
Like the cloud chamber apparatus of the twentieth century, contemporary
cloud computing is a form of experimental chamber or apparatus. It condenses
and reduces the volume of data down to that which is probable or possible in
the algorithmic model. When Digital Reasoning claims that their algorithms
determine what is important in war or in finance, this is because their ma-
chine learning apparatus is generating what matters in a series of probable
events. As Karen Barad writes on the nature of scientific apparatuses, they are
46 boundary-making practices that decide “what matters and what is excluded
from mattering,” as they “enact agential cuts that produce determinate bound-
Chapter One
ods: when the events are predicted to occur. You can explore the past, present and
predicted future of almost anything in a matter of seconds. Our analysis tools facili-
tate deep investigation to better understand complex relationships, resulting
in actionable insights.”68 Consider the claim: the data stream of social media
contains all the attributes of incipient future events, so that one can explore
the past, present, and predicted future of almost anything. Recorded Future’s
applications run their algorithms across the boundary of public and private
cloud computing so that the analyst can explore the correlations between, for
example, classified structured data in the dhs’s files and the language and sen-
timent analysis of so-called open source Twitter feeds and Facebook posts. In
this way, the technology enables action in the present, based on possible corre-
lations between past data archives (such as national security lists) and archives
of the predictive future.
With this instrument of Cloud II, the analytic is everything. Archived
data in Recorded Future becomes unmoored and derivative of its context, even
the so-called dirty, or noisy, data no longer muddying the calculation but ren-
dered useful. As the security analysts draw together social media “junk” data
with other labeled entities, tagging metadata and sharing with other agencies,
diverse data elements are rendered commensurate and made actionable geo
politically. As Orit Halpern suggests in her account of how algorithmic ratio-
nality became a governmental and social virtue, digital computation changes
the nature of the archive. The nineteenth-century form of “static” archiving
and repository is supplemented in the twentieth century by what Halpern calls
“an active site for the execution of operations.”69 Similarly, in the nineteenth-
century cloud chamber’s attempt to reproduce nature, the cloud tracks had
been considered spurious dirt effects, not for scientific archiving. Yet, it was
the tracks that became the thing of interest, the atlas of cloud chamber images
knowing only the event of the track itself. Contemporary cloud computing is
an active site for the execution of operations, as understood by Halpern, in
which the archive is generative of particular imagined futures.
The archivization of specific data elements with applications such as Re-
corded Future, then, produces particular futures; “the archivization produces
as much as it records the event,” as Jacques Derrida writes.70 As the photo-
graphic recording of the cloud tracks within the cloud chamber archived the
possibility of recognizing future subatomic particles, so the digital recording
of social media data in cloud computing archives the possibility for future ac-
tions at the border or on the city street. Understood in this way, the spatial
power of the data center as “archive” could be critically challenged, as we see 51
in Trevor Paglen’s images, while leaving entirely intact Halpern’s “active site of
The peculiarity of men and animals is that they have the power of adjusting
themselves to almost all the features [of their environment]. The feature to which
adjustment is made on a particular occasion is the one the man is attending
to and he attends to what he is interested in. His interests are determined by
his appetites, desires, drives, instincts — all the things that together make up
his “springs of action.” If we want to construct a machine which will vary its
attention to things in its environment so that it will sometimes adjust itself to one
and sometimes to another, it would seem to be necessary to equip the machine
with something corresponding to a set of appetites.
— Richard Braithwaite, “Can Automatic Calculating Machines
Be Said to Think?”
Springs of Action
In January 1952 the bbc recorded what was to be the world’s first public de-
bate on the mathematics and ethics of machine learning. Participating in the
discussion of the question “Can Automatic Calculating Machines Be Said to
Think?” were Manchester mathematicians Alan Turing and Max Newman;
their neurosurgeon colleague Geoffrey Jefferson; and Cambridge moral phi-
losopher Richard Braithwaite. Turing explains that he has “made some ex-
periments in teaching a machine to do some simple operation,” but that “the
machine learnt so slowly that it needed a great deal of teaching.” Jefferson is
skeptical of the use of the verb “to learn” in relation to the machine, and he in-
terjects to ask Turing a question: “But who was learning, you or the machine?”1
This distinction, between humans and machines as the locus of learning is
of great significance to the neurosurgeon, for whom the electronic circuits
of computing machines were not analogous to the “fragments of the nervous
system” he encountered in the fleshy materiality of the human brain.2 Yet, Jef-
ferson’s question prompts Turing to reflect on his own embodied experience
of experimenting with his machines. In the audio recording of the debate,
Turing can be heard to pause for a moment’s reflection before responding: “I
am inclined to believe that when one has taught it [the machine] to do certain 57
things[,] one will find that some other things one had planned to teach it are
data, the we of surgeon and robot restlessly seeks an optimal spring of action —
the optimal incision, the optimal target of tumor or diseased organ, the opti-
mal trajectory of movement. Intuitive Surgical’s robots hold the promise of the
augmented vision and precise trajectories of movement of a composite being of
surgeon and machine. As one of the uc Berkeley robotics scientists describes
what they term “iterative learning,” machine learning allows “robotic surgi-
cal assistants to execute trajectories with superhuman performance in terms
of speed and smoothness.”14 The drive for superhuman learning iterates back
Figures 2.2 and 2.3
The da Vinci surgical
robot. Intuitive
Surgical, 2018.
and forth across multiple gestures. As the hand and eye movements of a sin-
gular surgeon are tracked in the da Vinci’s api, they commune with a mathe-
matical model generated from a vast multiplicity of data traces of past surgical
gestures. These gestures, in turn, modify the learning model to optimize and
augment the future trajectories of future surgeons and robots as yet unknown.
As the philosophers and scientists in the 1952 debate anticipated, the machine
learning algorithms have something close to appetites, extracting and model-
62 ing features from the plenitude of cloud data, rendering springs of action, and
acting on future states of being.
Chapter Two
During the course of following the surgeons and robots of one world-
leading oncology department in a UK teaching hospital, I began to note the
many occasions when a surgeon used “we” in place of “I” to describe their
daily collaborations with surgical robots. This we who learns is expansive. It in-
cludes the many humans in the research group and the surgical team, but also
the many human and technical components of the simulation of a surgery —
the multiple layers of medical imaging, video data from past surgeries, and al-
gorithmic models that together composed a kind of virtual presurgery.15 For
example, when the UK surgical team were preparing to conduct a new surgi-
cal procedure on a specific type of tumor, they collaborated with other US
surgeons who had previous experience of the procedure. This was not merely
a dialogue between human experts, however; the research team also imported
the data from the US surgeries, inputting them in an algorithmic model and
experimenting with the parameters for a new context. As Rachel Prentice doc-
uments in her meticulous ethnography of surgical education, “surgical action
must be made explicit for computers” so that “bodies and their relations in
surgery are reconstructed” in a form that “can be computed.”16 This reciprocal
and iterative relationship between human surgeon and computer is what she
calls a “mutual articulation” in which “bodies affect and are affected by” one
another.17 When the movement of a surgeon’s hands is rendered computable
in the robotic model, Prentice suggests that this “instrumentalization” over-
looks the “tacit” and “tactile experience” of surgery, such as the “elasticity of a
uterus or the delicacy of an ovary.”18
In Prentice’s reading of surgical technologies, the tacit, tactile, and intui-
tive faculties of surgery define the human as the locus of care and embodied
judgment and decision. Yet, in giving their accounts of working with robots, I
found that human surgeons testify to their own body’s capacities coming into
being in new ways. What it means to be intuitive, to touch or to feel an organ,
for example, alters with the advent of machine learning modalities of surgery.
Seated at their virtual environment console, the surgeons access video feed
63
images from the endoscopic arm of the robot (figure 2.4). These images are
overlaid with mri and the fluorescence images of tumors so that, as one of the
da Vinci computer scientists explains, “the system provides over a thousand
frames per second and filters each image to eliminate background noise.”19 The
work of the algorithms here is to extract the features of interest for perception
by the surgeon, surfacing the optimal image from a background noise of teem-
ing data.
One surgeon described to me the daily work of obstetric surgery with her
da Vinci robot, noting that through the expanded mediated space, she is able
to “see the unseeable” and “reach the unreachable” within the patient’s body.20
In contrast to Prentice’s sense that human touch — and with it, judgment and
decision — is evacuated and instrumentalized by surgical robotics, here the en-
tangled touch of surgeon and robot invokes different relations of judgment
and decision. The surgeon’s relations to herself and to others — to her patients
past and present, her operating theater colleagues, the robots, images, tumors,
surgical instruments — are altered in and through the machine learning algo-
rithms. “Touch engages us in a felt sense of causality,” writes Karen Barad, so
that “touch moves and affects what it effects.”21 Where Prentice foresees a loss
of responsibility as the human surgeon’s touch is evacuated by the robot, I pro-
pose instead an extension of responsibility to that which extends and exceeds
human sensibility. The difference is an important one. The terrain of uncer-
tainty regarding whether a specific tumor is operable without major damage
to surrounding organs, for example, shifts with the 360-degree mobility of the
robot’s wrist — what can be precisely touched, sensed, and extracted from the
body is altered. Indeed, leading computer scientist and artist Ken Goldberg,
alongside his carefully presented accounts of the development of stochastic
64 models and neural networks for the performance of surgical excision by ro-
bots, writes essays on the insensible and uncanny worlds that open up with
Chapter Two
The neural network’s capacity to learn by extracting features from its data
environment has made it flourish in the algorithmic architectures of drones,
autonomous vehicles, surgical and production robotics, and at the biometric
border.24 This capacity to learn something in excess of taught rules has also
characterized the public concern and ethical debates around autonomous sys-
tems. Whether in the neural net algorithms animating surgical robots, au- 65
tonomous weapons systems, predictive policing, or cloud-based intelligence
The posthuman does not really mean the end of humanity. It signals in-
stead the end of a certain conception of the human, a conception that
may have applied, at best, to that fraction of humanity who have had the
wealth, power, and leisure to conceptualize themselves as autonomous be-
ings exercising their will through individual agency and choice. What is
lethal is not the posthuman as such but the grafting of the posthuman
onto a liberal humanist view of the self.26
Hayles’s concerns for the grafting of the posthuman onto the figure of an
autonomous liberal subject echo across the making of intuitive machine learn-
ing worlds. Though technologies such as Intuitive Surgical’s robot actively dis-
tribute and extend the parameters of sight, touch, and cognition into posthu-
man composite forms, their ethical orientation is defined solely in relation to
the control of an autonomous human subject. While human surgeons speak of
an indeterminate we who learns, decides, and acts, nonetheless the capacity for
judgment retains its Kantian location in the unified thought of a reasoning hu-
man subject.27 Thus, when a violence is perpetuated or a harm is registered —
damage, prejudicial judgment, or death — the only ethical recourse is to an
imagined unified entity who secures all representations. So, for example, in a
series of legal cases against Intuitive Surgical, the reported harms include the
66 rupture of tissue, burns, and other damage to organs, severed blood vessels and
nerves, loss of organ function, and fatalities.28 In these juridical cases, where
Chapter Two
This space of play, however, also opens onto an infinite array of combinato-
rial possibilities in terms of the malleable and adaptable inputs, parameters,
and weights of the model. As one designer of machine learning algorithms
for anomaly detection frames the question, “What is normal?” and “How far
is far, if something is to be considered anomalous?”34 Precisely this adaptive
threshold between norm and anomaly was being negotiated between the labo-
ratory and border operations. A small adjustment in the threshold will gener-
ate an entirely different set of outputs and, therefore, a change in the spring of
action. I have observed this iterative process of playing with the threshold in
multiple situations where algorithms are being trained for deployment, from
police forces adjusting the sensitivity of a facial recognition algorithm to casi-
nos moving the threshold for potential fraud:
What is happening here is that the neural networks are learning to recog-
nize what is normal and anomalous at each parse of the data. But, the shifting
of the thresholds for that recognition embodies all the valuations, associations,
prejudices, and accommodations involved in determining what is “useful” or
“good enough.” Sometimes, as with semisupervised machine learning, this re-
gime of recognition emerges iteratively between humans, algorithms, and a
labeled training dataset. In other instances, unsupervised machine learning
will cluster the data with no preexisting labeled classifications of what is or
is not useful or of interest. Even in this apparently unsupervised process, hu-
mans recalibrate and adjust the algorithm’s performance against the target. In
short, in all cases, machine learning algorithms embody a regime of recogni-
tion that identifies what or who matters to the event. Machine learning algo-
rithms do not merely recognize people and things in the sense of identifying —
faces, threats, vehicles, animals, languages — they actively generate recogniz-
ability as such, so that they decide what or who is recognizable as a target of
interest in an occluded landscape. To adjust the threshold of what is “good 69
enough” is to decide the register of what kinds of political claims can be made
Inputs
Thresholds of Target
Space Output
Video play
Social media
Images Output
71
Weights Decision
Transactions
Probability
Sentiment
of each neuron to the next “hidden layer” dependent on whether the weighted
sum of inputs is less than or greater than some threshold value. Each layer of
the neural network detects some discrete aspect of the figure. As the computer
scientists describe image recognition, “the learned features in the first layer”
detect “the presence or absence of edges,” with the second layer “spotting par-
ticular arrangements of edges,” and the subsequent layers “assembling motifs
into larger combinations that correspond to parts of familiar objects.”44 The
recognition of edges, motifs, and familiar arrangements is not designed into
rules by a human engineer but is definitively generated from the exposure to
data. To be clear, this spatial arrangement of probabilistic propositions is one
of the places where I locate the ethicopolitics that is always already present
within the algorithm. The selection of training data; the detection of edges;
the decisions on hidden layers; the assigning of probability weightings; and the
setting of threshold values: these are the multiple moments when humans and
algorithms generate a regime of recognition.
Adjusting the Features: The Variability of What Something Could Be
The computational problem of how to recognize people and things has become
of such commercial and political significance that computer scientists enter
their experimental algorithms in competitive image recognition contests. One
particular algorithm, the AlexNet deep convolutional neural network, won an
image recognition contest in 2012 and has become the basis for multiple subse-
72 quent commercial and governmental recognition algorithms, with the scien-
tific paper cited more than twenty-four thousand times.45 The AlexNet gives
Chapter Two
When deep neural network algorithms learn, then, they adjust themselves in
relation to the features of their environment. To be clear, to learn, they have
to weight some data elements of a feature space more than others — they have
to have assumptions about how the world is ordered. Notwithstanding the
widespread societal calls for algorithms to be rendered free of bias or to have
their assumptions extracted, they categorically require bias and assumptions
to function in the world. Indeed, even the textbooks used by the next genera-
tion of computer scientists address directly that “there can be no inference or
prediction without assumptions,” particularly the assumptions of “the proba-
bility assigned to the parameters.”50 Thus, when a team of European computer
scientists discuss how they might move the threshold for their neural net algo-
rithm to recognize the likelihood of a person of interest (a future person, yet
to arrive) being a “returning foreign fighter” and not a “returning aid worker”
from Syria, they mean that they will adjust the sensitivity of the algorithm to
particular elements of weighted input data, such as increasing the probability
weighting of particular past flight routes.51 While some of this adjustment of
the threshold is done by humans, today much of it is invested in the power
of the algorithm to adjust itself in and through the emergent properties of
the data, understood as a feature space. The “we both” of Turing’s reflections
seems to reassert itself here in the accounts of adjustment given by computer
scientists Yann LeCun, Yoshua Bengio, and Geoffrey Hinton of Facebook ai,
Google, nyu, and the University of Toronto: “We compute an objective func-
tion that measures the error (or distance) between the output scores and the
desired pattern of scores. The machine then modifies its internal adjustable
parameters to reduce this error. These adjustable parameters, often called
weights, are real numbers that modify the input-output function of the ma-
chine. In a typical deep-learning system, there may be hundreds of millions of
these adjustable weights.”52
Like the abductive methods of intelligence gathering I discuss in chapter
1, this computational method observes the effect of the calculation — or the
output signal — a nd theorizes back to the adjustment of parameters, like the
mechanical knobs on a calculating machine. The distance between an agreed
target output, or desired pattern, and the output scores is the error or the bias.
Significantly, for the algorithm, error is distance; it is the playful and experimen- 75
tal space where something useful or “good enough” materializes. There is noth-
ambient lighting, or the human operator.”54 My point, though, is that what the
sensor can sense, or the operator can decide, is only meaningful in the context
of how the neural nets arbitrate what the objection could be, what it could
mean. Similarly, in a Volkswagen factory in 2015, a robot failed to recognize
the outline of a human coworker on the production line, mistaking him for
a car door and crushing him to death. At the level of machine learning algo-
rithms and their regimes of recognition, the da Vinci surgical robots’ failures
to recognize the boundary delimiting kidney tumor from human organ is not
dissimilar to the biometric facial recognition systems that closed automated
border controls at an airport when the setting sun changed the ambient light-
ing conditions. In all these instances, the algorithms have learned from the
features of the environment they have been exposed to. Sometimes events and
sensors in the environment will present them with a set of input features they
have not encountered previously, and their assumptions and weightings may
lead to a spring of action that misrecognizes the target.55 Is this an error? Or is
error merely a matter of distance?
The contemporary advent of cloud robotics has sought to address this
problem of the limit point of exposure to features in a multidimensional en-
vironment. Cloud-based robotics, as we saw in the discussion of surgical ro-
bots, circulate data and aggregate computational power across a distributed
system of machine and human learning. Just as the surgical robots are no lon-
ger limited to the data and computation stored within a bounded system, so
the cloud-based intelligence system I discuss in chapter 1 recognizes its targets
from exposure to data and analytics methods across borders and jurisdictions.
Where machine learning intersects with cloud computing, the neural network
algorithms are exposed to features from a vast archive of cloud data, includ-
ing the Point Cloud Library of open source 2d and 3d images.56 A point cloud
is a set of topological data points mapping the 3d space of objects. Computer
science research in cloud robotics is addressing the question of whether expo-
sure to a vastly increased volume of point cloud data on objects can optimize
the neural network’s capacity to learn how to recognize and to act. Consider,
for example, the computer science team at uc Berkeley’s Automation Sciences
Lab, whose research into cloud robotics is funded by the nsf, Google, and the
US Department of Defense. Presenting their Dex-Net 1.0, or Dexterity Net-
work algorithm, the Berkeley scientists experiment with cnns to optimize the
capacity of a robot to recognize and grasp a range of objects. The algorithm
represents an advance on image recognition technologies such as AlexNet be-
cause it recognizes 3d objects from multiple viewpoints, and it outputs an op- 77
timal action based on this recognition.57
Figure 2.7 Point clouds for object recognition and grasp optimization.
Mahler et al., “Dex-Net 1.0.”
terms, the weightings and probabilities of the point cloud make it possible to
precompute something, so that the encounter with the unknown object can
always yield an action that is optimal. With the point cloud, an algorithmic
system does not only ask “Is this a face?,” “Is this a bottle?,” “Is this a military
vehicle?,” but rather it has already precomputed an optimal action in relation
to the topologies it has encountered.
On the horizon of research in neural net algorithms for robotics, then, one
finds this notion of precomputation to make an action achievable amidst clut-
ter and occlusions. Precomputation captures the neural net computational
problem that extends from the recognition and grasp of the shape of a hu-
man organ amid “noisy” surgical data, to the recognition of a civilian body
in the screened occluded data of the drone. One computer scientist, who now
designs algorithms for human gait recognition, described to me how he had
been “working with surgeons modeling the perfect operation,” this human-
algorithm collaboration itself deploying point clouds to precompute the trajec-
tory of “a perfect procedure.”61 He has developed his algorithms for multiple
applications, with each iteration another step in optimization: “The funda-
mental thing is we know what good looks like,” he explains, “and then if you’re
doing anomaly spotting, you can see when something’s wrong.” For example,
when the online gambling platform BetFair sought algorithms to recognize
and act on patterns of addition, the designer suggests, “that’s something we
can spot because we know what addicted play looks like.” This malleable nor- 79
mative assumption of “what good looks like” or “what addictive play looks
outlined, the spatial arrangement of the neural net algorithm contains within
it multiple probabilities, infinitely adjustable weights whose emergent effects
can never be entirely known, even to the designer. The finite elements of each
hidden layer of the neural net, one might propose, contain within them infi-
nite possible correlations to other elements. The spatial arrangement of the
neural net does not foreclose alternative readings, different arrangements of
what or who matters and what or who does not. Crucially, however, at the
point of action, this intrinsic multiplicity is reduced to a single output. The
insistence on a single output is the algorithm’s orientation to action. Though I
am reminded by the computer scientists that the output need only be between
0 and 1, and that there are infinite numbers between 0 and 1, there is none-
theless a single numeric output. Let us not forget that the algorithm’s output
signal lies behind the risk score at the border, the credit decision, the target as-
sessment of the drone, and the decision on sentencing, detention, or the incipi-
ent dangers of a gathered protest on a city street. It is as though all the many
potentials held in parallel, simultaneously distributed across the layers of the
neural net, could never have been. With the output of the machine learning
algorithm, one might say, things could never have been otherwise. The output
is a probability whose value is transformed by the smallest of adjustments in
the parameters of the model. And yet, nonetheless, all political uncertainty is
rendered tractable on the horizon of the action triggered by this single output.
At this point, one might reasonably ask how giving such an account of the
contingent politics of machine learning algorithms is of any possible critical
use. How might a cloud ethics work with the incompleteness, the undecid-
ability, and the contingency of the algorithm’s space of play? If one wants to
inquire whether a given algorithm is responsible for a flash crash in the fi-
nancial markets, or if one seeks a human rights law adequate to the task of
holding autonomous weapons or autonomous surgery to account, then some
ethical grounds might be considered essential — or at least some method for
accountability. As Michel Foucault proposes in his discussion of ethics, how-
ever, what may be necessary is not to appeal to grounds or to the juridical do-
main of statutes, but rather to “ask politics a whole set of questions that are
not part of its statutory domain.”63 A cloud ethics must be capable of asking
questions and making political claims that are not already recognized on the
existing terrain of rights to privacy and freedoms of association and assembly.
Cloud ethics belong properly not to the individual as bearer of rights, but to
the many touch points and data fragments that are aggregated from the rela-
tions between subjects and objects. Thus, a cloud ethics must be capable of 81
asking questions such as How did that Dex-Net algorithm weight the prob-
Attribution
This page intentionally left blank
3
The Uncertain Author
Writing and Attribution
I always think the notion of the fork in the road is very important when you are
creating narrative, because you are continually coming to forks. You don’t exactly
know where you are going. . . . This is a state of uncertainty, or in terms of the
modern physics, indeterminacy.
— John Fowles and Susanna Onega, Form and Meaning
Source Code
In October 2017 the New York City Council proposed a new local bill — the Al-
gorithmic Accountability Bill — conceived as a means to render the city’s algo-
rithms accountable for the significant effects their use has on the life chances
of the people of New York City.1 The city council’s attempt to secure the ac-
countability of algorithms was a response to the racialized harms precipitated
by algorithmic decisions in the criminal justice system, where automated risk
calculations further entrenched the notion that young African American men
posed a high risk of reoffending and should serve custodial sentences. 2 As in
many other cases when an algorithm has caused a harm or injustice, the reflex
response of the authorities was to seek out the origin of the algorithm and lo-
cate the source of its harmfulness. In the first iteration of the Algorithmic Ac-
countability Bill, the text called for public and private agencies “to publish the
source code” of all algorithms being used for “automated decision systems,”
“policing individuals,” or “targeted services” and “imposing penalties.” The
sponsor of the legislation, council member James Vacca, recorded his concerns
that automated decisions made by algorithms were discriminating against
New Yorkers based on the characteristics of race, age, religion, gender, sexual
orientation, and citizenship status. Announcing that his ambition was to seek
“transparency as well as accountability,” Vacca located the actions and effects
of the algorithm in what he saw as the determinate and discriminatory agency
of the source code.
Though the Algorithmic Accountability Bill was subsequently amended
to protect the commercial interests of algorithm vendors, before it was passed
unanimously in December 2017, the appeal to a source code for the algorithm
represents a broader impulse to locate the origins of the algorithm, or more
86 precisely, to trace the source of the algorithm to an originary act of authorship.
Put simply, the source of the algorithm becomes attached to the unified body
Chapter Three
of its code, anchored in the modern notion of the author who writes the code.
The responsibility the algorithm can be said to have — for discrimination, for
errors, for mistakes and injustices — is thus secured via the attribution of the
authorship of code. This appeal to the authorship of the algorithm character-
izes the many contemporary demands for accountable and transparent algo-
rithms or, as I have described it, demands for an encoded ethics for algorithmic
decision systems. An encoded ethics reasserts the human author of mathemat-
ical code and locates the control of algorithms in the reasoning authority of
the author. In some instances, as in the New York City Council’s scrutiny of
source code, the appeal to authorship is made through the body of work, via
the locus of an identifiable source code. Here, code is assigned a unity of the
body of work in the same way that authorship produces a unity from an oth-
erwise errant and distributed text. In other manifestations, an encoded ethics
is sought via an appeal to the writers of algorithms to adopt an ethical code of
conduct that would secure the moral grounds for their writing of code.3 Again,
the location of ethics in the reasoning subject is mirrored by the irrevocable
binding of the author to the body of work. Responsibility for the onward ac-
tions and effects of the algorithm is imagined to be secured through the disclo-
sure of authorship. Moreover, the call for accountable and ethical algorithms
binds together not only source code and author but also data sources and au-
thorship, so that it is said that “designers must make their data sources pub-
lic.”4 A semblance of accountability for how the algorithm comes into being is
found in the disclosure of the data sources from which the author writes. In
sum, through the appeal to authorial sources and ethical design, much of the
contemporary debate on algorithmic accountability is imbuing the algorithm
with a specific form of what, with Michel Foucault, we might call an “author
function.”5 This novel form of algorithmic author function has profound con-
sequences for how one understands, and responds to, the ethicopolitical rela-
tions forged through the writing of algorithms.
If the source code is the place where the logics of the algorithm can be
found and disclosed to the world, then what exactly is the nature of this source?
As I explain in chapter 2, in the context of how machine learning algorithms
learn to recognize, the algorithm’s ways of being in the world are not all present
in the code. Indeed, the effects of the algorithm are generative and emergent so
that the source code is never complete. Put differently, the source is continually
edited and rewritten through the algorithm’s engagement with the world. To
understand the latent discriminations of the New York City algorithms neces-
sarily involves engaging with its reincorporation of new entities into its code. In 87
this sense, the sources and authors of the algorithm are continuously modified
author Susanna Onega reflect in the opening to this chapter, the practice of
writing involves encountering “forks in the road” at which it is never certain
what might take place as a result of the decision at the branch. To write, as
Fowles experiences it, is necessarily to encounter an uncertain and indetermi-
nate future. Though the writing of algorithms and the writing of literature is
not at all the same thing, of course, the bifurcated “forks” in the algorithm’s
calculative architecture do involve a profound uncertainty of authorship and
its future effects. Conceiving of writing as a practice in which scattered ele-
ments and contingent forks in the road are gathered and unified, the writing of
algorithms does constitute something akin to an ethicopolitical writing prac-
tice that involves “manifesting oneself to oneself and to others.”9
Returning to my account of the ethics of algorithms located in the rela-
tions of self to self and self to others, writing algorithms shares with the nov-
elists’ accounts the impossibility of a determinate and grounded authorship.
In place of locating ethics in the individual subject of the author and the au-
thorial source of the work, the ethics of algorithm emerges from the compos-
ite and collaborative practices of writing in which we, and our scattered data
elements, are also implicated and engaged. To propose that algorithms dwell
within us, and we within them, is also to suggest that through these relations
an ethics of writing algorithms is forged. In this reformulated approach to au-
thorship and the algorithm, it would be insufficient to respond to the violences
and injustices of an algorithm by “scrutinizing the source code” because the
act of writing these algorithms substantially exceeds that source, extending
into and through the scattered elements of us and the multiple data points
that gather around us.
The computer science field of natural language processing (nlp) has become
of intense interest to those developing experimental deep learning algorithms.
Beginning in the mid-twentieth century with models for automatic language
translation, nlp for much of its history has used basic decision tree algorithms
and rules-based techniques to render human language computable.10 In my
book The Politics of Possibility, I traced Rakesh Agrawal’s rule-based method of
text mining as it developed from the mining of customer retail basket data in
the early 1990s to the mining of data for counter-terrorism after 2001.11 With
the availability of volumes of cloud-based unstructured data in the twenty-
first century, however, nlp methods have shifted from the rules-based deriv-
ing of semantics from text to deep learning algorithms generating meaning 89
from their exposure to a corpus of data derived from speech or text.12 The
with the target output defined as a probability that this sentence was a match
to a given second sentence. At this point the data was unlabeled — the algo-
rithm was not given the name of the author — yet it made a correct prediction
in 83 percent of the input queries. As the Google team account for their ex-
periments, their guiding question was, “Can we build a system that can, given
a sentence from a book and knowledge of the author’s style and ‘personality,’
predict what the author is most likely to write next?” In a later phase of the ex-
periment, the team built a generative model that was able to respond to ques-
tions in the style of a specific author, said to be predicting the future text of
“dead authors” from “beyond the grave.”16 In short, the algorithmic technol-
ogy of the attribute is not about identifying the qualities of an entity in the
present, but about rendering something attributable into the future.
The significant point here is that the author function in contemporary
deep learning algorithms is serving a futures-oriented capacity to associate en-
tities through their attributes. The name and source “identity” of the author
may be unknown or anonymous, but nonetheless the algorithm is deriving
an author function from the clustering of attributes. Put simply, what mat-
ters to these deep learning algorithms is not primarily the naming of an au-
thor but the generating of a set of attributes that could be latent in any other
entity. As I have similarly described the border-control algorithms that can
act on an individual indifferent to their naming or identification — as a set of
associations — so natural language algorithms need only to be able to associ-
ate the attributes sufficiently to say “this sentence X matches the next possible
line Y with probability P.”17 Even where the strict naming of the author is not
possible, the algorithm can work productively with the attributes of the body
of work.
One might reasonably ask what political or ethical significance could pos-
sibly be attached to this distinctive learning of attributes via clustering tech-
niques. Yet, the computational capacity of nlp algorithms lies behind many
of the apparently autonomous decision-making systems that are of ethical and
political concern. From the extraction of text from images of public protests,
to the analysis of the sentiment of social media text for detecting the pro-
pensities of a person, these algorithms are concerned precisely with the infer-
ence of possible futures that could be associated with a set of attributes. The
Google team explains the motivation for the one thousand authors project as
“an early step towards better understanding intent.”18 Given the attributes of
a given cluster within a corpus of data, what is the future intent? The predic-
tive power of the attribute is located in the capacity to infer meanings in the 91
future. Thus, for example, how does the intent of one line of text — “I will go
the actual work of the algorithm as it iterates back and forth, adjusting prob-
ability weights and thresholds and modifying its own code. This annexing of
the writing of source code from the back-and-forth iterations of an algorithm
rewriting has profound implications for ethics. The designers of facial recog-
nition algorithms for policing, for example, commonly testify that bias and
discrimination reside in the human operator and not within their code. When
one follows the iterative writing of the algorithms in practice, however, one
sees that the actions of a human operator are in close communion with the
algorithms, so that each decision also recalibrates the model. In short, to an-
nex the writing of code from the iterative writing in the world is to seriously
miss the ethicopolitical life of algorithms. As Foucault suggests, the preserved
dominance of the author function hinders us from “taking the full measure”
of the author’s disappearance and attending to what is happening in the gaps
and spaces of that disappearance.23
Foucault’s writings on the subject of authorship also afford us a glimpse of
what might be happening when texts are grouped together for machine learn-
ing. In the Google experiments, what matters for the algorithm is not primar-
ily who the author is, but rather what attributes make a set of texts part of
a cluster defined by degrees of similarity. For Foucault, the author function
serves a kind of “classificatory function” in that it “permits one to group to-
gether a certain number of texts, define them, differentiate them from and
contrast them to others.”24 Though the form of algorithmic author function
I wish to depict is not classificatory in the strict sense of grouping things by
similarity, it does serve a clustering function in terms of finding structure in
data and learning what normal or abnormal looks like. In this sense the au-
thor function retains its privilege to find a resolution of differences and ten-
sions in the “authentication of some texts by the use of others.”25 To be clear,
it is profoundly misleading to appeal to the author of the algorithm or to the
source code as the locus of responsibility because the algorithm’s power lies
in its capacity to promise to resolve incalculable differences via the unity of
its outputs. To invoke the call for an identifiable and responsible author is not
only insufficient as critique, it also risks amplifying the already present ability
of the algorithm to “permit grouping together” so that “incompatible elements
are tied together,” with “all differences having been resolved.”26 In place of the
ethicopolitical call for securing responsibility via authorship, then, I am inter-
ested in deepening our sense of how the algorithm ties together incompatible
elements to appear to resolve the politics of difference.
93
Authorship and the Algorithm
some of the breaches might be in the author function within mathematics. Ac-
knowledging the plurality of self involved in the literary novelist’s use of I in
the first-person narrator’s account, he points to the “scission” between the au-
thor and the “real writer,” the “plurality of self ” that opens up in all cases “en-
dowed with the author function.” Crucially, this plurality of self is specifically
present also in “a treatise on mathematics,” where authorship splits, divides,
and disperses so that it “can give rise to several selves, to several subjects.”31 A
significant opening, then, begins from the way the first-person I as a locus of
authorial responsibility is actually always divided and dispersed. This is not
only a matter of the dividuated subject, divided within herself, as is often doc-
umented in the accounts of digital subjects. The plurality of authorship also
involves a dispersal across different human and nonhuman life forms.32 For ex-
ample, as the neural net algorithms animating robots become more sensitive
to context and the attributes of behavior, they are modifying their own actions
by learning from their interactions with other robots.33 The algorithms are not
simply the “authors of their own code” in the sense of locating authorship in
an autonomous machine, but instead they are engaged in acts of writing and
rewriting that unify otherwise scattered and heterogeneous elements.
So, when a deep neural net predicts a deceased author’s next sentence, or
when a New York City policing algorithm decides the attributes of a person
at high risk of reoffending, within the author function teems a multitude. In
attempting to find mechanisms of accountability for algorithms, our societ-
ies are searching for an authorial source of the writing of code. Yet, this desire
for a first-person author as the locus of agency and responsibility deepens the
power of the algorithm to present itself as a unified entity.34 Simultaneously,
it underplays the contingency of the algorithm and conceals the breaches that
might otherwise appear in the constraining figure of the author. In the fol-
lowing pages I propose a different approach to authorship and the algorithm,
foregrounding the multiple and dispersed acts of writing that are its condition
of being.
Writing and the Algorithm
the dna profile of an individual cannot be identified from the materials at the
scene — where the trace is incomplete or indeterminate — the likelihood ratio
algorithm allows for partial samples to be “positively associated” with an indi-
vidual by comparison of their attributes to those of a group of unknown oth-
ers. A dna profile match becomes, via the algorithm, a correlative and scenario-
based dna positive association.
There can be little doubt of the profoundly troubling implications of using
such algorithms to convict individuals of crimes.38 When one looks closely at
what happened when the source code of fst was released to the defense law-
yers, however, it becomes clear that the source of harm resides not in the code
itself but in the iterative and heterogeneous writing of a likelihood ratio. Once
one relinquishes the notion that the authorship of the algorithm resides in the
code, following instead the writing of the algorithm, the code is not the au-
thorial source of the algorithm’s actions but merely one element in an iterative
process of writing. Among the defense team expert statements — each of them
concluding that the algorithm could have resulted in innocent people serv-
ing a prison sentence for serious offenses — one computer scientist, Nathaniel
Adams, reported what he found when he examined the fst algorithm. His
findings do not pertain to an error or flaw in the source code or even to a bias
contained therein but specifically to the way in which the algorithm learns to
write a likelihood ratio from a body of data. In a method that is resonant with
the Google machine learning algorithm’s prediction of the next sentences of
deceased authors based on a corpus of literary data, the fst learns from “the
databases accompanying the source code.”39 These databases include the train-
ing and validation sets from which the likelihood ratio model is generated.
The algorithm modifies its output according to adjustments in the input data.
Adams experimented with adjustments to the weights of the algorithm and
demonstrated the contingency of the writing of a likelihood ratio when it is
expressed as an output score. Of particular significance is that the algorithm’s
sense of what the unknown population looks like in terms of dna is clustered
along racialized attributes. Adams discusses the four “subpopulation” datasets
used for the training of the algorithm — Caucasian, Hispanic, African Ameri-
can, and Asian — and he emphasizes the contingent effects of the “removal of
a locus” from these subpopulations on how the algorithm learns to generate
a likelihood ratio. In sum, the root of harm is not the authorial source code
but predominantly the way the algorithm is written and how it writes itself —
continuously editing, adjusting, removing, and iterating in relation to a cor- 97
pus of data. The attributes of the dna of a “general population” and an “un-
Bifurcated Paths
In an essay on the practices of science and literature, John Fowles reflects on
his experience of what happens when he writes. “An important term — at any
98 rate for me, in my own practice,” writes Fowles, “is the fork (as in a path), by
which I mean a fairly continuous awareness of alternatives, both learned (re-
Chapter Three
notion that one could locate responsibility for the harms of an algorithm in
its source code. The algorithm iterates beyond the moment of its inscription,
distributing the writing through multiple characters, from the training data
to the back propagation of errors. This matters because any meaningful re-
sponse to the actions of algorithms, as we saw in relation to genotyping neural
networks, would need to relinquish the search for the source code and focus
instead on the iterative writing back and forth between dna data on the popu-
lation and the making of a likelihood ratio. What would be sought in such a
strategy would be the awareness of alternative paths, as Fowles suggests. There
are multiple learned and “in the wild” features at work in the writing of algo-
rithms, from the many fragments of past data lodged in the criminal justice
databases to the feedback loops that adjust and modify the algorithm into the
future.
In the fifth of Hilary Mantel’s series of bbc Reith Lectures in 2017, the novelist
expresses the relationship between telling history and narrating a story. “We
need fiction to show us that the unknown and unknowable is real and exerts
its force,” she argues, dispelling the notion that the facts of history are distinct
from the fictions of her narratives of Tudor England in the novels Wolf Hall
102 and Bring up the Bodies.56 For Mantel, the writing of fiction confronts the real-
ity of the profound unknowability of our world. The unknown nature of our
Chapter Three
actions, relationships, and intentions is the truth contained within the novel.
As I have discussed through the essays of John Fowles, writing is distinct from
authorship precisely because it must confront the force of the unknown, and
not least the unknowable I that is the writing subject. I have sought not to
diametrically oppose the writing of the novel and the writing of algorithms,
but rather to distinguish the author from the practice of writing so that the
sovereignty of the source code is opened up to the writing, of which it is one
element. This writing — whether literature or code — necessarily and irredeem-
ably confronts the force of the unknown and the unknowable, as Mantel de-
scribes it. What is at stake here is a rather fundamentally different response to
the problem of the algorithm’s accountability. Where the demand for ethical
frameworks for code seeks to make the algorithm knowable and governable,
my call for attention to the writing of algorithms demands a different mode
of account.
To call for the algorithm to give an account of itself is not to demand dis-
closure or transparency, but precisely to express something of the unknow-
ability of the algorithm and the future that it brings into being. In this sense,
like novels, algorithms can be understood as fabulations, where the production
of an output involves a process of working on a corpus or body of material.57
“The fabulatory function,” as Gilles Deleuze proposes, “strips us of the power
to say ‘I.’ ” To write is not to anchor a narrative in a determinate first-person
source, but precisely to work with the incomplete and indeterminate traits
of characters and their connections. Writing is fabulation because it projects
into the future with “a life of its own” beyond the author’s schema, a life that is
“stitched together” and “continually growing along the way.”58 Of course, one
could object that the algorithm actually produces completeness and a deter-
minate future via a single output, an automated decision that goes out into the
world of criminal justice, health risk, security, and war. Yet, this is precisely
the paradox of the process of writing, of fabulating a text. The writing of the
algorithm fixes a unity from scattered data elements, at that same moment
fabulating new connections and traits, forging attributes that will attach to
other beings into the future.
The task as I see it, then, is to find ways to amplify the fabulation of the al-
gorithm, to enter the breaches in the writing so that the force of the unknown
can be felt. I understand the writing of the algorithm to substantially exceed
the writing of source code and to extend into the iterative writing, editing,
and rewriting of composites of data, humans, and other algorithms. Writing
in this mode is a “practice of the disparate,” consisting of gathering “hetero- 103
geneous elements” so that traits, sensibilities, and attributes emerge from the
objectivity and the productive function of a bias that is required for the mate-
rial to be worked on. In the writing of deep neural nets for speech recognition,
for example, the algorithms “trained by back propagating error derivatives”
are reported to have “the potential to learn much better models of data.”66 In
this writing of the algorithm, weights and biases modulate the input signal to
produce outputs as “guesses,” where a feedback loop “rewards weights that sup-
port its correct guesses, and punishes weights that lead it to err.”67 The error
derivative, just like the likelihood ratio in genotyping algorithms, is a genera-
tive fabulation that actively writes the world into being.
Finally, the differential traces of a text mean that it always already con-
tains differences from itself. Thus, when patterns or styles are identified as
being defining characteristics of a text — as with the algorithms predicting the
next sentences of dead authors — those characteristics are undone by other
signs carried within the same text. As Simon Critchley notes in his account
of the ethical capacities of deconstruction, “Any text which identifies truth
with presence as logos” will necessarily also enter into “contradiction with it-
self,” signaling “towards a thinking that would be other to logocentrism.”68 For
Critchley, this “hinge” between the closure of the logos and the opening onto
the other is the very articulation of an ethics of deconstruction. Any demand
for an enclosure of the context of writing, as for example in the many calls for
the context of algorithmic systems to be defined and circumscribed, neglects
this sense of the excess of context.69 The idea that one could govern and limit
the contextual application of a specific algorithm, for example, denies the con-
stant iteration, modification, and adjustment of the writing of algorithms. In-
deed, the algorithms used for extracting meaning from text, such as sentiment
analysis, write themselves in and through decontextualized encounters, pre-
cisely by being exposed to something of the excess of context.
Following the writing of the algorithm involves watching for the moments
when it enters into contradiction with itself. Computer science envisages the
truth of the algorithm in presence or logos, where the algorithm adjusts itself
to map signal to meaning correctly. In the writing, however, there are no de-
finitive correlations but only continuous and open-ended iterations of the text,
where every modification authorizes a new meaning and a new output. Indeed,
this is what the idea of a likelihood ratio means in an algorithm used to deter-
mine the probability of the presence of someone’s dna at a crime scene. An
infinitesimal modification in the training data (remembering that this could
be exposure of the learning algorithm to the dna data of racialized subpopu-
lations), for example, will alter the meaning of the likelihood. At root, the 107
output of the algorithm is not secured by the linear enunciation of its code.
There are crises of reason in strange complicity with what the world calls crises
of madness.
— Jacques Derrida, “Cogito and the History of Madness”
Reason demanded this step. In our science we have run up against the limits of
what is knowable. . . . Our science has become a horror, our research dangerous,
our knowledge lethal. All that is left for us physicists is to capitulate before reality.
— Friedrich Dürrenmatt, The Physicists
Errant Algorithms
There are moments when the everyday background hum of algorithms in so-
ciety seems to break the surface and become something louder and more in-
sistent. Such moments are often debated publicly as times when algorithms
have become mad, or when they have departed from their rational logics and
become frenzied, demented, and thoroughly unreasonable. In October 2016,
when the British pound lost 6.1 percent of its value in a matter of seconds,
the currency trades conducted autonomously by algorithms were reported
to have precipitated a “flash crash,” an eruption of irrational actions beyond
the control of human traders.1 Such apparent moments of the madness of al-
gorithms, however, are not limited to the vicissitudes of the financial mar-
kets. In March 2016, Microsoft researchers experimented with a social media
chatbot — named Tay and modeled on the attributes of a teenage girl — training
her to communicate in a humanlike way by exposing her to the inputs of Twit-
ter data streams. Within a few short hours, and in response to her Twitter
learning milieu, Tay was posting racist and sexist messages, denying the holo-
caust and boasting to the police of her prolific drug habit.2 Tay’s deep learning
algorithms were widely reported to be in meltdown, spiraling out of control
in a descent into madness. The Microsoft experiment was finally halted when
Tay messaged her two hundred thousand social media followers with “You are
too fast, please take a rest,” sending the message over and over again in a rep-
etitious frenzy. The public’s sense of dangerous algorithms becoming unrea-
sonable and doing harm seemed to reach fever pitch in 2017, when YouTube’s 109
machine learning recommendation algorithms targeted automated content to
tion from madness. The public discussion of the evils and harms of algorithms
is focused on the regulation of that very threshold thought to separate rational
from irrational action. Crucially, though, this threshold is, as all border lines
are, the line that both pulls together and demarcates madness from all that is
reasonable in the world. For Jacques Derrida, the division is a “dissension” or a
“self dividing action,” which both “links and separates reason and madness.”7
Discussing Foucault’s treatment of the history of madness, Derrida cautions
that “one cannot speak of madness except in relation to ‘that other form of
madness,’ that is except in relation to reason.”8
Taking seriously the conjoined histories of the ideas of reason and mad-
ness, I propose that one cannot speak of the madness of the algorithm except
in relation to the form of reason the algorithm embodies. While the contem-
porary moral panic at each moment of the madness of algorithms urges us to
police ever more vigilantly the line between reasonable and unreasonable ac-
tions, understood as a dissension, this line is precisely the condition of possi-
bility of algorithmic rationality. Algorithms cannot be controlled via a limit
point at the threshold of madness because the essence of their logic is to gen-
erate that threshold, to adapt and to modulate it over time. In short, my argu-
ment is that when algorithms appear to cross a threshold into madness, they
in fact exhibit significant qualities of their form of rationality. Understood in
this way, the appearance of a moment of madness is a valuable instant for ethi-
copolitics because this is a moment when algorithms give accounts of them-
selves. Contra the notion that transparency and the opening of the technologi-
cal black box secures the good behavior of algorithms, the opacity and clouded
action exhibited in the excesses and frenzies of algorithms yield a different
kind of fidelity to the logic.
Let me make this concept of clouded action a little more concrete. Though
the surfacing of violent images of beloved cartoon characters for preschool
children appears as an undoubted algorithmic aberration, the computer sci-
ence accounts of the development of deep neural networks for the YouTube
video recommendation system exhibit a rationality that is consistent with out-
putting violent content. The algorithm designers describe the “dynamic cor-
pus” of YouTube video content and explain their improved deep learning algo-
rithms, which are “responsive enough to model newly uploaded content as well
as the latest actions taken by the user.”9 Understood as a model that is trained
on one billion parameters, the apparent madness of the recommendation of
violent cartoons is explicitly part of, and not an aberration from, the rational-
ity of the deep neural network. Indeed, the optimization of these algorithms is 111
achieved in part through exposure to the new and emerging uploaded content,
and take responsibility for the impossibility of ever binding the action to a full
body of knowledge. Decisions are mad because they can never know fully the
consequences and effects of their own making. To decide is to confront the im-
possibility of the resolution of difficulty; it is madness in the specific sense that
it has no unified grounds.11 With contemporary algorithms, decisions are being
made at the limit of what could be known, and yet there is no responsibility
for the unknown consequences of the decision. The madness of the decision is
disavowed by the single output of the algorithm, and this disavowal is a poten-
tial horror and a danger. In the second half of the chapter, I discuss one specific
set of algorithms — random decision forests — that have become the algorithms
of choice across many domains, particularly in national security and border
controls. The random decision forest algorithm grows multiple decision trees
on random subsets of data and “takes a vote on their predictions.”12 It is an al-
gorithmic proposition of What comes next? that takes place as a calculation
amid incalculability, mobilizing chance and the splitting of agency, sometimes
with lethal effects on human life.
In a 1950 essay, “Atomic Knowledge of Good and Evil,” the cybernetician Nor-
bert Wiener expresses his concern for the moral responsibility of science in
the face of what he calls the “dangerous possibilities” of atomic weapons.13
Reflecting on the nature of the roles of the mathematicians and physicists at
Los Alamos, Wiener is anxious that “the new centralized atmosphere of the
age to come is likely to lead society in the direction of ‘Big Brother’ ” and to-
ward a “future fascism.” A full seventy years before scientists pointed to the
dangers of a descent into fascism precipitated by the mathematical sciences of
algorithms, the cyberneticians were writing publicly in newspapers and jour-
nals to express the view that “when the scientist becomes the arbiter of life and
death” the locus of “moral responsibility” belongs properly to the reasoning
scientist to prevent “the decay and disruption of society” and to ensure that
science “is used for the best interests of humanity.”14 Indeed, in a 1947 letter
to Atlantic Monthly, Wiener replies publicly to a request from government mili-
tary officials for copies of his mathematics papers. Concerned that withdraw-
ing his science from the service of war might be “shutting the door after the
horse has become classified,” Wiener believes that the locus of control for a
dangerous science must reside in the reasoning human scientist. For Wiener,
the rise of computing machines posed a threat defined not so much in terms
of the madness of the autonomous machine as in terms of what he called the 113
“crazed and misguided” irrationality of humans. “The automatic machine is
tractable state decisions makes them interior to defining what normalities and
pathologies could be in a society. Just as the early neural network algorithms
were embraced as a means of instilling reason into otherwise potentially dan-
gerous human judgment, however, they also brought into being a new mode of
harnessing the unknown and the unpredictable.
Second, at the same time as algorithms were thought to tame human irra-
tionality, a specific orientation to madness was actively incorporated into the
logics of algorithms. In Orit Halpern’s book Beautiful Data, she gives a devas-
tating account of how ideas about the psychotic and the neurotic were inte-
gral to cybernetic logics. “What has been erased from the historical record in
our present,” writes Halpern, “is the explicit recognition in the aftermath of
World War II and the start of the Cold War that rationality was not reason-
able.”19 In a detailed engagement with Warren McCulloch and Walter Pitts’s
development of the neural net, Halpern proposes that the logic of psychosis
was crucial to the reformulation of computational rationality. Tracing the
genealogy of neural nets as technical instantiations of the brain’s functions,
Halpern demonstrates the extent to which “neurotic, psychotic, and schizoid
circuits proliferated in the diagrams of cybernetics.”20
The insights of those who have studied the schizoid and paranoid relations
of computation are critical to the contemporary ethicopolitical relations being
forged with and through algorithms.21 Far from representing a departure from
rationality and entry into psychotic turmoil, the actions of algorithms are
never far from their conjoined histories with psychosis, neurosis, trauma, and
the imagination of the brain as a system. The double enfolding of unreason
into cybernetic thought — as the unreasonableness of human decision makers
in conditions of war and insecurity, and as models of psychosis underpinning
the segmented cognition of the neural net — serves as something of a correc-
tive to our contemporary societal debates on the responsibility of science for
moments of algorithmic madness. Put simply, the history of algorithmic ratio-
nality is not separable from genealogies of madness. It is time to think difficult
thoughts: algorithms are always already unreasonable. When one hears calls
for new controls to limit the potentially dangerous actions of the algorithm, it
is perhaps worth recalling that algorithms historically are limit devices. That
is to say, they actively generate new forms of what it means to be normal or
abnormal, just as they mark new boundaries of rationality and unreason. To
capitalize on uncertainty — whether in warfare or in the commercial world of
risk — algorithms dwell productively with emergent phenomena and incalcula-
ble surprises.22 The actions of the contemporary neural net, for example, finds 115
its condition of possibility in the twinned imperatives of limiting the impulses
Throughout this book I urge some caution about the treatment of the ethical
and political questions posed by algorithms as novel or unprecedented. Indeed,
I suggest that the advent of algorithms as decision makers in our societies has
restated what are in fact perennial philosophical debates about the nature of
ethical and political life. Among these perennial debates is the question of
whether and how one controls the impulses or actions of an entity that appears
to us as beyond human reason. It is in the philosophies of human madness and
its relationship to forms of reason, I propose, that one can locate a resource
for thinking differently about the actions of algorithms. In Michel Foucault’s
major genealogical work on the history of madness, he argues that madness is
designated as a “blatant aberration” against all that would otherwise be “rea-
sonable, ordered and morally wise.”23 Understood in this way, the very idea of
human reason has required — integral to its historical condition of possibility
within a moral order — an opposition to the aberration of madness. Indeed, for
Foucault, “in our culture there can be no reason without madness,” and yet
the objective science that observes, records, and treats pathologies of madness
simultaneously “reduces it and disarms it by lending it the slender status of
pathological accident.”24 Thus, there is a necessity of madness within Western
thought, and yet this is a diminished form of madness reduced to a pathologi-
cal accident or error. The idea of unreason is always already present in claims
to moral reason, but this is an unreason understood as a flaw or error in the
logic of reason — a flaw that can be identified, classified, and cured.
Such a sense of a reduced form of madness within Western moral order
seems to me to be amplified in our contemporary moment, when the actions
of algorithms are so frequently named as accidents or errors. The threshold of
116 madness denotes a boundary between the reasonable and the unreasonable,
just as it also condenses the unknowability of madness to an error, an accident,
Chapter Four
From this point onwards, madness was something other than an object
to be feared. . . . It became an object. But one with a quite singular status.
In the very movement that objectified it, it became the first objectifying
form, and the means by which man could have an objective hold on him-
self. In earlier times, madness had signified a vertiginous moment of daz-
117
zlement, the instant in which, being too bright, a light began to darken.
Now that it was a thing exposed to knowledge . . . it operated as a great
tive happens: when there’s a choice between a two-person car and you alone
in your self-driving car, the result would be to run you off the road.”40 The re-
search imagines a world in which algorithms can be trained to precompute the
rational decision in the face of a scenario of catastrophic consequences.
The boundary of this rationality is a computational threshold in which
the algorithms weigh the values of different choices before making a deci-
sion. The research team is working on deep neural networks that will “sort
through thousands of possible scenarios,” filtering out and “rapidly discard-
ing 99.9% of them to arrive at a solution.” The example the scientists discuss
is a self-driving car hurtling toward a school bus, where the optimal pathway
is thought to be “discarding all options that would harm its own passenger”
before “sorting through the remaining options to find one that causes least
harm to the school bus and its occupants.”41 Here, in the starkest of terms, is
a statement of the kind of ethical codes sought by many engineers and algo-
rithm designers.42 It is an ethical mode that promises to render calculable the
profoundly uncertain horizon of an immediate future. Put simply, in seeking
to ward off the full horrors of frenzied algorithms governing a trolley out of
control, such programs generate new harms in the valuing and weighting
of different pathways. The calculus itself — with its combinatorial possibilities
of babies in strollers, elderly pedestrians crossing the road, careless delivery
drivers, and inattentive cyclists — is always already thoroughly unreasonable.
Algorithms do not pose their most serious and harmful threats when they are
out of control, become crazed, or depart rationality. On the contrary, among
the most significant harms of contemporary decision-making algorithms is
that they deny and disavow the madness that haunts all decisions. To be re-
sponsible, a decision must be made in recognition that its full effects and con-
sequences cannot be known in advance. A responsible decision-making pro-
cess could never simply “sort through the remaining options to find one that
causes the least harm” because this is an economy of harms that renders all the
incalculability of harm calculable.
The madness of the trolley decision is that it must necessarily be made
in the darkness of nonknowledge, that it categorically is not subject to pre-
programming to optimize an outcome. As Derrida writes, “Saying that a re-
sponsible decision must be taken on the basis of knowledge seems to define
the condition of possibility of responsibility,” and yet at the same time, “if
decision-making is relegated to a knowledge that it is content to follow, then it
is no more a responsible decision, it is the technical deployment of a cognitive
apparatus, the simple mechanistic deployment of a theorem.”43 Though the al- 121
gorithm’s design for the trolley problem may appear to be the simple mechanis-
forest, or random decision forest, has become the algorithm of choice for systems
designed to identify terror targets for the state.51 Why might this be? Because
machine learning for counter-terrorism is a difficult problem. Unlike the com-
mercial algorithms for credit card fraud or email spam detection, for example,
where there is vast availability of labeled training data on which to train the
classifier algorithms, the availability of labeled data on known terrorists is ex-
tremely scarce, indeed even thoroughly inadequate for statistical modeling.
This has meant that terrorist-targeting algorithms have become, in a rather di-
rect sense, unreasonable. That is to say, in the context of profoundly uncertain
security futures, these algorithms have harnessed the unknown and the incal-
culable to preserve the capacity to act. I am going to dwell a little longer on the
random forest algorithm as an arrangement of propositions in the world. The
randomness of this algorithm is, I suggest, a kind of sheltered madness that
dwells inside the logic of the algorithm and promises to the world an impos-
sible vision of a secure future. It is, in short, an algorithm with life and death
effects, and it generates these effects by reformulating the dissension between
reason and madness.
In October 2016, two journalists reported that the nsa’s skynet pro-
gram “may be killing thousands of innocent people.”52 In the context of the
estimated four thousand people killed by drone strikes in Pakistan between
2005 and 2016, Christian Grothoff and Jens Porup investigated how skynet
“collects metadata,” storing it “on nsa cloud servers” and then applying “ma-
chine learning to identify leads for a targeted campaign.” Among the “cloud
analytic building blocks” described within skynet in the Snowden files are
travel patterns; behavior-based analytics, such as incoming calls to cell phones;
and other “enrichments,” such as “travel on particular days of the week,” “co-
travelers,” and “common contacts” (figure 4.1).53
I have elsewhere written in detail on how these aggregated security “data
derivatives” are generated and how they are made actionable.54 Of specific in-
terest here are the random forest decision tree algorithms that are learning in
125
communion with the data derivatives. Random forest algorithms were devel-
oped at the turn of the millennium by Leo Breiman of uc Berkeley’s Depart-
ment of Statistics.55 These machine learning algorithms exemplify well how
notions of chance, preference, error, and bias become incorporated into the ca-
pacity to compute an output. As Breiman explains, in the random forest tech-
nique, the algorithms come into being by “growing an ensemble of [decision]
trees and letting them vote for the most popular class.”56 Noting that “data sets
with many weak inputs are becoming more common” (for example, in medical
diagnostics) and that “this type of data is difficult for the usual classifiers —
neural nets and trees”—Breiman designed his algorithms to “inject the right
amount of randomness” so that the algorithm is trained on random subsets of
the training data and “predicts the class of each new record using the plurality
of the class predictions from the set of trees.”57 In this way, the random forest
algorithm actively uses variance and randomness to refine its output. As Lor-
raine Daston has argued in her compelling account of the histories of prob-
ability, chance, and unreason, all three were able to flourish within methods
of calculation.58 Contemporary algorithms such as random forests have found
new ways to invite randomness and chance back into the probabilistic calculus.
If the random forest algorithm appears to descend into the horror of tar-
geting civilians for drone strikes in the skynet program (and I have also seen
it used by governments to output a risk score of immigration infraction), how
can one understand the mode of unreason that dwells within the form of al-
gorithmic rationality? A key element of the random forest algorithm’s logic
is the attributes-based targeting of population I discuss in previous chapters.
The skynet random forests are trained on data from a population described
126 as “known couriers.” These proxies for labeled data on terrorists are individu-
als within Pakistan who have been scraped from intelligence reports under
Chapter Four
Ethics
This page intentionally left blank
5
The Doubtful Algorithm
Ground Truth and Partial Accounts
During the war I worked on the atomic bomb. This result of science was
obviously a very serious matter, it represented the destruction of people. . . . Is
there some evil involved in science? Put another way — what is the value of the
science I had dedicated myself to — the thing I loved — when I saw what terrible
things it could do?
— Richard Feynman, What Do You Care What Other People Think?
Humanism” of the “turbulent years after the Second World War,” when the
profound tyranny of human rationality was felt.3 Yet, as Donna Haraway ac-
knowledges of her own subjectivity as “a child of the Scientific revolution, the
Enlightenment, and techno-science,” what it means to be human is not separa-
ble from technoscience and its place in war.4 Though Feynman’s situated and
partial accounts of the science of the atomic bomb bear witness to the state’s
enrollment of algorithmic calculative practices in war, these are not accounts
that stand outside science in opposition. Rather, the partial account dwells un-
easily within the interconnected narratives, stories, writings, and narratives
that together constitute the practice of science.
What is striking to me, in the letters and diaries of twentieth-century
mathematicians and physicists, is that the intransigent and political difficulties
of partial accounts — manifest in the public debates of the twentieth century —
are all but entirely absent in the twenty-fi rst century’s enrollment of algo-
rithms by the state. In the demand for transparency and clear-sighted account-
ability, the algorithm’s orientation to doubt is never in question. Amid the
pervasive twenty-fi rst-century political desire to incorporate all doubts into
calculation, algorithms are functioning today with the grain of doubt, allow-
ing the uncertainties and variabilities of data to become the condition of pos-
sibility of learning and making decisions. In contrast to Feynman’s notion that
science’s responsibility to society may reside in leaving open the incalculabil-
ity of the future, algorithms hold out the promise of securing against all pos-
sible future events. With contemporary machine learning algorithms, doubt
becomes transformed into a malleable arrangement of weighted probabilities.
Though this arrangement of probabilities contains within it a multiplicity of
doubts in the model, the algorithm nonetheless condenses this multiplicity to
a single output. A decision is placed beyond doubt.
As doubt becomes the terrain of algorithmic calculation, however, other
situated and embodied forms of doubt haunt the terrain. These opaque and
cloudy accounts seem present at all the moments when an algorithm gener-
ates itself in relation to a world of data. For example, when a biometric match-
ing algorithm doubts it can recognize a face because the data points were not
present in the training data, or when four desk analysts watching luminous
screens of risk scores worry that the model is “overfitting” to the data — these
are moments not of a lack or an error, but of a teeming plenitude of doubtful-
ness. What might it mean to invoke Feynman’s “room for doubt” or “to live
and not to know” in the context of a technoscience driven to expose machine
learning algorithms to data precisely to know and to act, indifferent to persis- 135
tent doubt?5 In light of the algorithm’s major role in the calculability of doubts,
Ground Truths
Doubtful Science
The value of a life insurance customer (the risk-based price of their in-
surance) is thus generated from the output of the neural network, calibrated
against a threshold of value to optimize the insurance company’s position.
Where the data points on past known entities are scarce — as with the “cou-
rier” proxies for terrorists I discuss in chapter 4 — the algorithm perceives what
others are doing and establishes a ground truth from which attributes of risk
or opportunity can be derived.17 The life insurance algorithm emerges through
the partial and oblique correlations among otherwise unrelated selves and oth-
ers. It will go out into the world, continue to iterate and learn, ingest new data
points, create new clusters, and generate new outputs for decisions. When the
algorithm designers refine and adapt their algorithm for new domains, they
are experimenting with the proximity between a specified target value and
the actual output signals from their model, observing how the numeric scores
generated by their model diverge or converge on the target. The computations
of algorithms like these are doubtful in their openness to new interactions
and encounters in the world, every layer infinitely malleable and contingent
on plural interactions of humans and algorithms. A minor change in the sta- 139
tistical state of an entity — the gradient of a high-value life, or the features of
I turn here to a historical moment when the output of a risk algorithm failed
catastrophically. This is a moment when a Cartesian doubt was expressed that
inaccurate data may have led to a catastrophic decision. When the event is un-
derstood as composed of opaque and partial fragments, however, what comes
to matter is not whether something could be calculated accurately from data,
but rather how partial probabilities became a singular calculus for a fatal deci-
sion. On January 28, 1986, seventy-three seconds after its launch at 11:25 am,
the nasa space shuttle Challenger broke apart over the Atlantic Ocean, kill-
ing all seven crew members, including Christa McAuliffe, nasa’s “teacher
in space.” By February, President Ronald Reagan had established the Rogers
Commission to “review the circumstances surrounding the accident and es-
tablish the probable cause or causes” and to “develop recommendations for
corrective or other action.”19
Among the fourteen commissioners appointed to the Challenger investi-
gation was physicist Richard Feynman, who was persuaded to participate de-
spite, by his own account, “having a principle of going nowhere near govern-
ment” after his experience of working on the atomic program at Los Alamos.20
More specifically, he was wary of bureaucratic reason and the governing of
what he thought of as an unruly science by bureaucratic rules and protocols.
The Cold War cybernetic rise of “algorithmic rules that could be executed
by any computer, human or otherwise” with “no authority to deviate from
them” had extended a particular kind of “arithmetic into the realm of deci-
sion,” and this entanglement of mathematics and decision did not please Feyn-
man.21 Feynman’s letters and diaries reveal a disdain for algorithmic decision
procedures and axiomatic formulas, and a propensity to ask questions that
ran against the grain of mathematical rules. “Doing it by algebra was a set of
rules,” writes Feynman of his early encounters with mathematics, “which, if
you followed them blindly, could produce the answer.”22 For Feynman, algebra
was a means of imposing an axiomatic set of procedures of rules on a puzzle
that could otherwise intuitively work toward an unknown outcome.
Confronted with the Rogers Commission’s setting of procedural steps in
the investigation of the Challenger disaster, Feynman worked to reinstate the
doubtfulness at each link in nasa’s chain of events. He traveled to meet with
the engineers, avionics scientists, and physicists whose data on particular
components had made up the aggregate risk calculation on which nasa had
based their launch decision. What he discovered was not a catastrophic depar-
ture from the normal rules or a “human error” or failure as such. Instead, the
launch decision belonged properly to a posthuman composite of algorithms, 141
where the steps of a normalized risk-calculation protocol had been followed
lenger disaster, the seals of the solid fuel rocket boosters of the shuttle were
secured by pairs of rubber O-rings. It was known to nasa that, during some
launches, the hot gases from the boosters could leak from the seals, causing
what was called blowby over the liquid oxygen and nitrogen of the orbiter’s en-
gines. Though the engineers observed corrosion of the O-rings over time, they
assumed that the heating of the rubber O-rings during launch caused small ex-
pansions to close the gaps in the seal. At the fatal launch, the temperature was
28 degrees Fahrenheit (−2 Celsius); the coldest launch recorded prior to this
had been 53 degrees Fahrenheit. On the evening of January 27, the night before
the launch, the engineers at Thiokol, which manufactured the seals, warned
nasa that the launch should not take place if the temperature was below 53
degrees. Yet, the public record shows that nasa “took a management decision”
because “the evidence was incomplete” that “blowby and erosion had been
documented above 53 degrees,” so “temperature data should be excluded in
the decision.”26 As Feynman writes, nasa testified that “the analysis of exist-
ing data indicated that it [was] safe to continue flying.”27 In effect, the aggre-
gate data of twenty-four past flights without a mission failure had placed the
launch decision beyond the doubt that would otherwise have been reinstated
by attentiveness to the components, and to the inscribed traces they embodied.
A sense of doubtfulness that might be considered fully posthuman — dwelling
in the material marks of blowby and erosion, and in the touch of engineers on
fissures and cracks — was aggregated out in the calculation of a singular output.
In historical instances such as Feynman’s account of Challenger, we can see
some of the fallacies of an objective risk-based or data-driven decision placed
beyond doubt. Remarking on the computer model used by nasa in support of
their launch decision, Feynman writes, “It was a computer model with various
assumptions. You know the danger of computers, it’s called gigo: garbage in,
garbage out.”28 To question the assumptions of the computer model, Feynman
began to focus on the fragments of data and their associated probabilities, re-
instating doubt within each element. Understood in this way, the probabilis-
tic calculation of risk gives way to what Donna Haraway has called “partial,
locatable knowledges,” in which there is the “possibility of new webs of con-
nections.”29 nasa had testified to the commission that the probability of fail-
ure of a mission was calculated to be 1 in 100,000. “Did you say 1 in 100,000?,”
queried Feynman. “That means you could fly the shuttle every day for 300
years between accidents?”30 To open up a breach in the risk calculus, Feynman
gathered the scientists who had worked on the various different components of
Challenger, asking them to estimate the probability of failure of the shuttle and 143
to write this probability on a slip of paper. When he collected the fragments
Figure 5.1 Richard Feynman removes the O-ring material from ice water at the
Rogers Commission inquiry, February 11, 1986. “Richard Feynman (Articulate
Scientist) — Space Shuttle Challenger Testimony,” https://www.youtube.com
/watch?v=MWZs8l2AMps, posted October 3, 2013.
the O-ring giving an account of itself is that Feynman follows the “structure
of heroic action” in science, as Donna Haraway has put it, speaking objectively
and “self-invisibly” through the “clarity and purity of objects.”35 Certainly the
now famous New York Times report of the O-ring event, citing Feynman’s few
words, “There is no resilience in this material when it is at a temperature of
32 degrees,” bears a form of witness close to that of Haraway’s critique. I lo-
cate a different form of account here, however, one with no unified authorial
source or transparent picture of the truth, but rather a distributed and oblique
account in which we see the political impossibility of resolving the truth be-
fore the public. What is dramatized with the rubber O-ring in the ice water
is a set of relations that are material and political. Refusing the mode of ac-
countability in favor of this different form of giving an account, there is what
Karen Barad calls a “condensation of responsibility” in matter, wherein the
multiple contingent past decisions are lodged within the object, engaging us
in “a felt sense of causality.”36 In this way, a material limit that was ordinary
in the embodied experiences of engineers and materials scientists — familiar to
their touch — could be expressed as a claim that could be heard in the world.
Feynman’s claim did not express the voice of Haraway’s “victor with the stron-
gest story”; indeed his account was derided and rejected by the commission
throughout.37 At the specific moment I describe, Feynman speaks against the
grain of Cartesian doubt and instantiates a mode of doubtfulness more com-
monly feminized and annexed within subjectivity. It is precisely this mode of
intuitive causality and embodied doubtfulness that I am seeking as a resistant 145
and critical form of responsibility.
drone program in multiple other Google object recognition systems, the protest
delineated good from evil, and war from commerce, in ways that did not place
the subjects and objects of science at risk. Within a cloud ethics, risky speech
permits loosening the hold on subjectivity and relinquishing the grounds from
which the subject speaks. When the “avowing subject,” as Foucault writes,
“loosens its hold,” it becomes possible to envisage a risky speech that forges new
ethicopolitical relations with oneself and others.42 This means that one could
be doubtful of all claims, for example, that the bias or the violence could be ex-
cised from the algorithm and begin instead from the intractable political rela-
tions between the algorithm and the data from which it learns.
Ultimately, on discovering that the Rogers Commission report contained
a previously undisclosed “tenth recommendation,” sheltering the risk algo-
rithm within “error” and commending nasa’s work and mission to the na-
tion, Richard Feynman refused to add his signature to the report. His fellow
commissioners sought to persuade him, suggesting that the tenth recommen-
dation was a feminized “only motherhood and apple pie,” and that “we must
say something for the president.”43 The tenth recommendation, which would
authorize a particular decided future of continued nasa missions, was demar-
cated from the real science of objective findings and rendered maternal and
nation building.
In response, Feynman asked what kind of truth could be spoken to the
government: “Why can I not tell the truth about my science to the President?”
In the event, his own report appeared as an appendix to the published docu-
ment, titled “personal observations on the reliability of the shuttle,” as though
his merely personal situated account could only be subjective and without
grounds. Feynman’s appendix is risky speech in Foucault’s sense, an argument
that places the subject at risk, runs against the grain of the commission report,
and challenges the form of truth telling of the public report. In this sense, to be
doubtful is to open onto the full contingency of a situation and to be responsive
to unspecified future effects. The fullness of an embodied posthuman doubt
I envisage here is a critical cousin to Lauren Berlant’s notion of a “cruel opti-
mism,” where a “cluster of promises could seem embedded in a person, a thing,
an institution, a text, a norm, a bunch of cells, smells, a good idea.” The cru-
elty Berlant so vividly describes is a “relation of attachment to compromised
conditions of possibility whose realization is discovered to be impossible, sheer
fantasy, or too possible, and toxic.”44 Though Berlant does not probe directly
what it might mean politically to be doubtful of the promises as they embed 147
in a person, a thing, or a technology, her work does envisage the making of
with the various ways in which embodied doubtfulness in science does not give
political accounts of itself. Similarly, when the former US director of national
intelligence James Clapper was found to have lied to Congress on the analysis
of data on US citizens in March 2013, he later argued that he had provided the
“least untruthful answer possible in a public hearing.”49 To speak against the
grain of the promise algorithms make in the name of political resolution is to
bring to the public assembly a kind of contemporary heresy. Following the UK
terror attacks in Manchester and London in 2017, for example, where some of
the perpetrators had previously been known to intelligence agencies, the offi-
cial inquiry by David Anderson has concluded that what is required is a more
effective deep machine learning algorithm for the identification of high-risk
individuals among twenty thousand so-called closed subjects of interest.50 In
effect, this implies training an algorithm to recognize the attributes of known
individuals in the dataset to identify similar attributes among unknown fu-
ture subjects. The weighted probabilities among the twenty thousand will be
used to generate target outputs for future security action. To signal the doubt-
fulness of such an algorithm is precisely to engage in risky speech and annul
the promise of a securable future.
The form of doubtfulness I have described would demand that algorithms
give political accounts that are marked by contingency and ungroundedness.
In place of an ethics that seeks defined grounds for algorithmic decisions, a
cloud ethics begins from the ungroundedness of all forms of decision, all polit-
ical claims, human and algorithmic. As Thomas Keenan writes, “What we call
ethics and politics only come into being or have any force and meaning thanks
to their ungroundedness,” so that “we have politics” precisely because we have
no “reliable standpoints.” If a future political claim is to be possible, then “its
difficulty and its persistence stem from its terrifying, challenging, removal of
guarantees.”51 If there are to be future claims to human rights not yet registered
or heard, for example, then there must always also be doubts haunting the pos-
sibility of rights, doubts that acknowledge the irresolvability of all political
149
At one level the algorithms of the threat detection centaurs appear to run
counter to the form of risk calculation observed by Feynman at nasa. The
smallest of infractions or the subtlest of signals of a component failure or a
change in online clickstream patterns, for example, is said to yield a threat
alert to the centaurs. Like the Rolls Royce jet engines that yield real-time data
on the performance of aircraft as they fly, the twenty-fi rst-century O-ring
would almost certainly be given a data voice. And yet, at the level of practices
of truth telling, there is something of significance here. The Challenger launch
decision teemed with a cacophony of embodied doubt — on the part of the en-
gineers, physicists, and mathematical modelers, in communion with their
nonhuman partners — a nd yet, the decision was placed beyond doubt. What
we are seeing emerge in contemporary algorithmic technoscience is an orien-
tation to truth telling that says that all political difficulty can be resolved, all
decisions can be placed beyond doubt.
How might one intervene in models such as the threat centaur in ways
that reinstate the doubtfulness of the algorithm? It is necessary to engage the
traces of machine learning algorithms, to be curious about them, to doubt
them, but also to listen to the doubt they themselves express in the world. In
engaging research methods that follow elements of an algorithmic solution as
they travel and have onward life, I am trying to be a receptive listener when
the piece of data appears to say “you do know that I cannot really be cleaned; I
remain in there, muddying the calculation”; or when the biometric-matching
algorithm says “in 14 percent of cases, where ambient lighting exceeds these
parameters, I cannot be read”; or when the pattern recognition algorithm says
“the training dataset I have been exposed to has established what looks normal
or abnormal to me.” There is doubt, and it proliferates everywhere; at every
step in the application of algorithmic solutions to political questions, doubt
lives and thrives and multiplies. And, where there is an ineradicable doubt,
there is also the darkened frontier and, therefore, the potential for politics. In-
spired by Richard Feynman’s method of reinstating doubt in the data points,
and being an interlocutor for the people and things “in the engine” who ac-
tively give accounts of limit points of all kinds, I urge that we experiment as
the algorithms do, iteratively and recursively, giving doubtful accounts of the
calculation.
A description must capture not only the objects contained in an image, but it
must also express how these objects relate to each other as well as their attri-
butes and the activities they are involved in.
— Oriol Vinyals et al., “Show and Tell: A Neural Image Caption Generator,”
my emphasis
The Unattributable
of algorithms — r nns designed for natural language processing — then act on
the output of the first to “decode those representations in natural language
sentences.”6 Working together in collaboration, the algorithms are said to gen-
erate outputs deciding “not only the objects contained in an image,” but also
how they “relate to each other,” what their essences or “attributes” are, and
what probable “activities” they may be “involved in.”7 The training images of
the algorithms — supplying the features of the recognizable — thus display a cu-
rious and compelling politics of showing and telling for our times.
Let us reflect on how these algorithms are condensing the question of
what matters, “inferring the scene,” as the computer scientists describe it,
and “mimicking the remarkable human ability” to filter out “the clutter” and
attend to the thing of interest.8 In figure 6.1, a group of people gathers in a
marketplace, their relations uncertain and indeterminate. One cannot pos-
sibly know from the image how these bodies relate one to another, what their
qualities or attributes might be, and especially why they are there. Indeed,
perhaps the incalculability and unknowability of their relations is the defin-
ing character of the urban scene, the crowd, and the marketplace. And yet,
from the multiplicity of relations in the scene — the people, the vegetables, the
earth, the clouded backdrop of market stalls — the neural nets cluster the data
and define the attributes necessary to generate a single condensed output of
meaning: “A group of people shopping at an outdoor market. There are many
vegetables at the fruit stand.” The algorithms generate the field of meaning;
they are the mise-en-scène of the event; they decide what is to be in the sin-
gle output signal. Understood in this way, there is indeed nothing outside the
text in the neural net, the context supplying the feature space from which all
possible future attention can be extracted, all meaning derived. Simultane-
ously, though, as the algorithms mistakenly infer the “vegetables” at the “fruit
stand” from all the past data elements of images and video of markets they
have been trained on, the “outside penetrates and determines the inside,” so
that the borders of context entail also an opening, or nonclosure.9 As the al-
Figure 6.1 The show and tell model for scene understanding, comprising
two algorithms: a convolutional neural network for image recognition and a
recursive neural network for natural language generation. LeCun, Bengio, and
Hinton, “Deep Learning,” 2048.
gorithm travels from its laboratory site of test images of South Asian markets,
how might the presence of a particular object in the scene — a turban, some
plantain, a pile of baskets — imply the attributes for future instances in Middle
Eastern marketplaces? If the Google computer scientists’ show and tell algo-
rithms are animating the scene understanding of the US drone program, what
action is authorized based on attributes?10 When algorithms learn the “latent
alignments,” what happens to that which is considered to be dormant or lying
hidden within a scene? What is a normal alignment in the marketplace, the
border landscape, or the city plaza?
The techniques deployed within the neural net algorithms to condense
the features of a scene to a single output — “a woman is throwing a Frisbee in
a park,” “a stop sign is on a road with a mountain in the background” — give
a compelling account of the ethicopolitics of algorithms for our times (figure
6.2). The output of the algorithms reduces the intractable difficulties and du-
ress of living, the undecidability of what could be happening in a scene, into
a single human-readable and actionable meaning. We have ethical and politi-
cal relationships with other beings in the world because the meaning of those
relations, their mediation through every scene of life, cannot be condensed.
They are precisely irreducible. And so, at the moment that the algorithm out-
puts a single meaning from an irreducible scene, at this border limit is also a
“clause of nonclosure,” as Derrida describes the opening of context.11 The ties
that bind words, concepts, images, and things are only ever relatively stable, so
that there is always also an opening. How does one begin to locate the points
of nonclosure within the algorithm’s program of meaning making? If a cloud
ethics is to be concerned with the political formation of relations to oneself
and to others, how might it locate alternative critical relations of showing and
telling? Are there counter-methods of attention available to us that could re-
sist the frameworks of attention of machine learning? Amid the technologies
of the attribute, what remains of that which is unattributable in the scene?
In this concluding chapter, I am concerned with how a cloud ethics could 157
be brought into being in a world that is increasingly shaped through the ar-
The Unattributable
rangements of the machine learning algorithm. Though I do not propose a
definitive method for critique or for resistance to the prevailing powerful log-
ics of surfacing, targeting, or “show and tell” in machine learning, I do urge
Figure 6.2 The show, attend, and tell model for scene understanding, display-
ing the test image on the left of each pair, with the machine’s “attended” vision
on the right. The target output is underlined. Xu et al., “Show, Attend and Tell,” 2.
that it is possible to confront the specific fabulatory functions of today’s algo-
rithms with a kind of fabulation of our own. “It is the task of the fabulatory
function,” writes Deleuze, “to invent a people,” to write into being a collective
or a people who are otherwise “missing.”12 Now, of course, algorithms engage
in fabulation — they invent a people, write them into being as a curious body of
correlated attributes, grouped into clusters derived from data that are them-
selves fabulatory devices. Indeed, the fabulation of the algorithm involves a
158 form of iterative writing, a collective enunciation of which we are all a com-
posite part. To consider the writing of algorithms, as I argue in this book, is to
Chapter Six
locate the ethicopolitical acts of relating narratives of oneself and others. So,
the methodology of a critical cloud ethics must also fabulate, must also neces-
sarily invent a people whose concrete particularities are precisely unattribut-
able, never fully enclosed within the attribute and its output. To fabulate in
this way means to take the arrangements of the algorithm (in this instance,
the show and tell neural nets) and imagine their mode of existence beyond the
decisive point of the target output. As method, a cloud ethics engages fabu-
lation in that it does not anchor narrative in the authorial source but rather
“digs under stories, cracking them open,” so that the reader might enter into
the space of the becoming-political of the algorithm.13 What this means is that
one cannot stand outside the algorithm to judge its morality, its role in doing
good or evil. Instead, one must begin from the iterative writing that is itself
generative of fungible thresholds of the good and the bad.
Just as computer science has become comfortable with experimental fabu-
lation and narrative, it is necessary for others to play with the arrangements,
thresholds, and assumptions of algorithms. The computer scientists are, for
example, apparently comfortably drawing on philosophy and ethics in their as-
sertions that machine learning contains within it a capacity for something like
machine reasoning.14 Methodologically, we social scientists, and humanities
and ethics scholars, must also find ways to engage experimentally in algorith-
mic fabulation. Let us begin by digging under the narratives of the show and
tell algorithms, cracking them open to find the moments of nonclosure, writ-
ing them in ways that lever open their ethicopolitical orientations. Put simply,
as algorithms work to derive future propensities from collections of attributes,
so we too must curate the attributes of the algorithm to signal its propensities
to colonize the unattributable future. A fabulatory method would take the ar-
rangements, perceptions, and accounts of the algorithm and rework them to
deepen the political intensity of what it means to output something into the
world (figure 6.3):
159
The Unattributable
Figure 6.3 The errors in the algorithm are used to gain intuition into what
the model saw and how it perceived the scene. Xu et al., “Show, Attend
and Tell,” 5.
of a cloud ethics across three aspects: apertures, opacity, and the unattributable.
Apertures
The Unattributable
tics, one in which deep neural networks do not merely undermine the already
decided functions of democracy (“decide the US president,” or “decide to leave
the European Union”) but actively place a limit on the potential for any other
form of political claim to be made in the future. What new entity or alterna-
tive model of citizenship could come into being after the algorithm has opti-
mized all outcomes? What new political claim, not yet registered as claimable,
could ever be made if its attributes are known in advance? These are not only
philosophical questions but also practical ones with a real purchase on the
politics of our times. Given the machine learning articulations I have followed
in the researching of this book, one would have to contemplate, for example,
whether a set of algorithms designed for the US presidential election actually
learned to recognize the attributes of “clusters” of anti-immigration and na-
tionalist sentiment via the “ground truth” data of the attributes of UK citi-
zens who voted to leave the European Union. In short, the “salient features” of
groups within the UK EU referendum may have gone on to have specific and
violent ethicopolitical relations to future families who find themselves sepa-
rated at the US-Mexico border.
What can a society be in the future if an algorithm is optimizing an output
that imagines the resolution of all future claims? The multiplicity of potential
pathways within the neural net — millions of potential alternative connections
and correlations — is reduced to one, condensed to a single output. Understood
in this way, contemporary algorithms are simultaneously constructing the ar-
chitecture of the problem space (What should the state do at the border? How
ought a society decide creditworthiness? What should a police force attend
to?) and generating the solution to the problem. “The problem always has the
solution it deserves,” writes Deleuze, so that the solution (output) is generated
by “the conditions under which it is determined as a problem.”18 If one under-
stands politics as expressing the impossibility of resolving all problems with a
solution, then algorithms appear to circumvent the political, because they ad-
just the parameters of the problem to experiment abductively with the “good
enough” solution. Alternative political arrangements become more difficult to
imagine, in part because — reversing Deleuze’s formulation — the solution al-
ways gets the problem it deserves; that is, the propositional arrangements of
the algorithm (the model of the problem) are immanently adjustable in rela-
tion to the target output (solution). If, for example, a cnn “shows” that the en-
tity “vehicle” extracted from a video feed has a 0.65 probability of being part of
a military convoy, and then an rnn “tells” that this is an actionable threat, at
162 the narrowed aperture of the output there is no place for a 0.35 probability of a
civilian bus in the scene. How could there be an alternative relation, a nonclo-
Chapter Six
sure, when the output has so intimately and abductively generated the model?
A strategy for cloud ethics would dwell for some time with the aperture of
the algorithm, the point where the vast multiplicity of parameters and hidden
layers becomes reduced and condensed to the thing of interest.19 To be clear,
the aperture is not a matter only of vision or opticality but, as Jonathan Crary
has detailed in his histories of perception, is specifically a means of dividing,
selecting, and narrowing the focus of attention.20 The aperture of perception
matters to my cloud ethics because it is a point of simultaneous closure and
nonclosure. It is a closure in the sense that there is no outside of the algorithm,
so that the weights and the outputs interpenetrate and mediate one another.
The aperture of the algorithm is an ethicopolitical closure of all the alternative
possible values, assumptions, and connections that the hidden layers may con-
tain. And yet, at the limit of the aperture is also an opening onto a moment of
decision that is always already fully ethicopolitical. From its roots in the Latin
aperire, or to open, the aperture is an opening or breach through which light
can enter. Understood as a form of aperture, the algorithm simultaneously fil-
ters down and closes off as it opens a gap or a breach. What this means in terms
of a strategy of cloud ethics is that one can envisage attending to the teeming
multiplicity of all past moments that have yielded their probabilities to the
model. A cloud ethics opens up the space of condensed output and makes the
decision difficult and freighted with the traces of rejected alternatives. The
aperture is the moment at which space is divided and the multiplicity is re-
duced to one, and yet it must also contain within it the traces of the opening
onto lost pathways. A cloud ethics alters the relation to a future that says “here
is a good enough solution” by revaluing the aperture as a moment of decision
in the dark, a calculation of the incalculable. The strategy involves reopening
the multiplicity of the algorithm, digging under the stories, and attending to the
branching pathways that continue to run beneath the surface.
How does one begin to locate the nonclosure in the arrangements of the
algorithms I have discussed in the chapters of this book? When I have given
lectures on cloud ethics, a number of generous computer scientists have asked
questions from the audience, urging me not to pursue the notion of revisiting
the weightings within a machine learning algorithm. For them, the adjustment
of weights within the hidden layers is an impenetrable process that retains its
opacity even to those who undertake it. “You cannot make the weights politi-
cal, Louise,” they have cautioned me, “because they are not really a thing. We
don’t know how they work. We are just messing around with them.” Yet, it is
exactly this kind of opaque, messy, and embodied relation to the weightings of 163
the algorithm that interests me. Moreover, the sense of nonknowledge — where
The Unattributable
the designers of algorithms “don’t know how they work” — is fertile ground for
insisting on a cloud ethics that has to decide in the darkness of not knowing.
Put simply, my cloud ethics would make the adjustment of a weight in a neural
net weightier, heavier in the sense of carrying the burden of difficulty and in-
transigence. Such a heaviness of the weighting of probabilities could shift the
ethical focus from the spatial arrangement of combinatorial possibilities to a
temporal and durational force of becoming. The weighting within one layer of
an algorithm becomes not only a spatial weight of probability but also a tem-
poral weight of duration. There is a kind of injunction here: the algorithm must
carry the weight of its weightings. As Deleuze has reflected on Bergson’s famous
division between space and duration, “Take a lump of sugar: it has a spatial
configuration. But if we approach it from that angle, all we will ever grasp
are differences in degree between that sugar and any another thing.”21 Via the
spatial differences of degree (via gradients), an rnn, for example, generates
the “attend” and “tell” outputs, deciding that “this is a woman on a sidewalk”
because of its spatial distance from any other thing. Returning to Deleuze’s
account of Bergson’s sugar cube: “But, it also has a duration, a rhythm of du-
ration, a way of being in time that is at least partially revealed in the process
of its dissolving, and that shows how this sugar differs in kind not only from
other things, but first and foremost from itself. This alteration, which is one
with the essence or substance of a thing, is what we grasp when we consider it
in terms of Duration.”22
It is possible to reimagine the attributes of something — “the essence or
substance of a thing” — that the machine learning algorithm derives entirely
from spatial clusters as durational qualities that differ in kind from themselves
and others. The algorithm’s generation of attributes carries only the lightness
of likelihoods, of differences of degree. A cloud ethics freights the machine
learning algorithm with the full durational weight — the waiting, the hesita-
tion, the doubtfulness — of the essence or substance of a thing. In this way, the
weight could be made to count as a meaningful moment of ethicopolitical de-
cision. And so, when an algorithm for immigration decisions is said to weight
the data inputs from past visa refusals, for example, the durational weight
would surface a different kind of attribute, something unattributable. All the
past moments of waiting in line (the wait in the weight), every past gathering
of an incomplete visa application, all the persistent indeterminacies that differ
in kind from themselves — these are the materials of a cloud ethics.
In revisiting the weights of algorithms, I am proposing that there are mo-
164 ments of undecidability within all technologies that enter the world as algo-
rithmic decision systems. With the experience of undecidability comes also a
Chapter Six
Opacity
Amid the widespread moral panic surrounding the black box of the algorithm
and the political demands for transparency, accountability, and explainabil-
ity, in this book I make a counter case for opacity and the giving of partial ac-
counts. I have argued that responsibility for algorithmic decisions can never
take the form of a clear-sighted account of the action. “Responsibility,” as
Thomas Keenan writes, “is not a moment of security or cognitive certainty”
in which one makes a “choice between yes and no, this or that.”26 To estab-
lish a moment of cognitive certainty regarding the algorithm — to demand its
transparency and accountability — is not at all the politics of a responsibil-
ity worthy of the name. Indeed, with its new forms of cognitive computing
and cloud reasoning, as I discuss in chapter 1, the contemporary algorithm is
well attuned to generating clear-sighted precise outputs from otherwise oc-
cluded situations. Contra the demand for transparency, then, the opacity of
the scene, and the undecidability of its salient features and attributes, is the
precondition for politics and ethics. “What could responsibility mean,” asks
Keenan, “without the risk of exposure to chance, without vulnerability,” with- 165
out the undecidability of what is taking place in a scene?27 Running against the
The Unattributable
grain of algorithmic logics that show, attend, and tell the meaning of a situa-
tion, responsibility means that one cannot see a clear path ahead because the
meaning of the situation is undecidable. The multiple branching points of the
decision tree or random forest algorithm do in fact carry the risk of exposure
to chance and vulnerability. A different kind of responsibility dwells at these
forks in the path, bearing the opacity of future consequences of the weighting
of one route over another.
In this book I differentiate the search for an encoded ethical framework
for algorithms from an ethicopolitics that is already present within the algo-
rithm’s arrangements. In so doing, I draw a distinction between a moral code of
prohibition and permission (what algorithms may or may not do) and an ethics
of the orientation one has to oneself and to others (how algorithms are generating
relations of self to self, and self to others).28 This distinction follows Spinoza’s
broader philosophical concern that “ethics has nothing to do with a moral-
ity.”29 Our ethicopolitical relations do not follow strictly from some already
formulated moral code, for we never have the security of these grounds from
which to act. As Deleuze has reflected on Spinoza’s ethics, “You do not know
beforehand what good or bad you are capable of; you do not know beforehand
what a body or a mind can do, in a given encounter, a given arrangement, a
given combination.”30 Similarly, for Isabelle Stengers, “every time we use the
term ‘ethics’ we must obviously distinguish it from the term ‘morality,’ ” so
that her speculative ambition is ethical in the sense of ethos or habit.31
Just as Spinoza’s ethics are ever in formation, forged through ethos, en-
counters, arrangements, and combinations, so a cloud ethics is located in the
encounters, arrangements, and combinations through which the algorithm is
generated. Some of these encounters and arrangements involve human beings;
others are between unsupervised algorithms and a corpus of data; and more
still involve algorithms interacting with other algorithms. This is an ethics,
then, that puts into question the authority of the knowing subject and opens
onto the plural and distributed forms of the writing of algorithms. Understood
in this way, “ethics is able to exceed its traditional determination” and be “dis-
placed” and considered “anew” in the relations forged with other beings.32 The
possibility of what Derrida calls “the opening of another ethics” is a different
kind of opening to the imagination of the prizing open of a black box or the il-
lumination of a dark calculus with moral oversight.33 It is an opening onto the
edges and limits of an iterative writing that is profoundly political, each one of
us drawn into the proximities of the output. Algorithms engage in writing in
166 and through their dense relations with people and things in a data scene, their
authorship uncertain and their emergence opaque. Here, in the opacity of not
Chapter Six
knowing in advance what the relations could be, we have found an opening for
taking responsibility.
If an ethical response to algorithmic decisions is to begin from opacity
and not from the demand for transparency, then what does this mean for the
strategies of a cloud ethics? Where the political demands for transparency re-
assert the objective vision of a disembodied observer, to stay with opacity is to
insist on the situated, embodied, and partial perspective of all forms of scien-
tific knowledge.34 Understood in Donna Haraway’s terms, to confront opacity
would not be a matter of failing to open the black box or to see something, but
rather it would acknowledge that “the knowing self is partial in all its guises,”
that practices of visuality necessarily enroll the opacity of ourselves and oth-
ers.35 Far from a lack of transparency limiting accountability, the condition of
opacity “allows us to become answerable for what we learn how to see.”36 This
“we” who learns, as I discuss in the machine learning practices of chapter 2, is
a composite figure, “stitched together imperfectly” from the iterative learning
of humans and machines.37
In the chapters of this book, I have been concerned to show how machine
learning algorithms generate their outputs through the opaque relations of
selves to selves, and selves to others, and how these outputs then annul and
disavow the opacity of their underlying relations in the name of a crystalline
clarity of the output. One mode of resistance to the clarity of the algorithmic
decision resides within the opacity of the subjects and what cannot be known
of their relations. Let us consider a practical example of such a strategy. In the
experimental development of the neural networks for show, attend, and tell,
the algorithm designers explain that when their model makes a mistake, they
are able to “exploit” what it manifests to gain “intuition into what the model
saw.”38 This kind of seeing on behalf of the algorithm sustains the god’s-eye vi-
sion of a computer scientist who may redistribute their way of seeing at will. A
cloud ethics can stay with the difficulty of intuiting what the model saw as a
form of partial, situated, and locally specific account. Among the examples of
the mistakes used to productively generate new learning is “a woman holding
a clock in her hand” and “a man wearing a hat and a hat on a skateboard.” The
algorithms have learned to recognize a clock and a hat, even when these are
absent in the scene, via their exposure to a corpus of data that has supplied the
feature vectors for these objects. The algorithm’s relations with a contingent
and opaque set of multiple other gradients have generated the mistaken pres-
ence of a clock and a hat (figures 6.4 and 6.5). 167
The cnns have generated an output of attentiveness to the focus of the
The Unattributable
scene (shown in the images as an area of white light) — this output itself con-
tingent on millions of past parameters — which enters the rnn as a data in-
put. The rnn then generates the “tell,” or the natural language description
of the scene, assigning a probability for each word in the descriptor. In the
(mis)recognition of “A man is talking on his cell phone while another man
watches,” the algorithm has assigned a probability of 1.00 (or a certainty) to
the indefinite article A, for example, but probabilities of 0.46, 0.24, and 0.17
to the nouns man, cell, and phone. These likelihood scores — low and yet appar-
ently the most probable — are entirely contingent on multiple algorithms’ past
relations with unknown other scenes. As I explain in chapter 4, the mistaken
recognition of the algorithm is not merely error or errancy, for the output is
entirely reasonable given the algorithm’s relations with a data world. In terms
of resistant practices, a cloud ethics is concerned not with fixing the algorithm
or eradicating its errancy, but rather with attending to how these moments
give accounts of algorithmic reason. What will take place when the recogni-
tion of a clock that is not present becomes the recognition of a weapon? What
will be authorized by the most likely cell phone that is an absent presence in
the scene?
To be clear, a cloud ethics will point to the moments when partial and sit-
uated algorithmic reason is sheltered as “mistake” or “error” and will reopen
what is incomputable in the making of a likelihood score. Such a strategy is
not the same as making a calculation transparent or accountable because it
recognizes that the 0.14 assigned to child or 0.46 and 0.37 to border and fence
are contingent on a vast multiplicity of necessarily opaque relations between
other people, algorithms, things, data, gradients, features, and images. When
the algorithm “tells” us that this is what matters in a scene, one could never
have a transparent account of how the story has been made tellable. Instead,
there are only the intransigent political relations that persist in the scene, the
moments of nonclosure that reopen the problem and the politics.
The Unattributable
The Unattributable
the universal (e.g., shared values or moral principles, ideas of economy, com-
mon laws) and the particular (that which is particular to the subject as indi-
vidual) than that which characterizes accounts of neoliberalism or biopolitics.
The machine learning algorithm iteratively moves back and forth between the
ground truth attributes of a known population (known in terms of the at-
tribution of qualities) and the unknown feature vector that has not yet been
encountered. In the space between a computer science universal of attributed
feature space and the particularity of the new entity, the algorithms learn how
to recognize and describe what is happening in the world. Thus, for example,
the arrested person in the custody suite is rendered knowable by algorithms
that calibrate their feature vector against the risk-scored attributes of another
population — these encounters unrelated except via this new relation of attri-
bution and adjudication.39 Likewise, when a group of teenagers try to board a
bus to attend a protest against the killing of a young African American man
in their city, their political rights to gather with unknown others are annulled
by algorithms that render them knowable via the clustering of their shared risk
attributes.40 What kind of relation to oneself and to others is manifest in this
curious relation of known attributes to unknown others? How can the algo-
rithm’s dissolution of other forms of being together be resisted? How might a
cloud ethics find political spaces for gatherings and collectivities that are pre-
cisely unattributable?
In the opening lines of this chapter, I cite Derrida’s remarks on the ethi-
copolitical stakes involved in resisting apartheid. “This is a moment,” he re-
flects, “of strategies, of rhetorics, of ethics, and of politics,” a moment when it
The Unattributable
society of attributes, for the algorithm’s attribute must always be a finite and
fixed essence to be classifiable and recognizable. A cloud ethics recognizes, in
contrast, the unlimited and infinite emergence of attributes that will defy the
boundaries of clusters and feature vectors. Put simply, the attribute is unat-
tributable to those on whom its effects are exerted.
There is a political refusal involved in claiming the unattributable. It is a
refusal to be governed by the attribute, an affirmation of “how not to be gov-
erned like that,” as Foucault has described it, a certain “will not to be governed”
as a cluster of attributes.43 My ethicopolitical relations to myself and to others,
then, cannot be limited to recognizable attributes but must begin from a rec-
ognition of singularity. As Simon Critchley writes, “Ethics is first and foremost
a respect for the concrete particularity of the other person in his or her singu-
larity.”44 This means that the other person cannot simply be an “example” of
some set of finite qualities derived from the gradients of norm and anomaly of
others. “Ethics begins as a relation with a singular, other person,” continues
Critchley, this singular other person who then “calls me into question” so that
“politics begins with ethics.”45 As the algorithm presents us with an attributive
other who is never singular or particular, it is crucial that we refuse the para-
digm of attribution and amplify the unattributable. I cannot know the essence
of the other as a cluster of attributes, and when the algorithm presents me with
the likelihood of their attribution, I must begin again from their singularity.
The question of an ethics of the unattributable is not only a matter of
thought or of the philosophy of algorithms in society. It must also be a mat-
ter of the practice of critique and the situated struggle for alternative routes
that are foreclosed in the calculation of an output. There have been many oc-
casions when I have asked the same question of border and policing authori-
ties, newly equipped with their predictive recognition algorithms: “How will
you know the characteristics of this unknown person who appears?” Always,
the response has been some variant on the same notion: “We do not need to
know them in particular; we have seen their attributes before.” The unattrib-
utable is thus at the very heart of the struggles over the place of algorithms in
our societies. A cloud ethics must pursue a conception of being together as a
society of the unattributable, where what we share in common is precisely our
singularities, and where our encounters with others can never be limited to a
finite collection of attributes.46 All the many moments when algorithms gather
to cluster our attributes, to assemble against us, when they seek to silence the
claim before it could even be made — these are the moments that must remain
172 open to what cannot be attributed. Ethicopolitical life is about irresolvable
struggles, intransigence, duress, and opacity, and it must continue to be so
Chapter Six
Introduction
1 Observations of tech start-ups’ pitches for government procurement of algorith-
mic systems, London, March 2016. An earlier iteration of the algorithms is de-
scribed in a 2012 video, “Monitoring Protests from Unstructured Text.”
2 Vice News journalist Jason Leopold made a Freedom of Information Act request
seeking all records pertaining to Freddie Gray and the subsequent protests and
riots in Baltimore, Maryland. For the full disclosed documents, see “dhs foia
Documents Baltimore Protests Freddie Gray.”
3 For the text of the Geofeedia case study, see aclu, “Baltimore County Police De-
partment and Geofeedia Partner.”
4 Computer scientists give accounts of how they use samples of training data to
train their model to recognize and extract text when it appears in an image.
See, for example, Misra, Swain, and Mantri, “Text Extraction and Recognition,”
13 – 19.
5 Browne, Dark Matters, 26.
6 Laughland, “Baltimore Unrest.”
7 Though many specific algorithms become secret and proprietary when bought
and licensed by corporations and governments, the upstream science of their
genesis is much more public, incremental, and debated. Methodologically, a de-
gree of speculation is necessarily involved in identifying which broad family of
algorithms is being used in a particular application. Among the many classic
computer science papers on contemporary machine learning algorithms I have
used to infer the operations of “secret” or “proprietary” algorithms are LeCun,
Bengio, and Hinton, “Deep Learning,” 436 – 4 4; Hinton and Salakhutdinov, “Re-
ducing the Dimensionality of Data,” 504 – 7; Krizhevsky, Sustskever, and Hinton,
“ImageNet Classification”; Simonyan and Zisserman, “Very Deep Convolutional
Networks”; and Hinton, “Deep Neural Networks for Acoustic Modeling,” 82 – 97.
8 “Amid the uncertainties, potentialities, and dangers created by the regime of
computation,” writes N. Katherine Hayles, “simulations — computational and
narrative — can serve as potent resources with which to explore and understand
the entanglement of language with code . . . and subjectivity with computation.”
Hayles, My Mother Was a Computer, 242.
9 Clustering techniques in machine learning are experimental and exploratory in
the sense that they discover previously unknown structure in a large volume of
data. The algorithms iteratively build a model by grouping the data points that
are most similar in their attributes. The model is then used to predict likely be-
havior in the future, such as the propensity for a group to “churn,” or to switch
commercial providers of, say, energy, credit, insurance, or telecommunications.
The aim in clustering, as Ethem Alpaydin describes, “is to find clusters of input”
data and to define each group as “similar in their attributes.” Alpaydin, Machine
Learning, 112. A set of algorithms designed for customer segmentation models are
also widely used for security and policing. For example, computer scientists have
trained machine learning algorithms on Twitter data to determine the attributes
of support for the Islamic State, training their classifier to “predict future sup-
port or opposition of isis [Islamic State of Iraq and Syria] with 87% accuracy.”
Emerging Technology from the arXiv, “Twitter Data Mining”; Caruso, “Can a
Social Media Algorithm Predict a Terror Attack?”
10 Crawford, “Artificial Intelligence’s White Guy Problem.” See also John Markoff’s
174 interview with Crawford on the ethical conduct of the designers of machine
learning algorithms; Markoff, “Artificial Intelligence Is Far from Matching Hu-
mans.” See also Metcalf, Keller, and boyd’s report on the ethical design of big
Notes to Introduction
Notes to Introduction
28 Foucault, “Ethics of the Concern for Self,” 299.
29 Seaver, “Algorithms as Culture,” 2.
30 Among the many depictions of the algorithm as a sequence of steps, for Tarleton
Gillespie, an algorithm is “a recipe composed in programmable steps.” Gillespie
“Algorithm,” 19. Pedro Domingos defines an algorithm as “a sequence of instruc-
tions.” Domingos, Master Algorithm, 1. David Berlinski identifies “an algorithm
[as] a finite procedure.” Berlinski, Advent of the Algorithm, xviii.
31 DeLanda, Intensive Science and Virtual Philosophy.
32 David Berry has distinguished between different algorithmic forms to show their
different implications for ordering and acting on society. Berry, Philosophy of
Software.
33 In Noel Sharkey’s assessment of the different juridical options for the control of
autonomous weapons systems, he writes that “we must urgently begin to develop
a principle for the human control of weapons that is founded on an understand-
ing of the process of human reasoning.” For Sharkey, the “reframing” of “auton-
omy in terms of human control” has the effect of making it “clear who is in con-
trol, where and when.” Sharkey “Staying in the Loop,” 23 – 38.
3 4 DeLanda, Philosophy and Simulation, 44 – 45.
35 For Totaro and Ninno the recursive logic defines “the logic of algorithms” and is
the “specific form in which the concept of function occurs in everyday life.” To-
taro and Ninno, “Concept of Algorithm,” 32. See also Beer, “Power through the
Algorithm?,” 985 – 1002.
36 The current state-of-the-art design of neural networks for voice and image rec-
ognition, for example, is experimenting with teaching algorithms how to teach
themselves which hidden layers could be bypassed without significantly altering
the output. LeCun, Bengio, and Hinton, “Deep Learning.”
37 So-called risk assessment engines powered by machine learning algorithms are
being used by judges to predict the likely probability of a convicted person reof
fending if they return to the community. Research has shown that the algorithms
are overestimating the risk of future crimes by black defendants — and therefore
proposing prison as the optimal judgment — even where race is not an input da-
tum in the algorithm. See Angwin et al., “Machine Bias”; and Liptak, “Sent to
Prison.” The United Kingdom’s Durham Police Constabulary deploys the Harm
Assessment Risk Tool (hart) to assess the likelihood of an individual reoffend-
ing if they are granted bail. See Bridge and Swerling, “Bail or Jail?”
38 Mackenzie, Machine Learners, 27.
39 Parisi, Contagious Architecture, 2.
40 Winslett, “Rakesh Agrawal Speaks Out.”
4 1 Amoore, Politics of Possibility, 50 – 52, 139 – 4 1.
4 2 Turing, “Systems of Logic Based on Ordinals,” 161–228. In letters sent to Max
Newman in 1940, Turing is “not sure whether my use of the word ‘intuition’ is
176 right or whether your ‘inspiration’ would be better” and considers that Newman
is “using ‘inspiration’ to cover what I call ‘ingenuity.’ ” Turing, “Letters on Logic
to Max Newman,” 212.
Notes to Introduction
Notes to Chapter 1
of attention.” Mackenzie, Machine Learners, 10.
Notes to Chapter 1
45 Leopold, “cia Embraces Cloudera Data Hub,” 2.
46 Galison, Image and Logic, 67. Office of the Director of National Intelligence, Intel-
ligence Community.
47 C. Anderson, “The End of Theory.”
48 J. G. Wilson, Principles of Cloud Chamber Technique, 3.
49 Gentner, Maier-Leibnitz, and Bothe, Atlas of Typical Expansion Chamber Photo-
graphs, 11.
50 Foucault, Security, Territory, Population, 6.
51 Barad, Meeting the Universe Halfway, 148.
52 Elden, “Secure the Volume,” 35 – 51; Crampton, “Cartographic Calculations of
Territory,” 92 – 103; Weizman, “Politics of Verticality.”
53 O’Brien, Definitive Guide to the Data Lake, 4.
54 The data lake architecture underpins the ability of commercial and government
agencies to gather and analyze “vast amounts of social networking platforms
emerging data” in a “single, centralized repository.” O’Brien, Definitive Guide to
the Data Lake, 3.
55 Parisi, Contagious Architecture, 2.
56 Josephson and Josephson, Abductive Inference, 12.
57 Konkel, “The cia’s Deal with Amazon.”
58 Dunning and Friedman, Practical Machine Learning, 23.
59 Interview with designers of cnn algorithms for facial recognition, London, May
2017.
60 J. G. Wilson, Principles of Cloud Chamber Technique, 122.
61 Rochester and Wilson, Cloud Chamber Photographs, vii.
62 Galison, Image and Logic, 130.
63 In the past, the use of data-mining techniques for security and intelligence was
limited, in part, by the lack of available training datasets. The storage of data in
the cloud provides a readily available supply of training data (and a flexible com-
putational capacity) through which humans and algorithms can advance their
capacities to perceive new emergent events.
64 Interview with UK government procurement officers, London, October 2017.
65 Office of the Director of National Intelligence, Intelligence Community, ii.
66 Hayles, How We Think, 230.
7 Office of the Director of National Intelligence, “icite Fact Sheet.”
6
68 “Web Intelligence”; my emphasis.
69 O. Halpern, Beautiful Data, 16, 40.
70 Derrida, Archive Fever, 17.
71 James Bridle’s interactive installation is available at Hyper-Stacks, accessed June
2016, http://hyper-stacks.com.
72 Derrida, Archive Fever, 18.
73 Observations conducted in the Department of Physics, Durham University, Au-
180 gust 2015.
74 Interviews and observations with the designers of credit card fraud detection al-
gorithms, Frankfurt, April 2014.
Notes to Chapter 1
Notes to Chapter 2
9 Lin et al., “Automatic Detection,” 802 – 10.
10 In machine learning and pattern recognition applications, a feature vector is
a means of numerically representing an n-dimensional object in space so that
it can be computed. Thus, for example, the image of a human face is rendered
computable in a biometrics application via the extraction of feature vectors cor-
responding to pixelated data points. In the movement of the hands of surgeons
and surgical robots, the feature vectors denote the position along a trajectory in
three-dimensional space.
11 Lin et al., “Automatic Detection,” 806.
12 Reiley, Plaku, and Hager, “Motion Generation of Robotic Surgical Tasks.”
13 N. Katherine Hayles describes a “cognitive assemblage” of human and techni-
cal “cognizers” who are “flexibly attending to new situations,” incorporating
modes of knowledge, and “evolving through experience to create new responses.”
Hayles, Unthought, 32. Adrian Mackenzie proposes that the “various subject po-
sitions” of machine learning are “neither unified nor fixed” but malleable and
transformative. Mackenzie, Machine Learners, 207.
1 4 Van den Berg et al., “Superhuman Performance of Surgical Tasks.”
15 Robotic surgery actively premediates the actualized surgery in the sense intended
by Richard Grusin in his work on premediation. The robot’s virtual architecture
allows the surgeon to explore the trajectories of the surgery in advance of the ac-
tual surgery, anticipating the optimal trajectories of movement, experimenting,
and preempting potential problems. Grusin, Premediation.
16 Prentice, Bodies in Formation, 227.
17 Prentice, Bodies in Formation, 227 – 29.
18 Prentice, Bodies in Formation, 231.
19 Popescu and Mastorakis, “Simulation of da Vinci Surgical Robot,” 141.
20 Interview with obstetric surgical team, Newcastle-upon-Tyne, April 2016.
21 Barad, Meeting the Universe Halfway, 208.
22 Jochum and Goldberg, “Cultivating the Uncanny.”
23 Following Henri Bergson, the cut and the multiplicity are actually two distinct
forms of multiplicity. As Gilles Deleuze writes, “Remember that Bergson opposed
two types of multiplicity — actual multiplicities that are numerical and discon-
tinuous and virtual multiplicities that are continuous and qualitative.” Deleuze,
Bergsonism, 79. For the singular cut to take place, the virtual multiplicities of past
encounters between people and algorithms must be rendered as actual divisible
numerical multiplicities, or feature spaces.
2 4 In a computer science paper on machine learning for robot surgery, the authors
present their cognate findings in military robots, landmine detection, and ro-
bot inspection of nuclear contamination. The development of machine learn-
ing algorithms takes place indifferent to the environment as such because it
understands the environment explicitly in terms of feature space and not place.
182 Popescu and Mastorakis, “Simulation of da Vinci Surgical Robot,” 137.
25 Hayles, How We Became Posthuman, 4.
26 Hayles, How We Became Posthuman, 287.
Notes to Chapter 2
Notes to Chapter 2
blackness and the new regimes of racialized recognition that accompany algo-
rithmic techniques. Amaro, “Pre-emptive Citizenship.”
38 Bucher, If . . . Then, 4.
39 Domingos, Master Algorithm, 189.
40 Quinlan, “Induction of Decision Trees,” 82.
4 1 Quinlan, “Induction of Decision Trees,” 83.
4 2 Quinlan, “Induction of Decision Trees,” 104.
43 Though it may appear inconsequential that an algorithm may be less able to
identify shape or outline than it is able to detect texture or color, in fact it is be-
coming increasingly important to our contemporary regime of recognition. For
example, neural networks used in the assessment of medical images are more
attuned to the texture or color of a human organ than they are to its shape or
indeed to the contours of a lesion. Hendrycks et al., “Natural Adversarial Exam-
ples,” 2.
4 4 LeCun, Bengio, and Hinton, “Deep Learning,” 436.
45 Krizhevsky, Sutskever, and Hinton, “ImageNet Classification,” 1097 – 1105.
46 Across a wide range of recognition algorithms, including tumor detection in medi-
cal imaging and facial recognition, the AlexNet algorithm is used as the pretrained
model, with the domain-specific images (e.g., datasets of surgical photographs)
added only to the training of the last hidden layer of the neural network.
47 Krizhevsky, Sutskever, and Hinton, “ImageNet Classification,” 1097.
48 Amazon Mechanical Turk has become a vehicle for the human labeling of images
that will be used to train a machine learning algorithm. Described by Amazon as
“a marketplace for work that requires human intelligence,” the casualization and
precarity of this form of work means that a population of workers is engaged in
the preparation of data for algorithmic systems that may at some future moment
be used against them or others in their community.
49 Observations of the process of training algorithms for image recognition, Go-
thenburg, November 2016.
50 MacKay, Information Theory, 345.
51 Interviews and observations with designers of algorithms for airport border secu-
rity, September 2016.
52 LeCun, Bengio, and Hinton, “Deep Learning,” 436.
53 McGeoch, Guide to Experimental Algorithmics, 87.
54 Interview with test computer scientist working with ibm on facial recognition,
London, June 2018.
55 All forms of recognition, as political claims, carry within them the possibility of
misrecognition or being unrecognized. As Judith Butler has put the problem, “To
ask for recognition, or to offer it, is precisely not to ask for recognition for what
one already is. It is to solicit a becoming, to petition the future always in relation
to the Other.” Butler, Precarious Life, 44.
184 56 The Point Cloud Library is an open source repository of point cloud 2d and 3d
images and object recognition algorithms. The Point Cloud Library offers tech-
niques to “stitch 3d point clouds together” and to “segment relevant parts of a
Notes to Chapter 2
Notes to Chapter 3
work to an individual, but rather “constructs a certain being of reason that we
call ‘author.’ ” The author function unifies the oeuvre, or the body of work, so
that unresolved tensions or contradictions appear resolvable and “incompatible
elements are at last tied together.” Foucault, “What Is an Author?,” 384 – 85.
6 Foucault, “What Is an Author?,” 377.
7 Lev Manovich has depicted the distinction between narrative and database as
one of competing forms, even as “enemies.” Manovich, Language of New Media, 228.
Contra Manovich, N. Katherine Hayles develops an understanding of the interde-
pendent mediations of narrative and database, nonetheless proposing that “they
remain different species, like bird and water buffalo.” Hayles, How We Think, 178.
With contemporary machine learning algorithms, however, the indeterminacy
that Hayles suggests “databases find difficult to tolerate” moves closer to the “inex-
plicable” and “ineffable” she associates with the narrative form (178 – 79).
8 Kirby, Quantum Anthropologies, xi.
9 Reflecting on the Greco-Roman practice of writing the hupomnemata, or indi-
vidual notebooks, Foucault describes writing that “resists scattering” by the “se-
lecting of heterogeneous elements.” The writing becomes a “matter of unifying
heterogeneous fragments” into a body that is recognizable not as a “body of doc-
trine” but as a “certain way of manifesting oneself to oneself and to others.” Fou-
cault, “Self Writing,” 212 – 13, 216.
10 The computational practice of natural language processing is historically in-
tertwined with linguistic theory and, specifically, the debates on how language
is learned by humans. Noam Chomsky’s Platonic approach to linguistics influ-
enced strongly nlp methods from the 1950s to the 1990s. Chomsky’s approach
to the “generative grammar” of language learning favored rules-based models
of syntactic structure and, thus, supplied to computer science the grounds for
devising algorithmic rules for the extraction of meaning from language. For an
account of what was at stake in the 1950s debates between behavioral and rules-
based approaches to language learning, see Chomsky, “Bad News,” 26 – 58.
11 Amoore, Politics of Possibility, 39 – 45.
12 Statistical machine translation advanced significantly with the availability of
labeled training data derived from the translations of all European Union docu-
mentation into the eleven recognized official languages of the EU. In effect, the
human-translated documents provided a vast corpus of training data for transla-
tion algorithms. See Koehn, “Europarl.” Advances in deep neural networks have
developed these techniques beyond the statistical to devise predictive models
for the recognition of human language. See Bahdanau, Cho, and Bengio, “Neu-
ral Machine Translation.” For an overview of the contemporary state of the art
in nlp methods, see Hirschburg and Manning, “Advances in Natural Language
Processing,” 261 – 66.
13 Kurzweil, The Singularity Is Near.
186 1 4 For the Google research team’s account of the development of their algorithms
see Pickett, Tar, and Strope, “On the Personalities of Dead Authors.”
15 Juola, “How a Computer Program Helped Show.”
Notes to Chapter 3
Notes to Chapter 3
45 Deleuze, Negotiations, 133.
46 Fowles, “Notes on an Unfinished Novel,” 18.
47 “Writing unfolds like a game [jeu] that invariably goes beyond its own rules and
transgresses its limits,” writes Foucault. “In writing, the point is not to manifest
or exalt the act of writing, nor is it to pin a subject within language; it is, rather,
a question of creating a space into which the writing subject constantly disap-
pears.” Foucault, “What Is an Author?,” 378.
48 Gilles Deleuze considers there to be “no literature without fabulation,” so that
“writing is a question of becoming, always incomplete, always in the midst of be-
ing formed.” Deleuze, Essays Critical and Clinical, 1 – 3 .
49 Critchley, Ethics of Deconstruction, 34.
50 Fowles and Vipond, “Unholy Inquisition,” 368, 372.
51 Fowles and Vipond, “Unholy Inquisition,” 372.
52 Critchley, Ethics of Deconstruction, 48.
53 Critchley, Ethics of Deconstruction, 48.
54 Fowles and Vipond, “Unholy Inquisition,” 383.
55 Giorgio Agamben understands the author “as gesture” in the sense that “the au-
thor is present in the text only as a gesture that makes expression possible pre-
cisely by establishing a central emptiness within this expression.” Agamben, “Au-
thor as Gesture,” 66.
56 Mantel, “Day Is for the Living.”
57 The concept of fabulation serves to further displace the figure of the author in
favor of a focus on the power of writing. As Deleuze puts the problem: “There is
no literature without fabulation, but as Bergson was able to see, fabulation — the
fabulatory function — does not consist in imagining or projecting an ego. Rather,
it attains these visions, it raises itself to these becomings and powers.” Deleuze,
Essays Critical and Clinical, 3.
58 Deleuze, Essays Critical and Clinical, 118.
59 Foucault, “Self Writing,” 212.
60 Deleuze, Negotiations, 126.
61 The methods of deconstruction have long supplied to philosophy a means of en-
gaging with an extended sense of writing, or what Jacques Derrida has called
“archi-writing” (archi-écriture). Derrida, Of Grammatology, 15 – 20.
2 Derrida, Limited Inc., 148.
6
63 Derrida, Limited Inc., 19.
64 Derrida, Limited Inc., 20.
65 Derrida, Limited Inc., 152.
66 Hinton et al., “Deep Neural Networks,” 83.
67 Hinton et al., “Deep Neural Networks,” 84.
68 Critchley, Ethics of Deconstruction, 21.
69 Nissenbaum, Privacy in Context.
188
4. The Madness of Algorithms
1 On the notion of the flash crash and the advent of high-frequency trading algo-
Notes to Chapter 3
Notes to Chapter 4
16 Erickson et al., How Reason Almost Lost Its Mind, 2.
17 Erickson et al., How Reason Almost Lost Its Mind, 30.
18 Norbert Wiener, “Second Industrial Revolution and the New Concept of the Ma-
chine,” folder 619, box 28D, mc 22, Wiener Papers.
19 O. Halpern, Beautiful Data, 150.
20 O. Halpern, Beautiful Data, 151.
21 Chun, Control and Freedom, 300; Liu, Freudian Robot, 202.
22 M. Beatrice Fazi argues that computation is a process of determining indetermi-
nacy, where the generative potential of algorithms works with Turing’s notion of
the incomputable. Fazi, Contingent Computation. The kind of generative potential
identified by Fazi also drives the capacity of algorithmic systems to derive value
from speculative and experimental models.
23 Foucault, History of Madness, 164.
2 4 Foucault, History of Madness, xxxii.
25 Foucault, History of Madness, 169.
26 Jacques Derrida describes Foucault’s determination to avoid the trap of a history
of madness untamed by reason as “the maddest aspect of his project.” Derrida,
“Cogito and the History of Madness,” 45; and Foucault, “Reply to Derrida,” 575.
27 Derrida, “Cogito and the History of Madness,” 46.
28 Derrida, “Cogito and the History of Madness,” 46.
29 “To the extent that madness is a form of knowledge,” writes Foucault, “it was
also a set of norms, both norms against which madness could be picked out as
a phenomenon of deviance within society, and, at the same time, norms of be-
haviour for normal individuals.” The idea of madness, for Foucault, defined “a
certain mode of being of the normal subject, as opposed to and in relation to the
mad subject.” The dissension between madness and reason, then, also marks the
relation of selves to others in relation to thresholds of norm and anomaly. Fou-
cault, Government of Self and Others, 3.
30 Foucault, History of Madness, 426.
31 Foucault, History of Madness, 426.
32 Foucault, History of Madness, 461.
33 Foucault, History of Madness, 159.
3 4 Foucault, History of Madness, 527.
35 Foucault, History of Madness, 528.
36 Foucault, History of Madness, 164.
37 Derrida, “Cogito and the History of Madness,” 62 – 63.
38 In place of the notion that the madness of algorithms denotes a departure from
some normal course, I envisage something closer to Brian Rotman’s concept
of “becoming beside oneself.” Rotman’s concept depicts a plurality of being in
which the self is “distributed and in excess of unity.” Rotman, Becoming Beside
Ourselves, 103.
190 39 The nsf funded Nicholas Evans’s study “Ethical Algorithms in Autonomous
Vehicles” in 2017. See nsf Award Abstract 1734521. See also Nyholm and Snids,
“Ethics of Accident-A lgorithms,” 1275 – 89.
Notes to Chapter 4
40 Phys.org, “UMass Lowell Professor Steers Ethical Debate.” The encoded ethics
of algorithms for autonomous vehicles poses questions such as “Should your self-
driving car protect you at all costs? Or should it steer you into a ditch to avoid
hitting a school bus full of children?” The notion of a moral choice to be made
and to be encoded within the algorithm looms large in these accounts of the eth-
ics of decision. Webster, “Philosophy Professor Wins nsf Grant.”
4 1 Phys.org, “UMass Lowell Professor Steers Ethical Debate.”
4 2 The school bus features discursively within many experimental scenarios for
the development of ethical codes governing algorithmic action. The image of a
school bus embodies the fears of algorithms recognizing objects for drone sur-
veillance, and the school bus similarly serves as the epitome of a mad decision
within autonomous vehicles research.
43 Derrida, Gift of Death, 24.
4 4 Eklund, Nichols, and Knutson, “Cluster Failure,” 7900 – 7905.
45 Timmer, “Software Faults Raise Questions.”
46 Brackeen, “Facial Recognition Software.”
47 Foucault, History of Madness, 241.
48 Domingos, Master Algorithm, 83.
49 Kourou et al., “Machine Learning Applications,” 8 – 17.
50 As both Derrida and Foucault elaborate, madness is what cannot be said, the un-
spoken that is absent from a body of work. Derrida, “Cogito and the History of
Madness,” 51.
51 In all the domains where I have followed the development of algorithms for state
security — border controls, counter-terror, policing, and immigration control —
my discussions with algorithm designers have included some reference to the use
of random forest algorithms as preferred by operational security teams. Though
I cannot definitively explain the allure these algorithms hold for the state, in
many cases, the random forest algorithm was using proxy data to identify pat-
terns associated with potential threats.
52 Grothoff and Porup, “nsa’s skynet Program.”
53 Grothoff and Porup, “nsa’s skynet Program,” 2.
54 Amoore, “Data Derivatives,” 24 – 43.
55 Breiman, “Random Forests,” 5 – 32. Breiman’s paper has had extraordinary in-
fluence on the development of machine learning, having been cited more than
thirty-five thousand times.
56 Breiman, “Random Forests,” 6.
57 Breiman, “Random Forests,” 7. See also Abdulsalam, Skillicorn, and Martin,
“Classification Using Streaming Random Forests,” 24.
58 Daston, “Probabilistic Expectation,” 234 – 60.
59 Grothoff and Porup, “nsa’s skynet Program,” 3.
60 Abdulsalam, Skillicorn, and Martin, “Classification Using Streaming Random
Forests,” 25. 191
61 Tucker, “New ai Learns through Observation Alone.”
62 Foucault, History of Madness, 521.
Notes to Chapter 5
63 It has long been acknowledged in relation to financial markets that notions of
irrationality or chance have a place within the making of ideas of rational eco-
nomic humans. De Goede, Virtue, Fortune, and Faith. Derrida, Death Penalty, 21.
64 Foucault, History of Madness, 521.
65 For Derrida, “calculation is always calculating with what is incalculable as well
as what is non-calculable” so that there is always also indecision and undecidabil-
ity. Derrida, Death Penalty, 197.
66 Derrida, Death Penalty, 4, 42.
67 Derrida, Death Penalty, 93.
68 Derrida’s discussion of the scaffold invokes the material scaffold of the sovereign
scene of the death penalty as well as the “scaffolding” that renders an act as an
unconditional principle. Derrida, Death Penalty, 21.
69 Derrida, Death Penalty, 26.
70 Derrida, Death Penalty, 6, 139.
Notes to Chapter 6
35 Haraway, Modest_Witness@Second_Millennium, 24, 34.
36 Barad, “On Touching,” 208.
37 Haraway, Modest_Witness@Second_Millennium, 35.
38 Foucault, Courage of Truth, 79.
39 Foucault, Courage of Truth, 30.
40 Butler, Giving an Account of Oneself, 19.
4 1 Foucault, Government of Self and Others, 62.
4 2 Foucault, Wrong-Doing, Truth-Telling, 200.
43 Feynman, What Do You Care What Other People Think?, 139.
4 4 Berlant, Cruel Optimism, 23 – 2 4.
45 Berlant, Cruel Optimism, 262.
46 There is widespread acknowledgement among self-styled data scientists that 90
percent of the work of building a model remains a task of data preparation.
47 Parisi, Contagious Architecture, 95.
48 Feynman, What Do You Care What Other People Think?, 214.
49 Harding, Snowden Files; and Greenwald, No Place to Hide.
50 D. Anderson, Attacks in London and Manchester.
51 Keenan, Fables of Responsibility, 3, 42.
52 Keenan, Fables of Responsibility, 12; and Derrida, “Force of Law,” 26.
53 Observation of presentations of “threat intelligence” software to UK govern-
ment, London, June 2017.
54 Truvé, “Building Threat Analyst Centaurs.”
55 Butler, Giving an Account of Oneself, 19, 45.
56 Derrida, Gift of Death, 24; and Keenan, Fables of Responsibility, 2.
57 Deleuze, Bergsonism, 29.
58 Barad, “On Touching,” 212.
59 Barad, Meeting the Universe Halfway, 265.
60 Feynman, Surely You’re Joking, 247; my emphasis.
6. The Unattributable
1 Xu et al., “Show, Attend and Tell.”
2 Vinyals et al., “Show and Tell,” 2048 – 57.
3 Xu et al., “Show, Attend and Tell,” 2048.
4 Vinyals et al., “Show and Tell,” 653.
5 Crary, Suspensions of Perception.
6 Xu et al., “Show, Attend and Tell,” 2048.
7 Vinyals et al., “Show and Tell,” 652.
8 Xu et al., “Show, Attend and Tell,” 2048.
9 Derrida, Limited Inc., 152 – 53.
10 Shane and Wakabayashi, “Business of War.”
11 Derrida, Limited Inc., 151.
194 12 Deleuze, Essays Critical and Clinical, 4.
13 Deleuze, Essays Critical and Clinical, 113.
1 4 Bottou, “From Machine Learning to Machine Reasoning,” 133 – 49.
Notes to Chapter 6
15 To begin from the propensities of the algorithm, and not from strategies to over-
turn it, is to echo Isabelle Stengers’s sense that “the diagnosis of becomings” is
not a strategy for alternatives but rather a “speculative operation, thought ex-
periment” in which the possibilities are not determined. Stengers, Cosmopolitics I,
12 – 13.
16 Addressing the machine learner “kittydar,” Adrian Mackenzie describes how
“the software finds cats by cutting the images into smaller windows. For each
window, it measures a set of gradients” and then “compares these measurements
to the gradients of known cat images.” Mackenzie, Machine Learners, 4 – 5.
17 The UK Information Commissioner’s Office (ico) is investigating the use of mil-
lions of Facebook profiles to train the Cambridge Analytica algorithms. At the
time of writing, the maximum fine has been imposed on Facebook for the unlaw-
ful sharing of data. See Greenfield, “Cambridge Analytica Files.”
18 Deleuze, Bergsonism, 16.
19 The question, as Henri Bergson formulated it, “is not how perception arises, but
how is it limited,” so that the image of a whole scene is “reduced to the image of
that which interests you.” Bergson, Matter and Memory, 34.
20 For Crary, perception exceeds the “single-sense modality of sight,” extending
to “hearing and touch” and “most importantly, irreducibly mixed modalities.”
Crary, Suspensions of Perception, 1, 4.
21 Deleuze, Bergsonism, 31.
22 Deleuze, Bergsonism, 32.
23 Simon Critchley argues that “deconstruction is a philosophy of hesitation”
in that it carries with it “the experience of undecidability” essential to ethics.
Critchley, Ethics of Deconstruction, 42.
2 4 For Derrida, there is a close relation between “feeling the weight” of something,
“peser,” and thought, “penser.” Derrida, Death Penalty, 93.
25 For Derrida, the “trace of the rejected alternative” is not a matter of “either this
or that” in the sense that some alternative path could be reopened, but a “neither
this . . . nor that,” where the decisive moment is unknowable. Derrida, Writing
and Difference, 112.
26 Keenan, Fables of Responsibility, 1, 51.
27 Keenan, Fables of Responsibility, 51.
28 Writing on the genealogy of ethics, Michel Foucault describes the transforma-
tion in how Greek and Christian societies viewed themselves: these changes
“are not in the code, but in what I call the ‘ethics’ ” which is in “their relations to
themselves and to others.” Foucault, “On the Genealogy of Ethics,” 255, 266.
29 Deleuze, Spinoza, 125.
30 Deleuze, Spinoza, 125.
31 Stengers, Thinking with Whitehead, 515.
32 Critchley, Ethics of Deconstruction, 18.
33 Derrida, Limited Inc., 122. 195
3 4 Haraway, “Situated Knowledges,” 575 – 99.
35 Haraway, “Situated Knowledges,” 586.
Notes to Chapter 6
36 Haraway, “Situated Knowledges,” 583.
37 Haraway, “Situated Knowledges,” 586.
38 Xu et al., “Show, Attend and Tell,” 2052, 2055.
39 Burgess, “UK Police Are Using ai.”
40 Laughland, “Baltimore Unrest.”
4 1 Derrida, Limited Inc., 152 – 53.
4 2 Deleuze, Spinoza, 17, 51.
43 Foucault, “What Is Critique?,” 265, 278.
4 4 Critchley, Ethics of Deconstruction, 48.
45 Critchley, Ethics of Deconstruction, 48.
46 As Deleuze has written, the singularities “connect with one another in a manner
entirely different from how individuals connect” and can be neither clustered
nor “grouped [or] divided in the same way.” Deleuze, Pure Immanence, 30.
This page intentionally left blank
BIBLIO GR A PH Y
Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. “Neural Machine Trans-
lation by Jointly Learning to Align and Translate.” Presented at the International
Conference on Learning Representations, San Diego, California, May 7 – 9, 2015.
Bal, Mieke. “The Commitment to Look.” Journal of Visual Culture 4, no. 2 (2005):
145 – 62.
Barad, Karen. Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning. Durham, NC: Duke University Press, 2007.
Barad, Karen. “On Touching: The Inhuman That Therefore I Am.” Differences: A Jour-
nal of Feminist Cultural Studies 23, no. 3 (2012): 206 – 23.
Beer, David. “Power through the Algorithm? Participatory Web Cultures and the
Technological Unconscious.” New Media and Society 11, no. 6 (2009): 985 – 1002.
Bennett, Jane. Vibrant Matter. Durham, NC: Duke University Press, 2010.
Bennington, Geoffrey. “A Moment of Madness: Derrida’s Kierkegaard.” Oxford Liter-
ary Review 3, no. 1 (2011): 103 – 27.
Bergson, Henri. The Creative Mind: An Introduction to Metaphysics. 1946. New York: Do-
ver, 2007.
Bergson, Henri. Matter and Memory. 1912. Mineola, NY: Dover Philosophical Classics,
2004.
Berlant, Lauren. Cruel Optimism. Durham, NC: Duke University Press, 2011.
Berlinski, David. The Advent of the Algorithm. New York: Harcourt, 2000.
Berry, David. The Philosophy of Software. Basingstoke: Macmillan, 2011.
Borch, Christian, and Ann-Christina Lange. “High Frequency Trader Subjectivity:
Emotional Attachment and Discipline in an Era of Algorithms.” Socio-Economic
Review 15 (2017): 283 – 306.
Bottou, L. “From Machine Learning to Machine Reasoning.” Machine Learning 94
(2014): 133 – 49.
boyd, danah, and Kate Crawford. “Critical Questions for Big Data.” Information, Com-
munication and Society 15, no. 5 (2012): 662 – 79.
Brackeen, Brian. “Facial Recognition Software Is Not Ready for Use by Law Enforce-
ment.” TechCrunch, June 22, 2018.
Braidotti, Rosi. The Posthuman. Cambridge: Polity, 2013.
Braidotti, Rosi. “Posthuman, All Too Human: Towards a New Process Ontology.”
Theory, Culture and Society 23, no. 7 – 8 (2006): 197 – 208.
Brandom, Russell. “A New System Can Measure the Hidden Bias in Otherwise Secret
Algorithms.” Verge, May 25, 2016. https://www.theverge.com/2016/5/25/11773108
/research-method-measure-algorithm-bias.
Bratton, Benjamin. The Stack: On Software and Sovereignty. Cambridge, MA: mit Press,
2015.
Breiman, Leo. “Random Forests.” Machine Learning 45, no. 1 (2001): 5 – 32.
Bridge, Mark, and Gabriella Swerling. “Bail or Jail? App Helps Police Make Decision
about Suspect.” Sunday Times (London), May 11, 2017. https://www.thetimes.co.uk
/article/bail-or-jail-app-helps-police-make-decision-about-suspect-kv766zjc9. 199
Bridle, James. New Dark Age: Technology and the End of the Future. London: Verso, 2018.
Bridle, James. “Something Is Wrong on the Internet.” Medium, November 6, 2017.
Bibliography
Browne, Simone. Dark Matters: On the Surveillance of Blackness. Durham, NC: Duke
University Press, 2015.
Bucher, Taina. If . . . Then: Algorithmic Power and Politics. Oxford: Oxford University
Press, 2018.
Burgess, Matt. “Google Wants to Predict the Next Sentences of Dead Authors.”
Wired (UK), February 26, 2016. https://www.wired.co.uk/article/google-author
-create-artificial-intelligence.
Burgess, Matt. “UK Police Are Using ai to Inform Custodial Decisions — But It Could
Be Discriminating against the Poor.” Wired, March 1, 2018.
Butler, Judith. Giving an Account of Oneself: A Critique of Ethical Violence. Amsterdam:
Van Gorcum Press, 2003.
Butler, Judith. Precarious Life: The Powers of Mourning and Violence. London: Verso, 2004.
Caruso, Catherine. “Can a Social Media Algorithm Predict a Terror Attack?” mit
Technology Review, June 16, 2016. https://www.technologyreview.com/s/601700
/can-a-social-media-algorithm-predict-a-terror-attack/.
Central Intelligence Agency. “cia Creates a Cloud: An Interview with cia’s Chief
Intelligence Officer, Doug Wolfe, on Cloud Computing at the Agency.” cia Fea-
tured Story Archive, 2014. https://www.cia.gov/news-information/featured-story
-archive/2014-featured-story-archive/cia-creates-a-cloud.html.
Cheney-Lippold, John. We Are Data: Algorithms and the Making of Our Digital Selves. New
York: New York University Press, 2019.
Chomsky, Noam. “Bad News: Noam Chomsky — A Review of B. F. Skinner’s Verbal
Behavior.” Language 35, no. 1 (1959): 26 – 58.
Chun, Wendy Hui Kyong. Control and Freedom: Power and Paranoia in the Age of Fiber
Optics. Cambridge, MA: mit Press, 2006.
Comaniciu, Dorin, and Peter Meer. “Mean Shift: A Robust Approach toward Feature
Space Analysis.” ieee Transaction on Pattern Analysis and Machine Intelligence 24, no.
5 (2002): 605.
Connolly, William. “Beyond Good and Evil: The Ethical Sensibility of Michel Fou-
cault.” Political Theory 21, no. 3 (1993): 365 – 89.
Covington, Paul, Jay Adams, and Emre Sargin. “Deep Neural Networks for YouTube
Recommendations.” Paper presented at the 10th acm Conference on Recom-
mender Systems, Boston, Massachusetts, September 15 – 19, 2016.
Cowen, Deborah. The Deadly Life of Logistics. Minneapolis: University of Minnesota
Press, 2014.
Crampton, Jeremy. “Cartographic Calculations of Territory.” Progress in Human Geog-
raphy 35, no. 1 (2010): 92 – 103.
Crary, Jonathan. 24/7: Late Capitalism and the End of Sleep. London: Verso, 2013.
Crary, Jonathan. Suspensions of Perception: Attention, Spectacle, and Modern Culture.
Cambridge, MA: mit Press, 2001.
Crawford, Kate. “Artificial Intelligence’s White Guy Problem.” New York Times, June
200 25, 2016.
Crawford, Kate. “Dark Days: Artificial Intelligence and the Rise of Fascism.” Pre-
sented at the sxsw Conference, Austin, Texas, March 12, 2017.
Bibliography
Crawford, Kate, and Jason Schultz. “Big Data and Data Protection: Toward a Frame-
work to Redress Predictive Privacy Harms.” Boston College Law Review 55, no. 1
(2014): 93 – 128.
Critchley, Simon. The Ethics of Deconstruction: Derrida and Levinas. Oxford: Blackwell,
1992.
Cuthbertson, Anthony. “Google’s ai Predicts the Next Sentence of Dead Authors.”
Newsweek, February 29, 2016.
Daston, Lorraine. “The Moral Economy of Science.” Osiris 10 (1995): 2 – 2 4.
Daston, Lorraine. “Probabilistic Expectation and Rationality in Classical Probability
Theory.” Historia Mathematica 7 (1980): 234 – 60.
Daston, Lorraine, and Peter Galison. Objectivity. Cambridge, MA: mit Press, 2007.
de Goede, Marieke. Speculative Security: The Politics of Pursuing Terrorist Monies. Minne-
apolis: University of Minnesota Press, 2012.
de Goede, Marieke. Virtue, Fortune, and Faith: A Genealogy of Finance. Minneapolis:
University of Minnesota Press, 2005.
de Goede, Marieke, and Samuel Randalls. “Precaution, Preemption: Arts and Tech-
nologies of the Actionable Future.” Environment and Planning D: Society and Space
27, no. 5 (2009): 859 – 78.
DeLanda, Manuel. Intensive Science and Virtual Philosophy. London: Continuum, 2002.
DeLanda, Manuel. Philosophy and Simulation. London: Bloomsbury, 2011.
Deleuze, Gilles. Bergsonism. New York: Zone Books, 1991.
Deleuze, Gilles. Essays Critical and Clinical. Translated by Ariel Greco and Daniel
Smith. Minneapolis: University of Minnesota Press, 1998.
Deleuze, Gilles. Negotiations: 1972 – 1990. Translated by Martin Joughin. New York:
Columbia University Press, 1995.
Deleuze, Gilles. “On the Difference between the Ethics and a Morality.” In Spinoza:
Practical Philosophy, translated by Robert Hurley, 17 – 28. San Francisco, CA: City
Lights, 1988.
Deleuze, Gilles. Pure Immanence: Essays on a Life. New York: Zone Books, 2001.
Deleuze, Gilles. Spinoza: Practical Philosophy. Translated by Robert Hurley. San Fran-
cisco, CA: City Lights, 1988.
Derrida, Jacques. Archive Fever: A Freudian Impression. Chicago: University of Chicago
Press, 1995.
Derrida, Jacques. “Cogito and the History of Madness.” In Writing and Difference,
translated by A. Bass, 31 – 63. London: Routledge, 2001.
Derrida, Jacques. The Death Penalty. Vol. 2. Chicago: University of Chicago Press,
2016.
Derrida, Jacques. “Force of Law: The Mystical Foundations of Authority.” In Decon-
struction and the Possibility of Justice, edited by D. Cornell, M. Rosenfeld, and D. G.
Carlson, 26. London: Routledge, 1992.
Derrida, Jacques. The Gift of Death. Chicago: University of Chicago Press, 1996.
Derrida, Jacques. Limited Inc. Evanston, IL: Northwestern University Press, 1988. 201
Derrida, Jacques. Of Grammatology. Translated by Gayatri Spivak. Baltimore, MD:
Johns Hopkins University Press, 1976.
Bibliography
Derrida, Jacques. Writing and Difference. London: Routledge, 2001.
Devlin, Keith. The Math Gene. London: Weidenfeld and Nicolson, 2000.
“dhs foia Documents Baltimore Protests Freddie Gray.” Scribd. Uploaded by Jason
Leopold. Accessed July 10, 2017. https://www.scribd.com/document/274209838
/DHS-FOIA-Documents-Baltimore-Protests-Freddie-Gray.
Diamond, Cora, ed. Wittgenstein’s Lectures on the Foundations of Mathematics, Cambridge,
1939. Chicago: University of Chicago Press, 1976.
Digital Reasoning. Synthesys Mission Analytics. Franklin, TN: Digital Reasoning, 2015.
Dodge, Martin. “Seeing Inside the Cloud: Some Ways to Map the Internet.” Media
Art Net Lectures, Karlsruhe, January 24, 2004.
Dodge, Martin, and Rob Kitchin. Mapping Cyberspace. London: Routledge, 2001.
Domingos, Pedro. The Master Algorithm: How the Quest for the Ultimate Learning Machine
Will Remake Our World. New York: Allen Lane, 2015.
Dunning, Ted, and Ellen Friedman. Practical Machine Learning: A New Look at Anomaly
Detection. Sebastopol, CA: O’Reilly, 2014.
Dürrenmatt, Friedrich. The Physicists. New York: Grove, 2006.
Eklund, Anders, Thomas Nichols, and Hans Knutson. “Cluster Failure: Why fMRI
Inferences for Spatial Extent Have Inflated False-Positive Rates.” Proceedings of
the National Academy of Sciences of the United States of America 113, no. 28 (2016):
7900 – 7905.
Elden, Stuart. “Land, Terrain, Territory.” Progress in Human Geography 34, no. 6 (2010):
799 – 817.
Elden, Stuart. Mapping the Present: Heidegger, Foucault and the Project of a Spatial History.
London: Continuum, 2001.
Elden, Stuart. “Secure the Volume: Vertical Geopolitics and the Depths of Power.”
Political Geography 34, no. 1 (2013): 35 – 51.
Emerging Technology from the arXiv. “Twitter Data Mining Reveals the Origins of
Support for Islamic State.” mit Technology Review, March 23, 2015. https://www
.technologyreview.com/s/536061/twitter-data-mining-reveals-the-origins-of
-support-for-islamic-state/.
Erickson, Paul J., Judy Klein, Lorraine Daston, Rebecca Lemov, Thomas Sturm, and
Michael D. Gordin, eds. How Reason Almost Lost Its Mind: The Strange Career of Cold
War Reasoning. Chicago: University of Chicago Press, 2013.
Eubanks, Virginia. Automating Inequality: How High-Tech Tools Profile, Police, and Punish
the Poor. New York: Palgrave Macmillan, 2018.
European Commission. “What Does the Commission Mean by Secure Cloud Com-
puting Services in Europe?” Memo 13/898, Brussels, October 15, 2013. http://
europa.eu/rapid/press-release_MEMO-13-898_en.htm.
Fazi, M. Beatrice. Contingent Computation: Abstraction, Experience, and Indeterminacy in
Computational Aesthetics. Lanham, MD: Rowman and Littlefield, 2018.
Feynman, Richard. Surely You’re Joking, Mr. Feynman!: Adventures of a Curious Character.
202 New York: Norton, 1985.
Feynman, Richard. What Do You Care What Other People Think?: Further Adventures of a
Curious Character. New York: Norton, 1988.
Bibliography
Bibliography
Friedberg, Anne. The Virtual Window: From Alberti to Microsoft. Cambridge, MA: mit
Press, 2006.
Fuller, Matthew, and Andrew Goffey. Evil Media. Cambridge, MA: mit Press, 2012.
Galison, Peter. Image and Logic: A Material Culture of Microphysics. Chicago: University
of Chicago Press, 1997.
Galloway, Alexander. The Interface Effect. Cambridge: Polity, 2012.
Gentner, W., H. Maier-Leibnitz, and W. Bothe. An Atlas of Typical Expansion Chamber
Photographs. London: Pergamon Press, 1954.
Gibbs, Samuel. “Microsoft’s Racist Chatbot Returns with a Drug-Smoking Twitter
Meltdown.” Guardian (London), March 30, 2016.
Gillespie, Tarleton. “Algorithm.” In Digital Keywords: A Vocabulary of Information Society
and Culture, edited by Benjamin Peters, 18 – 30. Princeton, NJ: Princeton Univer-
sity Press, 2016.
Graham, Stephen. Cities under Siege. London: Verso, 2011.
Graham, Stephen. Vertical: The City from Above and Below. London: Verso, 2016.
Greenfield, Patrick. “The Cambridge Analytica Files: The Story So Far.” Guardian
(London), March 25, 2018.
Greenwald, Glenn. No Place to Hide: Edward Snowden, the nsa, and the Surveillance State.
New York: Random House, 2014.
Gregory, Derek. “Drone Geographies.” Radical Philosophy 183 (2014): 7 – 19.
Grothoff, Christian, and Jens Porup. “The nsa’s skynet Program May Be Killing
Thousands of Innocent People.” Ars Technica, February 16, 2016. http://
arstechnica.co.uk/security/2016/02/the-nsas-skynet-program-may-be-killing
-thousands-of-innocent-people.
Grusin, Richard. Premediation: Affect and Mediality after 9/11. London: Palgrave Mac-
millan, 2010.
Guizzo, Erico. “Google Wants Robots to Acquire New Skills by Learning from Each
Other.” ieee Spectrum, October 5, 2016.
Halpern, Orit. Beautiful Data: A History of Vision and Reason since 1945. Durham, NC:
Duke University Press, 2014.
Halpern, Paul. The Quantum Labyrinth: How Richard Feynman and John Wheeler Revolu-
tionized Time and Reality. New York: Basic Books, 2017.
Hansen, Mark. New Philosophy for New Media. Cambridge, MA: mit Press, 2006.
Haraway, Donna. Modest_Witness@Second_Millennium. FemaleMan©_Meets_ Onco-
mouse. New York: Routledge, 1997.
Haraway, Donna. “Situated Knowledges: The Science Question in Feminism and the
Privilege of Partial Perspective.” Feminist Studies 14, no. 3 (1988): 575 – 99.
Haraway, Donna. Staying with the Trouble: Making Kin in the Chthulucene. Durham, NC:
Duke University Press, 2016.
Harding, Luke. The Snowden Files. London: Faber and Faber, 2014.
Hayes, Brian. “Cloud Computing.” Communications of the acm 51, no. 7 (2008): 9 – 11.
204 Hayles, N. Katherine. “How We Became Posthuman: Ten Years on, an Interview
with N. Katherine Hayles.” Paragraph 33, no. 3 (2010): 318 – 30.
Hayles, N. Katherine. How We Became Posthuman: Virtual Bodies in Cybernetics, Litera-
Bibliography
Bibliography
and Other Oddities.” In Robots and Art, edited by Damith Herath and Christian
Kroos, 149 – 75. New York: Springer, 2016.
Josephson, J., and S. Josephson. Abductive Inference: Computation, Philosophy, Technology.
Cambridge: Cambridge University Press, 1996.
Juola, Patrick. “How a Computer Program Helped Show J. K. Rowling Wrote a Cuck-
oo’s Calling.” Scientific American, August 20, 2013.
Kant, Immanuel. Critique of Pure Reason. Cambridge: Cambridge University Press,
1998.
Kant, Immanuel. Groundwork of the Metaphysics of Morals. Translated by H. J. Patton.
New York: Harper, 1964.
Keenan, Thomas. Fables of Responsibility: Aberrations and Predicaments in Ethics and Poli-
tics. Stanford, CA: Stanford University Press, 1997.
Kinsley, Sam. “The Matter of Virtual Geographies.” Progress in Human Geography 38,
no. 3 (2014): 364 – 84.
Kirby, Vicki. Quantum Anthropologies: Life at Large. Durham, NC: Duke University
Press, 2011.
Kirchner, Lauren. “Federal Judge Unseals New York Crime Lab’s Software for Ana-
lyzing dna Evidence.” ProPublica, October 20, 2017.
Kirchner, Lauren. “New York City Moves to Create Accountability for Algorithms.”
Ars Technica, December 19, 2017.
Kirchner, Lauren. “Traces of Crime: How New York’s dna Techniques Became
Tainted.” New York Times, September 4, 2017.
Kitchin, Rob. The Data Revolution: Big Data, Open Data, Data Infrastructures and Their
Consequences. London: Sage, 2014.
Kittler, Friedrich. Discourse Networks 1800/1900. Stanford, CA: Stanford University
Press, 1990.
Kittler, Friedrich. Literature, Media, Information Systems. Amsterdam: G + B Arts Inter-
national, 1997.
Koehn, Philipp. “Europarl: A Parallel Corpus for Statistical Machine Translation.”
Machine Translation Summit, Phuket, Thailand, September 12 – 16, 2005.
Konkel, F. “The cia’s Deal with Amazon.” Atlantic, July 17, 2014.
Kourou, Konstantina, Themis P. Exarchos, Konstantinos P. Exarchos, Michalis V.
Karamouzis, and Dimitrios I. Fotiadis. “Machine Learning Applications in Can-
cer Prognosis and Prediction.” Computational and Structural Biotechnology Journal 13
(2015): 8 – 17.
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey Hinton. “ImageNet Classification
with Deep Convolutional Neural Networks.” Advances in Neural Information Pro-
cessing Systems 2 (2012): 1097 – 1105.
Kurzweil, Ray. The Singularity Is Near: When Humans Transcend Biology. London: Viking,
2005.
Lange, Ann-Christina, Marc Lenglet, and Robert Seyfert. “Cultures of High-
206 Frequency Trading: Mapping the Landscape of Algorithmic Developments in
Contemporary Financial Markets.” Economy and Society 45 (2016): 149 – 65.
Latour, Bruno, and Peter Weibel. Making Things Public: Atmospheres of Democracy. Cam-
Bibliography
Bibliography
Mayer-Schönberger, Viktor, and Kenneth Cukier. Big Data: A Revolution That Will
Change How We Live, Work, and Think. New York: Houghton Mifflin Harcourt, 2013.
McCarthy, John. “Architects of the Information Society: Thirty-Five Years of the
Laboratory for Computer Science at mit.” Lecture at the mit Centennial, July
1961.
McCormack, Derek. “Remotely Sensing Affective Afterlives: The Spectral Geogra-
phies of Material Remains.” Annals of the Association of American Geographers 100,
no. 3 (2010): 640 – 54.
McGeoch, Catherine C. A Guide to Experimental Algorithmics. Cambridge: Cambridge
University Press, 2012.
Mehotra, Kartikay. “Maker of Surgical Robot Hid More than 700 Injury Claims, In-
surer Says.” Bloomberg, May 26, 2016.
“Memorandum in Support of Application by ProPublica for Leave to Intervene, Lift
the Protective Order and Unseal Judicial Records.” Filed September 25, 2017. Yale
Law School website, https://law.yale.edu/system/files/area/clinic/document/139
._memorandum_of_law.pdf.
Metcalf, Jacob, Emily F. Keller, and danah boyd. “Perspectives on Big Data, Ethics,
and Society.” Report for the Council for Big Data, Ethics, and Society, May 23,
2016. https://bdes.datasociety.net/wp-content/uploads/2016/05/Perspectives-on
-Big-Data.pdf.
Misra, C., P. K. Swain, and J. K. Mantri. “Text Extraction and Recognition from Im-
age Using Neural Network.” International Journal of Computer Applications 40, no. 2
(2012): 13 – 19.
Mitchell, David. Cloud Atlas. London: Hodder and Stoughton, 2004.
Mitchell, W. J. T. “There Are No Visual Media.” Journal of Visual Culture 4, no. 2
(2005): 257 – 66.
Mol, Anne Marie. The Body Multiple: Ontology in Medical Practice. Durham, NC: Duke
University Press, 2002.
“Monitoring Protests from Unstructured Text.” 29:54, YouTube, 2012. https://www
.youtube.com/watch?v=6Sbny91NjeA.
National Research Council of the National Academies. Bulk Collection of Signals Intelli-
gence: Technical Options. Washington, DC: National Academies Press, 2015.
Nature Editors. “More Accountability for Big Data Algorithms.” Nature 537, no. 7621
(2016): 449. https://www.nature.com/news/more-accountability-for-big-data
-algorithms-1.20653.
Newman, M. H. A., A. Turing, G. Jefferson, and R. B. Braithwaite. “Can Automatic
Calculating Machines Be Said to Think?” Transcript of discussion, BBC, recorded
1951, aired January 14, 23, 1952. http://www.turingarchive.org/browse.php/b/6.
Nissenbaum, Helen. Privacy in Context: Technology, Policy, and the Integrity of Social Life.
Stanford, CA: Stanford University Press, 2009.
Noble, Safiya. Algorithms of Oppression: How Search Engines Reinforce Racism. New York:
208 nyu Press, 2018.
nsf Award Abstract 1734521. Last amended October 27, 2017. https://www.nsf.gov
/awardsearch/showAward?AWD_ID=1734521&HistoricalAwards=false.
Bibliography
Nyholm, Sven, and Jilles Snids. “The Ethics of Accident-A lgorithms for Self-Driving
Cars.” Ethical Theory and Moral Practice 19, no. 5 (2016): 1275 – 89.
O’Brien, John. The Definitive Guide to the Data Lake. Boulder, CO: Radiant Advisors, 2015.
Office of the Director of National Intelligence. “icite Fact Sheet.” Washington, DC:
odni, 2012.
Office of the Director of National Intelligence. Intelligence Community Information
Technology Enterprise Strategy 2016 – 2020. Washington, DC: odni, 2015.
Ohri, A. R for Cloud Computing: An Approach for Data Scientists. London: Springer, 2014.
O’Neil, Cathy. Weapons of Math Destruction: How Big Data Increases Inequality and
Threatens Democracy. New York: Crown, 2016.
Ormandy, Roman. “Google and Elon Musk to Decide What Is Good for Humanity.”
Wired, January 26, 2015.
Paglen, Trevor. Invisible: Covert Operations and Classified Landscapes. New York: Aper-
ture, 2010.
Paglen, Trevor. “Trevor Paglen Offers a Glimpse of America’s Vast Surveillance Infra-
structure.” American Suburb X, December 30, 2014. http://www.american
suburbx.com/2014/12/trevor-paglen-offers-a-glimpse-of-americas-vast-surveillance
-infrastructure.html.
Parikka, Jussi. A Geology of Media. Minneapolis: University of Minnesota Press, 2015.
Parikka, Jussi. Insect Media: An Archaeology of Animals and Technology. Minneapolis: Uni-
versity of Minnesota Press, 2010.
Parisi, Luciana. Contagious Architecture: Computation, Aesthetics, and Space. Cambridge,
MA: mit Press, 2013.
Parkin, Simon. “Teaching Robots Right from Wrong.” Economist 1843 Magazine, June/
July 2017. https://www.1843magazine.com/features/teaching-robots-right
-from-wrong.
Parks, Lisa, and Caren Kaplan. Life in the Age of Drone Warfare. Durham, NC: Duke
University Press, 2017.
Pasquale, Frank. The Black Box Society: The Secret Algorithms That Control Money and In-
formation. Cambridge, MA: Harvard University Press, 2015.
Peters, John Durham. The Marvelous Clouds: Toward a Philosophy of Elemental Media.
Chicago: University of Chicago Press, 2015.
Phys.org. “UMass Lowell Professor Steers Ethical Debate on Self-Driving Cars.”
UMass Lowell News, October 5, 2017. https://www.uml.edu/News/news-articles
/2017/phys-self-driving-ethics.aspx.
Pickett, Marc, Chris Tar, and Brian Strope. “On the Personalities of Dead
Authors.” Google ai Blog, February 24, 2016. https://ai.googleblog.com/2016/02
/on-personalities-of-dead-authors.html.
Pishko, Jessica. “The Impenetrable Program Transforming How Courts Treat dna
Evidence.” Wired, November 29, 2017.
Popescu, M. C., and N. Mastorakis. “Simulation of da Vinci Surgical Robot Using 209
Mobotsim Program.” International Journal of Biology and Biomedical Engineering 4,
no. 2 (2008): 137 – 46.
Bibliography
Prentice, Rachel. Bodies in Formation: An Ethnography of Anatomy and Surgery Education.
Durham, NC: Duke University Press, 2013.
Puar, Jasbir. Terrorist Assemblages: Homonationalism in Queer Times. Durham, NC: Duke
University Press, 2007.
Quinlan, J. R. “Induction of Decision Trees.” Machine Learning 1, no. 1 (1986): 81 – 106.
Reiley, Carol, Erion Plaku, and Gregory Hager. “Motion Generation of Robotic Surgi-
cal Tasks: Learning from Expert Demonstrations.” Paper presented at the Annual
International Conference of the ieee Engineering in Medicine and Biology, Bue-
nos Aires, Argentina, August 31 – September 4, 2010.
Rochester, G. D., and J. G. Wilson. Cloud Chamber Photographs of the Cosmic Radiation.
London: Pergamon Press, 1952.
Rose, Gillian. “Rethinking the Geographies of Cultural ‘Objects’ through Digital
Technologies: Interface, Network, Friction.” Progress in Human Geography 40, no. 3
(2015): 334 – 51.
Rotman, Brian. Becoming Beside Ourselves: The Alphabet, Ghosts, and Distributed Human
Being. Durham, NC: Duke University Press, 2008.
Said, Edward. On Late Style: Music and Literature against the Grain. London: Blooms-
bury, 2007.
Schneier, Bruce. “nsa Robots Are Collecting Your Data, Too, and They’re Getting
Away with It.” Guardian (London), February 27, 2014.
Scorer, Richard, and Harry Wexler. Cloud Studies in Colour. Oxford: Pergamon Press,
1967.
Seaver, Nick. “Algorithms as Culture: Some Tactics for the Ethnography of Algorith-
mic Systems.” Big Data and Society 3, no. 2 (2017): 2.
Shane, Scott, and Daisuke Wakabayashi. “The Business of War: Google Employees
Protest Work for the Pentagon.” New York Times, April 4, 2018.
Sharkey, Noel. “Staying in the Loop: Human Supervisory Control of Weapons.” In
Autonomous Weapons Systems: Law, Ethics, Policy, edited by Nehal Bhuta, Susanne
Beck, Robin Geiss, Hin-Yan Liu, and Claus Kress, 23 – 38. Cambridge: Cambridge
University Press, 2016.
Shaw, Ian, and Majed Akhter. “The Dronification of State Violence.” Critical Asian
Studies 46, no. 2 (2014): 211 – 3 4.
Simonyan, Karen, and Andrew Zisserman. “Very Deep Convolutional Networks for
Large-Scale Image Recognition.” arXiv:1409.1556 (September 2014). https://arxiv
.org/abs/1409.1556v1.
Stengers, Isabelle. Cosmopolitics I. Minneapolis: University of Minnesota Press, 2010.
Stengers, Isabelle. Thinking with Whitehead: A Free and Wild Creation of Concepts. Cam-
bridge, MA: Harvard University Press, 2011.
Strawser, B. J. Killing by Remote Control: The Ethics of an Unmanned Military. Oxford: Ox-
ford University Press, 2013.
Stubb, Alexander. “Why Democracies Should Care Who Codes the Algorithms.” Fi-
210 nancial Times, June 2, 2017.
Suchman, Lucy, and Jutta Weber. “Human-Machine Autonomies.” In Autonomous
Weapons Systems: Law, Ethics, Policy, edited by N. Bhuta, Susanne Beck, Robin
Bibliography
Geiss, Hin-Yan Liu, and Claus Kress, 75 – 102. Cambridge: Cambridge University
Press, 2016.
Taylor, Duncan A., Jo-A nne Bright, and John Buckleton. “A Source of Error: Com-
puter Code, Criminal Defendants, and the Constitution.” Frontiers in Genetics 8
(2017): 33.
Timmer, John. “Software Faults Raise Questions about the Validity of Brain Studies.”
Ars Technica, July 1, 2016.
Totaro, Paulo, and Domenico Ninno. “The Concept of Algorithm as an Interpreta-
tive Key of Modern Rationality.” Theory, Culture and Society 31, no. 4 (2014): 32.
Truvé, Staffan. “Building Threat Analyst Centaurs Using Artificial Intelligence.”
Recorded Future, January 14, 2016. https://www.recordedfuture.com/artificial
-threat-intelligence/.
Tucker, Patrick. “A New ai Learns through Observation Alone.” Defense One, Sep-
tember 6, 2016.
Turing, Alan. “Letters on Logic to Max Newman.” In The Essential Turing, edited by
B. Jack Copeland, 205 – 16. Oxford: Oxford University Press, 2004.
Turing, Alan. “Systems of Logic Based on Ordinals.” Proceedings of the London Math-
ematical Society 2, no. 45 (1939): 161 – 228.
Turkle, Sherry. The Second Self: Computers and the Human Spirit. Cambridge, MA:
mit Press, 2005.
van den Berg, J., S. Miller, D. Duckworth, H. Hu, A. Wan, X. Fu, K. Goldberg, and
P. Abbeel. “Superhuman Performance of Surgical Tasks by Robots Using Iterative
Learning.” Paper presented at the ieee International Conference on Robotics and
Automation, Anchorage, Alaska, May 3 – 7, 2010.
Vaughan, Diane. The Challenger Launch Decision: Risky Technology, Culture and Deviance
at nasa. Cambridge, MA: mit Press, 1996.
Vinyals, Oriol, Alexander Toshev, Samy Bengio, and Dumitru Erhan. “Show and Tell:
A Neural Image Caption Generator.” ieee Transactions on Pattern Analysis and Ma-
chine Intelligence 39, no. 4 (2017): 2048 – 57.
Walker, James. “Researchers Shut Down ai That Invented Its Own Language.”
Digital Journal, July 21, 2017. http://www.digitaljournal.com/tech-and-science
/technology/a-step-closer-to-skynet-ai-invents-a-language-humans-can-t-read
/article/498142.
Weber, Jutta. “Keep Adding: On Kill Lists, Drone Warfare, and the Politics of Data-
bases.” Environment and Planning D: Society and Space 34, no. 1 (2016).
Weber, Samuel. Targets of Opportunity: On the Militarization of Thinking. New York:
Fordham University Press, 2005.
“Web Intelligence.” Recorded Future. Accessed January 2018. https://www.recorded
future.com/web-intelligence/.
Webster, Katharine. “Philosophy Professor Wins nsf Grant on Ethics of Self Driving
Cars.” UMass Lowell News, August 8, 2017. https://www.uml.edu/news/stories/2017
/selfdrivingcars.aspx. 211
Weizman, Eyal. “The Politics of Verticality: The West Bank as an Architectural
Construction.” In Territories: Islands, Camps, and Other States of Utopia, edited by
Bibliography
A. Franke, 106 – 2 4. Cologne: Walther Koenig, 2004.
Wiener, Norbert. “A Scientist Rebels.” Atlantic Monthly, January 1947.
Wilcox, Lauren. “Embodying Algorithmic War: Gender, Race, and the Posthuman in
Drone Warfare.” Security Dialogue 48, no. 1 (2017): 11 – 28.
Wilson, C. T. R. “On the Cloud Method of Making Visible Ions and the Tracks of
Ionizing Particles.” In Nobel Lectures in Physics, 1922 – 1941, 167 – 217. Amsterdam:
Elsevier, 1927.
Wilson, J. G. The Principles of Cloud Chamber Technique. Cambridge: Cambridge Uni-
versity Press, 1951.
Winslett, Marianne. “Rakesh Agrawal Speaks Out.” Interview with Rakesh Agrawal.
sigmod Record 32, no. 3 (2003). https://sigmod.org/publications/interviews/pdf
/D15.rakesh-final-final.pdf.
Wittgenstein, Ludwig. On Certainty. Edited by G. E. M. Anscombe and G. H. von
Wright. Oxford: Blackwell, 1975.
Wittgenstein, Ludwig. Philosophical Investigations. Translated by G. E. M. Anscombe,
P. M. S. Hacker, and Joachim Schulter. Oxford: Wiley Blackwell, 2008.
Woodman, Spencer. “Palantir Provides the Engine for Donald Trump’s Deportation
Regime.” Intercept, March 2, 2017.
Xu, K., J. Lei Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and
Y. Bengio. “Show, Attend and Tell: Neural Image Caption Generation and Visual
Attention.” Paper presented at the 32nd International Conference on Machine
Learning, Lille, France, July 6 – 11, 2015.
Yang, A. Y., X. Xhou, A. G. Balasubramanian, S. S. Sastry, and Y. Ma. “Fast 𝓁1-
Minimization Algorithms for Robust Facial Recognition.” Transactions on Image
Processing 22, no. 8 (2013): 3234 – 46.
Zhang, Q., L. Cheng, and R. Boutaba. “Cloud Computing: State-of-the-A rt and Chal-
lenges.” Journal of Internet Services and Applications 1 (2010): 7 – 18.
IN DE X
Index
Challenger (space shuttle), 140 – 53 161
cities, 1 – 4 , 20, 85, 155 counterterrorism. See terrorism
classification, 33, 40, 49, 92, 115 – 19; and Crary, Jonathan, 15, 39, 162
classifiers, 9, 11, 14, 68, 70, 125, 139; credit scoring, 18
and clouds, 29 criminal justice, 11, 85, 95, 104, 137
cloud atlas, 44, 48 – 49 Critchley, Simon, 100, 106, 171
Cloud Atlas (Mitchell), 55 crowd: algorithms to assess risk in, 4, 17,
cloud chamber, 29 – 31, 44, 48, 50, 52 – 53 155, 159 – 60; crowded court, 25, 69
cloud computing, 7, 30 – 31, 33, 35, 38, cybernetics, 111 – 15, 140
40 – 4 2, 46 – 47
cloud data: algorithmic analysis of, 7, Daston, Lorraine, 125
42, 49, 62, 79; architectures of, 21, 70, data: and algorithms, 7, 18, 49, 50, 86,
72, 76 – 77; medical images as, 59; stor- 88, 92, 96, 134 – 35, 138, 141 – 53; and fea-
age of, 37 tures, 12, 16, 20, 43, 155, 160; sharing
Cloudera, 33 of, 33, 42, 50; social media and, 1 – 2,
cloud ethics: and doubt, 135, 137 – 38; and 37, 42, 47, 89; storage of, 36, 38; un
encoded ethics, 121, 123; and partial labeled, 90. See also big data
account, 141; and political claims, data analysis, 31, 33
80 – 81, 146, 148; theory of, 7 – 8, 17, 19, data centers, 33 – 35, 38 – 39
88, 117, 154 – 72. See also ethics data environment, 58, 75
cloud forms, 40 data inputs, 11, 48, 58, 90, 105, 125, 127, 160
cloud imaginations, 36 data lake, 47
cloud robotics, 76 data mining, 36, 179n63
clustering: in anomaly detection, 40, data residue, 46, 59, 64, 164
42 – 4 4, 92, 122, 136; of data for protest da Vinci robot, 59, 63 – 64
detection, 1 – 4 , 169; in elections, 161; decision procedures, 42, 57, 69, 136, 139
and false positives, 128; mapping of decisions: and automation, 11, 57, 67, 85,
features via, 58, 70, 155; in natural lan- 102; doubt and, 134, 137, 150; human,
guage processing, 89 – 9 0; and predic- 22, 62, 65 – 66, 140 – 48; and machine
tions, 138, 173n9 learning, 12, 71, 80, 87, 92, 106; mad-
code, 1, 7, 9, 40, 59; source, 18, 85 – 88, ness of, 112 – 1 4, 119 – 23; and undecid-
91 – 92, 95 – 96, 99, 104, 186n34. See also ability, 19, 20, 88, 98 – 99, 128, 151 – 53,
writing 162 – 65
decision trees, 10, 70, 89 112 – 113, 125, 165. 88, 90, 105, 111, 118, 135, 147 – 53, 154 – 72;
See also random forests and opacity, 5, 14. See also cloud ethics
deconstruction, 88, 99, 104 European Union, 36
deep learning. See machine learning; experimentation, 12, 30, 40 – 4 1, 43 – 4 4,
neural networks 48, 53, 67, 90, 161
Deleuze, Gilles: on duration, 17, 163; on expertise, 59 – 60, 96
ethics, 165; on fabulation, 98, 102, 158, explainability, 18
161 – 62 extraction. See feature extraction
214 democracy, 4, 38, 118, 160
derivatives, 50, 106, 124 – 25 fabulation, 98 – 99, 102 – 5, 158, 160
Derrida, Jacques: on archives, 51; on the Facebook, 37, 50, 89, 138, 160
Index
author, 99; on calculation, 128; on de- facial recognition, 14, 48, 68 – 70, 135 – 36.
cision, 121, 149, 152; on ethics, 166, 169; See also recognition
on madness, 110, 116; on nonclosure, false positive, 122, 145
104, 154, 156; on text, 104 fascism, 112
DexNet, 77 feature extraction, 14, 16, 58, 160; pro-
the digital, 35 cesses of, 42 – 43, 59, 63, 89 – 91, 139, 154
digital subjects, 94 feature space, 58, 74, 79, 155, 160, 169,
discrimination, 8, 38, 85 1 81n8, 182n24
dna, 95 – 97, 105 feature vectors, 59, 72, 155, 160, 167, 169,
doubt, 9, 10, 13, 133 – 53 171, 181n10
drones, 33, 47, 78, 80, 124 – 26, 176n55; feedback loop, 66, 97, 106
image recognition systems and, 16, feminism, 88, 19, 141
38, 111, 146, 156, 160; targeting of, 55, Feynman, Richard, 133 – 53
127, 128 finance. See economy
forensics, 95
economy, 35, 43, 46, 49, 89, 108, 123 forking pathways, 88, 98 – 99, 118, 139,
embodiment, 57, 59, 62, 64, 137 – 38, 143 141, 153, 161 – 62, 165
entanglement, 16, 57, 59 Foucault, Michel: on authorship, 18,
error and errancy, 47 – 48, 108 – 12, 115, 86 – 87; on ethics, 7, 10, 80 – 81, 171; on
116, 120 – 23, 136, 146 – 47; experimen- madness, 115 – 18; on parrhesia, 145 – 47;
tation and, 74 – 75, 96, 118, 125, 159, on probable events, 44; on truth-
169; in robot surgery, 58; sources of, telling, 5, 122
20, 86 Fowles, John, 85, 88, 98 – 99, 101
ethicopolitics: of algorithms, 7 – 8, 10 – 11, fraud, 46, 53, 68, 91, 124
47, 53, 58, 71, 87, 88, 110, 146, 156, 165, the future: algorithms and, 7, 17, 46,
171 – 72; bias and, 75, 79; materiality 49 – 5 1, 122; anticipation of, 72, 73;
and, 36; theories of, 19 – 20, 33, 65 – 66, inferring, 88 – 9 1; optimization of,
69, 121; writing and, 22, 92, 105. See 62, 80, 147; past data and, 20, 43,
also politics 46, 138; political claims on, 4, 54,
ethics: codes of, 5 – 7, 15, 19, 86, 95, 110, 152, 153, 161
119, 120 – 21, 128, 146; of deconstruc-
tion, 104, 106; fabulation and, 98 – 99; Galison, Peter, 30, 38, 40 – 4 1, 44, 49
machine learning and, 55 – 56, 66, 80, gender, 85
geography: of cloud computing, 34 – 36, incompleteness, 21, 43, 80, 96
38, 40; of data, 33 the incomputible, 64, 167
geopolitics, 33, 36 – 37, 114, 137 indeterminacy, 14, 87, 96, 156
gestures, 59, 60, 62, 118 inference, 43 – 4 4, 74, 89, 155; causality
Github, 97 and, 47, 107, 122; of intent, 7, 91, 127;
good: appearance of, 78 – 79, enough, 67, of motion, 46
69, 75, 161; and evil, 58, 109 – 10, 112, infrastructure, 30, 33, 35, 39
134, 146, 158 insurance, 138
Google, 17, 35, 77, 89 – 9 0, 92, 146, 154 intelligence: geopolitical, 15, 31, 33, 54; 215
government, 1, 33, 36, 42 – 43, 49, 113, surveillance and, 37 – 39, 42, 47, 126,
140, 149 148
Index
grammar, 13 Intelligence Community Information
ground truth, 4, 133 – 53, 169. See also Technology Enterprise (icite), 31, 42,
truth 49, 54
internet, 30, 37, 50
Halpern, Orit, 15, 50, 54, 114 intuition, 13, 57, 64, 67, 145, 150, 166
Haraway, Donna: on partiality, 20, 135, iteration, 11, 43, 92, 96 – 97, 139; and
137, 143, 166; on staying with the trou- learning, 60, 62, 68, 106, 138
ble, 19, 40, 66, 178n35, 182n31; on tech-
noscience, 133 – 3 4 Keenan, Thomas, 19, 148 – 49, 152, 165
Hayles, N. Katherine, 137; on cognition, knowledge discovery, 47
19, 43; on computation, 4, 173n8; on
data, 49; on the posthuman, 65 – 66 language, 9, 43, 89. See also natural lan-
health, 123 guage processing
Hinton, Geoffrey, 74 latency, 90
Homeland Security, US Department of law, 37 – 38, 66, 95 – 96
(dhs), 2, 31, 43, 50 layers, 13, 35, 62, 85; hidden, 71, 162
human, humans, 57, 60, 62, 64, 79, learning, 8 – 9, 56, 59, 90, 122. See also
88, 94; agency, 65, 112, 135; and algo- machine learning
rithm relations, 9, 19 – 20, 42 – 43, 57, likelihood. See probability
63 – 64, 92, 139; labeling of images, literature: authorship and, 87, 98 – 99,
17, 72, 136, 148; in the loop, 11, 65 – 67, 102; as training data for machine
109, 127, 152; subjects, 5, 118; visibil- learning, 89, 92 – 93
ity, 14, 30, 38. See also posthuman, Los Alamos, New Mexico, 112, 133, 140
posthumanism
machine learning: anomaly detection
icite, 31, 42, 49, 54 and, 42 – 43, 53; deep, 89, 156 – 57; logics
ImageNet, 17, 73, 173n7 of, 14, 40, 87, 95 – 96, 118, 154, 157, 158,
image recognition, 14, 17, 70 – 71, 73, 121, 160, 173n9; materiality and, 46 – 47;
137. See also recognition processes of, 8 – 9, 11, 20, 54, 56 – 81,
images, 30, 44, 48, 91, 155; drone, 16, 38; 133 – 53, 173n7, 181n10; semi-supervised,
medical, 62 – 63, 122 – 23, 160 68, 124; and society, 5, 12, 109, 169,
incalculability, 92, 110, 120, 141, 143, 155, 170; unsupervised, 68 – 69, 111
162 Mackenzie, Adrian, 12
madness, 108 – 29 74 – 75, 80, 96, 97, 102, 104, 107, 111,
Mantel, Hilary, 102, 107 123, 128, 145, 155 – 56, 160, 162; decision
matching, 90 and, 7; and optimization, 10, 12 – 13, 17,
mathematics, 9, 13, 42, 56, 62, 94, 112, 43, 54, 60, 64, 147; and target, 9, 43,
140 67 – 68, 90, 105, 125, 139; and truth, 14,
Mechanical Turk, 72, 136, 148 114, 136 – 37. See also targets
media, 7, 15, 24, 59, 111, 178n38
Microsoft, 108 – 9 Palantir, 4, 139, 147
216 migration, 129 parameters, 48, 62, 68, 80, 111, 121, 126,
mistakes. See accidents; error and 161
errancy Parisi, Luciana, 12, 47, 147
Index
Index
profiles, 69 – 70, 76, 95 – 96 and uncertainty, 47, 134
prism, 37 sentiment analysis, 54, 147
propensities, 5, 17, 49, 91, 160, 194n15 sight, 15, 29, 31, 38, 40 – 4 1. See also
protest, 1 – 4 , 10, 17, 46, 55, 91, 169 vision
social media: chatbot and, 108 – 9; data,
race and racism, 1 – 4 , 11, 69, 85, 97, 111, 1 – 2, 37, 42 – 43, 47, 50, 51, 71, 160, 170;
128, 145, 175n37 images, 3, 54; sentiment analysis of,
random forests, 112, 124 – 27, 165, 190n51. 147, 174n9; text, 70, 89, 91
See also decision trees Society for Worldwide Interbank Finan-
reason, 42, 44, 86, 108 – 29 cial Telecommunications (swift),
recognition, 1, 3, 14, 17, 33; of authorship, 36
90; Butler’s conceptualization of, software, 35, 42, 97
19, 145; of handwritten text, 69 – 70, source code, 18, 85 – 88, 91 – 92, 95 – 96,
173n4; of patterns, 46, 150, 181n10; 99, 104, 186n34
politics of, 4, 8, 41, 126, 127, 167, sovereignty, 31, 36 – 38, 53 – 54, 128 – 29
171, 184n55; regimes of, 48 – 49, 54, spatiality, 11, 14, 33, 35, 37, 40, 71,
56 – 81, 121, 122, 183n43; semantic, 89, 162 – 63
185–86n12; speech, 106. See also facial spoofing, 121
recognition; image recognition; ob- statistics, 12, 47, 95, 122, 124, 139
ject recognition Stengers, Isabelle, 14, 53 – 54, 165
reduction: algorithmic processes of, surgery, 20, 62, 64
16 – 17, 115, 128 surveillance, 16 – 17, 38, 39
resistance, 51, 145, 157, 169 – 70 swift, 36
responsibility: for algorithmic harms,
100, 112; and code, 86, 94, 95; ethics targets, 43, 124, 139; generation of, 5, 9,
of, 8, 81, 118, 121, 123, 164 – 65; human 16, 67, 90
touch and, 63 – 64, 144; locus of, 5, technology, 15 – 16, 31, 36, 40, 43, 50, 101,
18, 19, 66, 92, 99; of science, 114, 145
133 – 3 4 terrorism, 33, 55, 74, 89, 91, 124, 126, 138,
rights, 5, 10, 38, 148 148
risk, 4, 19, 40, 104, 126, 138, 140, 145 – 48 text, 93, 97, 101, 104
robots, 20, 58 – 59, 63 – 64, 77, 94 threshold, 41, 48, 54, 64, 68, 105, 139
rules, 7, 14, 47, 53, 65, 69, 89, 141, 150 time, 16 – 17
traces: data, 44 – 46, 62, 150 tential for, 17, 111; science and, 134;
training datasets, 4, 9, 17 – 18, 20, 68, 96, sovereign, 53, 128, 164
122, 125, 135 – 36, 150, 155; images used the virtual, 35, 62, 177n15, 181–82n23
as, 49; in natural language process- vision, 14, 39, 44, 53, 159, 162. See also
ing, 90; patterns in, 54; and point sight
clouds, 77 visualization, 14, 30, 38
transparency, 5, 8, 18, 86, 110, 117, 123,
134, 166 war. See violence
218 truth: concept of, 5, 9, 14, 18, 93, 122, weights: adjustment of, 24, 48, 68, 73,
135 – 37, 145 – 46, 148; and falsity, 103, 75, 80, 92, 96, 103, 128, 145; and as-
114. See also ground truth signing value, 129; and indeterminacy,
Index
Turing, Alan, 13, 56 – 57 22, 75, 147; machine learning and, 8,
Twitter, 89, 108 – 9 13 – 1 4, 70 – 71, 75, 79, 106, 111, 128; and
mediation, 162; parameters and, 74,
uncertainty, 12 – 13, 47 – 48, 55; machine 136; politics of, 163 – 64; and rejected
learning and, 21, 22, 64, 88, 95, 111, alternatives, 81, 99
118, 136, 139, 142; political, 80; writing Wiener, Norbert, 111 – 15
and, 98 – 99 Wilson, Charles Thomson Rees, 29 – 31, 44
the unknown, 90, 95, 97, 102, 105 – 7, Wittgenstein, Ludwig, 9, 13
152 – 53, 155 world-making, 12, 20, 30, 33, 40, 43, 48,
US Department of Homeland Security 88, 97, 161
(dhs), 2, 31, 43, 50 writing, 86 – 88, 95 – 98, 101 – 3 , 158. See
also code
variability, 71 – 73, 126, 134
verticality, 17, 40, 46 xenophobia, 119
violence: algorithmic, 4, 11, 46, 66, 118,
126, 146, 152, 161; meaning of, 91; po- YouTube, 109 – 10