Professional Documents
Culture Documents
Now You See It
Now You See It
The value of
information is not realized until\ you use it to do something. Used properly, it can
make the world a better place.
This poem by Edna St. Vincent Millay eloquently and poignantly describes our
situation today. Our problem is not a lack of data, but rather our inability to make
sense and use of what we have.
The amount of information that is available to us has grown much faster than
our ability to make use of it. We lack both skills in data analysis and tools that
can be used to productively support the process.
Most of the data analysis that is needed in the normal course of business requires
relatively simple data visualization techniques, leaving little that requires the
sophisticated techniques of statistical and financial analysis.
If you search for resources that teach data analysis skills, youll find many books
and courses that present the sophisticated techniques needed by the few, but few
resources if any that teach the simple techniques that most of us need to make
sense of business data. The skills that most of us need to infuse our businesses
with needed insights can be learned without a background in statistics, but these
skills dont come naturally they must be learned. You must develop expertise, but
it is expertise that can be easily learned with the proper direction and practice. You
must learn to see particular patterns in data that are meaningful.
People can learn pattern-detection skills, although the ease of gaining these
skills will depend on the specific nature of the patterns involved. Experts do
indeed have special expertise. The radiologist interpreting an X-ray, the
meteorologist interpreting radar, and the statistician interpreting a scatter plot will
each bring a differently tuned visual system to bear on his or her particular
problem. People who work with visualizations must learn the skill of seeing
patterns in data.
(Information Visualization: Perception for Design, Second Edition, Colin Ware,
Morgan Kaufmann Publishers, 2004, page 209)
Good tools, from stones for crushing or cutting to computers for augmenting
cognition, can set us free and make the world a better place, but only when used
properly. When misused, they make us lazy, dumb, slaves. The choice is ours.
To date, business intelligence has mostly focused on technology and project methodology, resulting
in great advances. As a result, we have huge and fast warehouses of information. It is now time to
focus on the true essence of business intelligenceimportant, meaningful, and actionable
informationand the most powerful resources for tapping into its value are those that engage the
tremendous capacities of human perception and intelligence to make sense of and communicate
information.
Many organizations aren't effectively analyzing the data they do have to improve their business.
What's more troubling, perhaps, is many companies that purchase powerful analysis and
business-intelligence tools don't use them effectively. Users of these products often generate the
most basic and obvious reports and never get their hands dirty with the deep-analysis tools.
By ignoring these products' deep-analysis capabilities, organizations could be missing important
trendsinformation that might show, for example, where a company is losing business.
Companies might also see that business decisions that were made based on basic information
were wrong and ended up costing money in the long run.
(eWeek, With Data Analysis, Less Isnt More, Jim Rapoza, Ziff Davis Media Incorporated, June
7, 2004)
Today much of science and engineering takes a machine-centered view of the design of
machines and, for that matter, the understanding of people. As a result, the technology that is
intended to aid human cognition and enjoyment more often interferes and confuses than aids and
clarifies.
It will take extra effort do design systems that complement human processing needs. It will not
always be easy, but it can be done. If people insisted, it would be done. But people dont insist:
Somehow, we have learned to accept the machine-dominated world. If a system is to
accommodate human needs, it has to be designed by people who are sensitive to and
understand human needs. I would have hoped such a statement was an unnecessary truism.
Alas, it is not.
(Things That Make Us Smart, Donald A. Norman, Basic Books, New York, 1993, page s 9 and
227)
Data visualization has the potential to help business intelligence fulfill its promise of
helping organizations function intelligently.
Data visualization is the loom that will weave the data that we collect into the fabric
of understanding. Pictures of data can make visible the meanings that might forever
otherwise remain hidden.
10
11
Today, 220 years later, graphs are commonplace, fully integrated into the fabric
of modern communication. Surprisingly, however, Playfairs innovative efforts
spring from meager precedentare superior to most of the graphs produced
today.
12
13
Hans Rosling of Gapminder.org has become one of the real stars of information
visualization in the last couple of years. When Rosling took the stage at the TED
conference for the first time in 2006, he managed to get people up on the edges of
their seats to watchbelieve it or nota bubble plot that used animation (motion) to
show change through time. When he finished, the crowd rose to their feet to give
Rosling a standing ovation. For most of the people there, data presentation had
never been so compelling.
Rosling has used relatively simple visualization techniques, featuring animated
plots, to tell statistical stories that are compelling, not only because they are told
with great charisma, but because they reveal important truths about the world, such
as the changing relationship between wealth and child mortality. I applaud his
success, in part because it is success that we can share in, for it illustrates to the
world at large what infovis can do when it is done well and it is used for worthwhile
purposes.
14
When Al Gore rode a scissor crane up to the top of the CO2 emissions graph in the
film An Inconvenient Truth, he became a superstar of visual communications. He
compellingly used graphs to tell the story of global warming, which helped public
opinion in America to finally reach the tipping point.
15
Data visualization begins with (1) searching through the data to discover potentially
meaningful facts, then involves (2) examining that data more closely to understand
it, including what caused it to occur, so that you can then (3) explain what youve
learned to those who can use that knowledge to make good decisions. Most of what
we need to recognize and understand in our business data is not all that
complicated.
16
Human perception is amazing. I cherish all five of the senses that connect us to the world, that allow
us to experience beauty and an inexhaustible and diverse wealth of sensation. But of all the senses,
one stands out dramatically as our primary and most powerful channel of input from the world around
us, and that is vision. Approximately 70% of the bodys sense receptors reside in the eye.
Perhaps the worlds top expert in visual perception and how its power can be harnessed for the
effective display of information is Colin Ware, who has convincingly described the importance of data
visualization. He asks:
Why should we be interested in visualization? Because the human visual system is a pattern
seeker of enormous power and subtlety. The eye and the visual cortex of the brain form a
massively parallel processor that provides the highest-bandwidth channel into human cognitive
centers. At higher levels of processing, perception and cognition are closely interrelated, which is
the reason why the words understanding and seeing are synonymous. However, the visual
system has its own rules. We can easily see patterns presented in certain ways, but if they are
presented in other ways, they become invisibleThe more general point is that when data is
presented in certain ways, the patterns can be readily perceived. If we can understand how
perception works, our knowledge can be translated into rules for displaying information.
Following perception-based rules, we can present our data in such a way that the important and
informative patterns stand out. If we disobey the rules, our data will be incomprehensible or
misleading.
(Information Visualization: Perception for Design, Second Edition, Colin Ware, Morgan Kaufmann
Publishers, 2004, page xxi)
Perhaps the best known expert in data visualization, Edward Tufte, says: Clear and precise seeing
becomes as one with clear and precise thinking. (Visual Explanations, Edward R. Tufte, Graphics
Press: Cheshire, CT.1997 page 53)
17
The presentation of data as text, such as you see in this table, is perfect when you
need precise values or when the purpose is to look up or compare individual values,
but not when you wish to see patterns, trends, and exceptions, or to make
comparisons. When this is your goal, visualizations work best.
When data is presented visually, it is given visible form, and from this we can easily
glean insights that would take a long time to piece together from the same data
presented textually, if ever. This graph of the same data that appears in the table
makes brings to light several of the stories contained in the data that werent
obvious before, and it did so instantly.
When] we visualize the data effectively and suddenly, there is what Joseph
Berkson called interocular traumatic impact: a conclusion that hits us between
the eyes.
(Visualizing Data, William S. Cleveland, Hobart Press, 1993, page 12)
Modern data graphics can do much more than simply substitute for small
statistical tables. At their best, graphics are instruments for reasoning about
quantitative information. Often the most effective way to describe, explore, and
summarize a set of numbers even a very large set is to look at pictures of
those numbers. Furthermore, of all methods for analyzing and communicating
statistical information, well-designed data graphics are usually the simplest and
at the same time the most powerful.
(The Visual Display of Quantitative Information, Edward R. Tufte, Graphics
Press: Cheshire, CT 1983, Introduction)
2009 Stephen Few, Perceptual Edge
18
So much of what is called data visualization gives it a bad name and causes
confusion about what it is, how it works, and what can be accomplished when it is
properly done.
19
Software vendors are competing to out dazzle one another with silly visual effects
that treat data visualization like its a video game.
20
This notion of data visualization is not about understanding and communication, its
about bling.
21
Dashboards are notorious for featuring graphical glitz over substance. Too many
dashboard vendors and designers have lost sight of the bottom line: communication.
They emphasize graphical glitz over clear and meaningful content. For every item of
information on the screen the designer should ask the question: How can I display
this information in the most meaningful, clear, and efficient way possible?
The graphics in this dashboard from Business Objects, created with Xcelsius, are
beautifully rendered, but is the information effectively displayed? The Xcelsius team
clearly possesses exceptionable graphical skill. This isnt surprising, given the fact
that most of the original team of developers formerly developed video games.
Unfortunately, they failed to make the transition from video games to data
visualization.
22
Data visualization essentially helps us to do two things: (1) think about information
more effectively so we can understand it means, and then (2) tell its story clearly
and accurately to others.
23
24
Data visualization is much more than just graphical reporting, more than
dashboards. Beyond its use for communicating information that cannot be
communicated with tabular data, its greatest potential is exhibited in its use for
analysis. The best techniques for making sense of business data are visual
techniques, which extend our ability to find and understand meaningful patterns in
data by offloading much of the work traditionally performed by the conscious mind to
preconscious and parallel processors in the brains visual cortex. Most BI vendors
provide some graphical functionality in their software, but few actually support visual
analysis in more than rudimentary ways.
25
During the last year I wrote a new book, Now You See It, to help people develop the
fundamental skills that are needed to make sense of quantitative information.
26
Visual analysis involves comparing the magnitudes of values, but not just one to
another. We must compare many values. To do so, we must see how they relate to
one another to form patterns. We not only compare the magnitudes of values; we
also compare patterns formed by sets of values. We look for how they are similar
and we look for how they are different, especially differences that appear to be
dramatic departures from the norm. When we spot these visual characteristics in the
data, we then interact with the data to find out why these things have happened.
27
28
Many of the ways that visual perception work are not intuitive.
Looking at these two sets of objects, we naturally see those on the left as convex
and on those the right as concave.
29
The effect has now been reversed: we see the objects on the left as concave and
those on the right as convex. All I did, however, was turn each sets of objects
upside downI didnt switch them. The reason that we now see those on the left
as concave is because, through eons of evolution, visual perception learned to
assume that light was shining from above, which causes us to see the objects on
the left as concave, because the shadows are on the top, and those on the right
as convex, because the shadows are on the bottom.
30
31
Despite how differently they look in the original image, squares A and B are
exactly the same color. What we see is not a simple recording of what is actually
out there. Seeing is an active process that involves interpretations by our brains
of data that is sensed by our eyes in an effort to make sense of it in context. The
presence of the cylinder and its shadow in the image of the checkerboard triggers
an adjustment in our minds to perceive the square labeled B as lighter than it
actually is. The illusion is also created by the fact that the sensors in our eyes do
not register actual color but rather the difference in color between something and
whats nearby. The contrast between square A and the light squares that surround
it and square B and the dark squares that surround it cause us to perceive
squares A and B quite differently, even though they are actually the same color,
as you can clearly see above after all of the surrounding context has been
removed.
The ability to use graphs effectively requires a basic understanding of how we
unconsciously interpret what we see.
32
This image illustrates the surprising effect that a simple change in the lightness of
the background alone has on our perception of color. The large rectangle displays
a simple color gradient of a gray-scale from fully light to fully dark. The small
rectangle is the same exact color everywhere it appears, but it doesnt look that
way because our brains perceive visual differences rather than absolute values,
in this case between the color of the small rectangle and the color that
immediately surrounds it.
Among other things, understanding this should tell us that using a color gradient
as the background of a graph should be avoided.
33
There is a distinct image that has been worked into the picture of the rose, which
isnt noticeable unless we know to look for it. Once primed with the image of the
dolphin, however, we can easily spot it in the rose.
34
From this set of six playing cards, select one and remember it. I will now identify and
remove the card that youve selected, then rearrange those that remain. As I
advance to the next slide, youll discover that your card has been eliminated.
35
Amazing. And I can do this again and again. If you go back to the previous slide and
again pick a card, when you return to this slide you will see that Ive once again
eliminated it.
Actually, as Im sure you realize, this card trick is an illusion that makes use of the
limitations of short-term memory. None of the cards on the second screen are the
same as the cards on the first screen, but you probably didnt notice this because
you only remembered the card that you selected, not the others.
36
Memory plays an important role in human cognition. Because memory suffers from
certain limitations, visual analysis tools must be able to augment memory.
The example above illustrates one of the limitations of working memory. We only
remember that to which we attend. Any part of this image that never gets our
attention will not be missed when we shift to another version of the image that lacks
that particular part. If we dont attend to it, we might notice the change from one
version of the image to the next, but only if the transition shift immediately from one
to another, without even a split second of blank space between them.
In addition to not remembering, we also dont clearly see that on which we dont
focus. To see something clearly, we must focus on it, for only a small area of
receptors on the retinas of our eyes are designed for high-resolution vision.
(Source: This demonstration of change blindness was prepared by Ronald A.
Rensink of the University of British Columbia. Several other examples of this visual
phenomenon can be found at
http://www.psych.ubc.ca/%7erensink/flicker/download/index.html.)
37
When we think about things, trying to make sense of them, the place where
information is temporarily stored to support this process is called working
memory. Working memory is a lot like RAM (random access memory) in a
computer in that it is limited in capacity and designed for temporary storage.
Compared to that hard disk drive, which is built into your computer or attached
to it externally, RAM seems very limited, but compared to working memory in
the human brain, RAM seems enormous. Only around three chunks of visual
information can be stored in working memory at any one time. Information that
comes in through our eyes or that is retrieved from long-term memory in the
moment of thought is extremely limited in capacity. If all four storage slots are
occupied, you must let something go to allow something new to come in. When
you release information from working memory, it can take one of two possible
routes on its way out: 1) it can be stored permanently in long-term memory by
means of a rehearsal process that we call memorization, or 2) it can simply be
forgotten.
38
39
Good visual analysis software can help us overcome the limitations of working
memory in several ways. The goal is to enable as many meaningful
comparisons as possible. Good tools can help us increase:
The different views of the information that we can compare (that is,
multiple perspectives)
40
Traditional BI relies mostly on tabular data displays. Tables are wonderful if you
need to look up individual values, compare a single value to another, or know
values precisely, but they dont display patterns or trends. This is a problem,
because data analysis relies heavily on our ability to spot and make sense of
patterns and trends in data. Take a look at the table and compare it to this line
graph, which displays the same data. Relying on the table to discern the ups
and downs of sales through time and to compare the patterns of change from
region to region would yield very little of the information that is obvious in this
graph. Visual representations give form to data, making pattern, trends, and
exceptions easy to see.
Another advantage of properly designed graphs over tables for analytical
purposes is less obvious. If you needed to remember information in the table,
you could hold only about four of the values (that is, four of the monthly sales
numbers) in working memory at any one time. But by relying on the graph, 12
values are combined into each of the four lines to form a pattern that you could
hold entirely as a single chunk in working memory. Simply by giving visual form
to the values, you can hold much more information in memory.
41
You can extend the benefits of data visualization further by arranging several
graphs on the screen at the same time, such as shown in this visual crosstab.
Here you can see 24 small graphs arranged in familiar crosstab fashion to
present sales across four different dimensions at once: products within product
types by row, regions by column, and market size by the color of the line. Not
only does this approach make a great deal of data available to your eyes, it
does so across several dimensions, thus expanding the dimensionality of the
data well beyond traditional graphical displays.
42
43
Theres an old folktale that youve probably all heard about three blind men who encounter an
elephant one day for the first time and do their best to learn about it by touch alone. The
experience of each is unique because each touches a different part of the elephant. This
ancient story, originally from China, can teach us something important today about business
intelligence (BI). According to the original Chinese tale, the first man touches the elephants
ear, the second his legs, and the third his tail. From this point, heres how the story goes:
The three blind men then went their way. Each one was secretly excited over the
experience and had a lot to say, yet all walked rapidly without saying a word.
"Let's sit down and have a discussion about this queer animal," the second blind man said,
breaking the silence.
"A very good idea. Very good." the other two agreed for they also had this in mind. Without
waiting for anyone to be properly seated, the second one blurted out, "This queer animal is
like our straw fans swinging back and forth to give us a breeze. However, it's not so big or
well made. The main portion is rather wispy."
"No, no!" the first blind man shouted in disagreement. "This queer animal resembles two
big trees without any branches."
"You're both wrong." the third man replied. "This queer animal is similar to a snake; it's long
and round, and very strong."
How they argued! Each one insisted that he alone was correct. Of course, there was no
conclusion for not one had thoroughly examined the whole elephant. How can anyone
describe the whole until he has learned the total of the parts.
If I retold this story today to teach a lesson about BI, I might call it Three blind analysts and a
data warehouse. Business people struggle every day to make sense of data, stumbling blindly,
touching only small parts of the information, and coming away with a narrow and fragmented
understanding of what it means.
44
The tabular model forces us to view small slices of information one piece at a
time, which cannot possibly be stitched together in our brains to tell the whole
story.
45
The process of visual data analysis involves several common interactions with data to uncover whats meaningful. Here are
some of the primary interactions:
Sorting. The act of sorting data, especially by the magnitude of the values from high to low or low to high, features the ranking
relationship between those values and makes it easier to compare the magnitude of value to the next.
Adding/removing variables. You might need to view different variable at different times during the analysis process, so it is
common to add or remove field of data from view as necessary
Filtering. When you want to focus on a subset of data, nothing makes it easier to do so than filteringthe removal from view of
everything your not interested in at the moment.
Highlighting. Sometimes you want to focus on a subset of information, but do so in a way that allows you to maintain a sense of
how that subset relates to the whole. Rather than filtering out the data that falls outside your range of focus, you can simply
reduce its visual salience or increase the visual salience of the data you wish to focus on. This allows you to focus on the subset
with less distraction from the whole in a way that allow you to remain aware of the whole. This is one way of achieving whats
called a focus+context view.
Aggregating/Disaggregating. Analysis often requires that you examine data a different levels of detail. Aggregation involves
viewing data at a higher level of summarization. Disaggregation involves viewing data at a lower level of detail.
Drilling. Similar to disaggregation, drilling involves viewing data at a lower level of detail, but in a specific manner. Drilling also
means that you are changing the view to the next level in a defined hierarchy, and excluding from view all data that is not directly
related to the specific data value that you chose to drill into. For instance, if you drill into a particular product family, your next
view only products that belong to that product family. In other words, a form of filtering is involved.
Grouping. Sometimes it is useful to combine members of a variable together, treating them as a single member of the variable.
This may take the form of combining some members and leaving others as they are, or of creating an entirely new variable that
combines all members of an existing variable into a groups to form members of a higher level variable.
Zooming/Panning. When a data visualization contains so much that it is difficult to clearly see all the data at once, it is useful to
zoom in on that portion that you want to see more clearly. Panning involves moving around (for example, up, down, right, or left)
in a zoomed view to focus on a different part of the larger visualization.
Re-visualizing. No one visual representation of data can show you everything there is to see, so visual analysis involves shifting
from one type of visualization to another to explore data from various perspectives.
Re-expressing. Sometimes it is useful to express a quantitative variable as a different unit of measure, such as expressing
dollars as percentages.
Re-scaling. No single quantitative scale on a graph can serve every analytical need. Rescaling involves changing the range of
the quantitative scale to make it easier to see particular patterns and sometimes even changing the nature of the scale, such as
from a normal scale to a logarithmic scale.
46
Direct dynamic interaction with the properly visualized data allows us to see
discover meaningful patterns, trends, and exceptions in the display and to interact
with it directly to filter out what we dont need, drill into details, combine multiple
variables for comparison, etc., in ways that promote a smooth flow between seeing
something, thinking about it, and manipulating it, with no distracting lags in between.
This is what I call visual analysis at the speed of thought.
47
When new recruits by intelligence organizations are trained in spy craft, they are
taught a method of observation that begins by getting an overview of the scene
around them while being sensitive to things that appear abnormal, not quite right,
which they should then focus in on for close observation and analysis.
A visual information-seeking mantra for designers: Overview first, zoom and
filter, then details-on-demand.
(Readings in Information Visualization: Using Vision to Think, Stuart K. Card,
Jock D. Mackinlay, and Ben Shneiderman, Academic Press, San Diego,
California, 1999, page 625)
Having an overview is very important. It reduces search, allows the detection of
overall patterns, and aids the user in choosing the next move. A general heuristic
of visualization design, therefore, is to start with an overview. But it is also
necessary for the user to access details rapidly. One solution is overview +
detail: to provide multiple views, an overview for orientation, and a detailed view
for further work.
(Ibid., page 285)
Users often try to make a good choice by deciding first what they do not want,
i.e. they first try to reduce the data set to a smaller, more manageable size. After
some iterations, it is easier to make the final selection(s) from the reduced data
set. This iterative refinement or progressive querying of data sets is sometimes
known as hierarchical decision-making.
(Ibid., page 295)
2009 Stephen Few, Perceptual Edge
48
Shneidermans technique begins with an overview of the datathe big picture. Let
your eyes search for particular points of interest in the whole.
49
50
Once youve zoomed in on it, you can examine it more closely and in greater detail.
51
Often you must remove data that is extraneous to your investigation to better focus
on the relevant data.
52
Filtering out extraneous data removes distractions from the data under investigation.
53
Visual data analysis relies mostly on the shape of the data to provide needed
insights, but there are still times when you need to see the details behind the shape
of the data. Having a means to easily see the details when you need them, without
having them in the way when you dont works best.
54
Information cannot speak for itself. It needs our help. It relies on us to give it a voice.
When we do, information can tell its story, and will thus become knowledge. The
ultimate goal, however, isnt knowledge; it is wisdom. Knowledge becomes wisdom
when it is used to do something good. Only when we use what we know to make
the world a better place has information served its purpose and we have done our
job.
Our networks are awash in data. A little of it is information. A smidgen of this
shows up as knowledge. Combined with ideas, some of that is actually useful.
Mix in experience, context, compassion, discipline, humor, tolerance, and
humility, and perhaps knowledge becomes wisdom.
Turning Numbers into Knowledge, Jonathan G. Koomey, 2001, Analytics Press:
Oakland, CA page 5, quoting Clifford Stoll.
55
56
57
58
59
60
61
62