When Can You Trust the Experts?: How to Tell Good Science from Bad in Education

Introduction: What Are You to Believe?

Before obtaining certainty we must often be satisfied with a more or less plausible guess.

—George Polya¹

Try this sometime. Ask a friend, Why do you believe what you believe? What sort of evidence persuades you that someone is right or that a product is good? This question seldom elicits a careful, thoughtful response. Rather, it elicits silence and narrowed eyes. Most people think that their beliefs are shaped by logic and reason. Your friend will likely detect a whiff of insult in the question.

But our beliefs are fueled by much more than reason and fact. Yes, we are persuaded by solid evidence assembled into arguments that conform to principles of logic. But that’s true only for the messages that we examine, and we don’t have the time to audit every advertisement we hear and blog posting we read. We are pelted by information almost constantly. Just think of the ubiquity of screens. At airport gates, in restaurants, in waiting rooms, in the post office, even in hotel elevators. If a location provides a captive human audience, there is likely to be a screen, flashing updates from Afghanistan, coverage of a golf tournament, or an advertisement for Claritin. Much of this information is not neutral. It is meant to persuade us of something. Yet we don’t have the time or the mental energy to think through every message that comes our way.

Are we influenced by messages that we ignore? I stand in line at my bank and notice a large television behind the teller, displaying a channel exclusive to my bank. An advertisement appears, showing a sedan wending along a New England country road, scattering autumn leaves. I go into a reverie, thinking of the Berkshire mountains. I haven’t consciously noticed the make of the car . . . but am I nevertheless influenced? When I next need a car, even if it’s four years from now, perhaps I’ll be a bit more likely to buy this model because I was exposed to this ad. Will I be more likely to apply for a car loan at this bank, rather than shopping around for the best rate? Is it possible for attitudes to change outside my awareness? Although it makes us uncomfortable to contemplate it, psychological research from the last fifty years indicates that the answer is yes.

Sometimes, of course, I do pay attention to these messages, and I don’t fully trust what I’m hearing. For example, when I read Mother Jones or the Weekly Standard, I am aware that each has a political point of view, and I try to remember that information may be omitted or the interpretation of facts stretched to be consistent with that view. When I hear the president of Iran give a speech, I recall that he has denied that the Holocaust took place, so I am wary of any claim he makes. When I listen carefully to messages, am I able to account for the bias or trustworthiness of the source? To some extent, yes, but not completely.

I am making it sound as though we all are buffeted about—no, worse, systematically manipulated—by forces that operate outside our awareness or, even if we are aware of them, outside our control. Putting it that way is a bit dramatic, but it’s not far from the truth.

This book will tell you how to evaluate new ideas—in particular, those related to education—so that you are less likely to be persuaded by bad evidence.

The Golden Ratio

Forewarned is forearmed. The first step in defending yourself from hidden persuaders is identifying them. I begin with what is perhaps the strangest example. The very shape that carries information to you has an impact on whether or not you believe this information. This story is a bit complex, although the mathematics behind it is relatively simple.

You and I have a number in common, a number that influences what we consider beautiful and worthy of our sustained attention: 1.618. (Actually, it’s 1.6180339887, but I’ll use the truncated version.) It’s important not as a number but as a ratio, and the simplest way to understand it is to consider the rectangle shown in Figure I.1.

FIGURE I.1: A rectangle with sides proportional to the Golden Ratio.

The ratio of the length of side b to side a is 1.618, and people find rectangles of this proportion more aesthetically pleasing than other rectangles. Confronted with, say, thirty rectangles of various proportions, most people pick this one as the most attractive. Because of its importance in aesthetics, 1.618 is called the Golden Ratio.

Researchers have observed this ratio in classical architecture. For example, the width and height of the façade of the Parthenon in Greece respects the Golden Ratio. It is also observed in the great pyramid of Giza. If one forms a triangle as shown, the ratio of the length of one face to half the length of the base is within 1 percent of the Golden Ratio (Figure I.2).

FIGURE I.2: Classic works of architecture such as the Parthenon (or the reproduction in Nashville, Tennessee, shown here) and the Great Pyramid of Giza have the Golden Ratio embedded in their proportions.

The Golden Ratio is observed in smaller-scale works of art as well, including the placement of figures in paintings by da Vinci and the elements of a Stradivarius violin (Figure I.3).

FIGURE I.3: Iconic works of Western art that show the Golden Ratio in their proportions.

Why would this ratio be aesthetically pleasing across cultures and across centuries? A reasonable suggestion is that it is commonly observed in nature. Indeed, the Golden Ratio is found in proportions of the human body (Figure I.4) and the human face, especially faces that others find attractive.

FIGURE I.4: Ratios of body parts also show the Golden Ratio. See text for description.

If the distance between the navel and the foot is taken as 1 unit, the height of a human being is typically equivalent to 1.618. Some other golden proportions in the average human body are

The distance between the finger tip and the elbow/distance between the wrist and the elbow

The distance between the shoulder line and the top of the head/head length

The distance between the navel and the top of the head/distance between the shoulder line and the top of the head

The distance between the navel and the knee/distance between the knee and the end of the foot

Naturally, there is variation across individuals in these proportions. The Golden Ratio is observed when we take averages across many individuals, and individuals with the ideal proportions are judged by others as having well-proportioned bodies.

The same is true for faces, and here the relationship to attractiveness is easy to appreciate. Faces are attractive not only because the eyes and the mouth are well shaped. The proportions of the face must be right. If a person’s eyes are too close together or too far apart, he or she is not attractive. The actress Jessica Alba, commonly considered to be very attractive, not only has a dazzling smile and beautiful eyes, but the distances between her features match the Golden Ratio perfectly (Figure I.5).

FIGURE I.5: Jessica Alba (a) is commonly considered one of the most beautiful women in Hollywood. These photos show some of the Golden Ratios observed in the proportion of features observed in the ideal human face: (b) distance between pupils / distance between eyebrows; (c) width of mouth / width of nose; and (d) distance between lips and where eyebrows meet / length of nose.

The Golden Ratio is observed elsewhere in nature as a spiral. To understand how, you need a basic understanding of the underlying mathematics. The Golden Ratio was first described by twelfth-century mathematician Leonardo Fibonacci. Perhaps you’ve heard of the Fibonacci sequence: I begin with the numbers 0 and 1, and then add the last two numbers in the sequence to generate the next number. That is, 0 + 1 = 1, so the sequence begins 0, 1, 1. To obtain the next number, I add the final two in the sequence thus far, hence, 1 + 1 = 2. So now the sequence is 0, 1, 1, 2. Continuing, the sequence is: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, and so on. If I take the ratio of successive numbers, the values converge on the Golden Ratio (Table I.1).

TABLE I.1: The ratio of neighboring numbers in the Fibonacci sequence converge on the Golden Ratio.

Now suppose that I create squares, each with sides equivalent to the numbers in the Fibonacci sequence (that is, I create squares whose sides are of lengths 1, 1, 2, 3, 5, and so on). Each square I create is added to the others so that they form a rectangle (Figure I.6). I can create an arc by connecting opposite corners of the squares.

FIGURE I.6: A Fibonacci arc. See text for description.

This is called a Fibonacci arc, and it too is observed in nature—for example, in the shape of seashells like the nautilus, and in the pattern of the seeds of flowers (such as the sunflower and daisy, as shown in Figure I.7). Spirals are observed in other plants as well—for example, the cauliflower, although easier to see in the Romanesco (a kind of broccoli-cauliflower hybrid).

FIGURE I.7: Examples of Fibonacci arcs observed in nature.

Fibonacci sequences are also present, though more subtly so, in the arrangement of leaves of many plants.

For example, in the rubber plant shown in Figure I.8, starting from the top we have three clockwise rotations before we meet another leaf directly below the first, passing five leaves on the way. If we go counterclockwise, we need just two rotations. Note that 2, 3, and 5 are consecutive Fibonacci numbers. This ratio of rotations to leaves is commonly observed.

FIGURE I.8: The leaves of many plants grow in a Fibonacci spiral, centered on the stem.

The interpretation of the aesthetic value of the Golden Ratio would seem to be clear: we are naturally drawn to objects showing the Golden Ratio because this ratio is found throughout nature.

But what is the connection of the Golden Ratio to persuasion? The great nineteenth-century British poet John Keats ended Ode on a Grecian Urn with these words: Beauty is truth, truth beauty. That is all ye know on earth, and all ye need to know. Keats, it turns out, was an excellent psychologist. We associate beauty and truth. When we see something that is physically beautiful, we assume that it has other good qualities, including truthfulness.

In semiotics (the study of symbols) one would call this a sign. Just as red means hot and blue means cold, beauty means truth. But the significance of red and blue to temperature is a cultural convention, and one that each of us must learn. The connection of beauty and truth is made across cultures, and need not be learned. It seems to be a natural part of the human makeup.

People are more likely to believe the contents of a book or magazine if its dimensions correspond to the Golden Ratio. Children’s books might be square, and so might art or cookbooks, but something like 95 percent of the nonfiction books that seek to persuade are sold in dimensions within 2 percent of the Golden Ratio (Figure I.9). The figure for magazines is over 90 percent.

FIGURE I.9: A surprisingly high percentage of nonfiction books use page formats corresponding to the Golden Ratio, but only those that seek to persuade.

The Golden Ratio does exert a powerful and powerfully subtle influence on persuasion. Or it would, if not for a small problem: the Golden Ratio theory is bunk.

Some of the statistics I’ve cited here are just plain inaccuracies. Studies have been conducted in which people (ordinary people² or professional artists and designers³) are shown a large selection of rectangles and are asked which they find most attractive. It’s not the case that people select the Golden Ratio rectangles. Another study examined the dimensions of 565 rectangular paintings by famous artists. Artists showed no predilection for canvas sizes that respected the Golden Ratio; the mean ratio was 1.34.⁴ And natural objects like the human body, faces, and seashells show lots of variability. It’s not the case that the most attractive show the Golden Ratio.⁵ The statistics about the dimensions of books and magazines are complete fabrications.

Some of the Golden Ratio phenomena are accurate but trivial—trivial because examples that fit the Golden Ratio are emphasized, and examples that do not fit are ignored. Why evaluate the Parthenon and not the Pantheon? Why the pyramid of Giza and not the pyramid of Khafre? For that matter, why not the Roman Colosseum, the Taj Mahal, the Alhambra, or the Eiffel Tower? Then, too, a complex figure like the Parthenon or The Last Supper has many measurable features; that makes it too easy to pick and choose measurements that yield the desired ratio.⁶

I apologize for beginning this book with a sucker punch. (Maybe some part of me wanted company. I fell for the Golden Ratio hook, line, and sinker when I first heard it.a) The Golden Ratio is not interesting because it’s true. It’s interesting because the idea survives and continues to attract believers even though it is known to be wrong. In that way, it’s an object lesson for this book. Knowing what to believe is a problem.

The Problem

People believe lots of things for which the scientific evidence is absent: that a special coin brings them luck, that aliens visit Earth regularly, or that astrological predictions are better than chance.b Many such beliefs, though unfounded, are harmless. Maybe they cost us a little time or money, but we find them fun or interesting, and we don’t take them all that seriously anyway.

But unfounded beliefs related to schooling are of greater concern. The costs in time and money can be substantial; worse, faulty beliefs about learning can potentially cost kids their education. Scientific tools can be a real help in sorting out which methods and materials truly help students learn and which do not. We cannot afford to let educational practice be guided by hunch or hope if better information is available. But even though scientific tools are routinely applied, the product is often ignored, or else it’s twisted by people with dollars on their minds.

Consider learning styles theories. These theories maintain that different people have different ways of learning, and that we can identify an individual’s style, tune our teaching to that style, and make learning easier or more effective. For example, the most popular theory of learning styles holds that some people learn best by seeing things (visual learners), some by hearing things (auditory learners), and some by manipulating objects (kinesthetic learners). This theory has been around for at least twenty-five years, and it has been tested in scientific experiments. In fact, testing the theory is quite straightforward.

1. Take one hundred people and identify them as visual or auditory learners. (Let’s skip kinesthetic learners for the sake of simplicity.)

2. Devise comparable visual and auditory materials to learn. For example, people might listen to a story (auditory) or watch a silent slide show depicting the same story (visual).

3. Have fifty people experience the story in their preferred way, and fifty people experience the story in their nonpreferred way.

4. The next day, test everyone’s memory for the story. If the learning styles theory is true, people who experienced the story in their preferred way ought to remember it better.

Experiments like this have been conducted, and there is no support for the learning styles idea.⁷ Not for visual, auditory, or kinesthetic learners, nor for linear or holistic learners, nor for any of the other learners described by learning styles theories.

Yet if you search for learning styles on the Internet, you will not find a brief, academic obituary for this interesting idea that turned out to be wrong. You’ll find almost two million hits. You’ll find almost two thousand books on Amazon. You’ll find the term mentioned on the syllabi of thousands of college courses. And you’ll find lots and lots of products that promise improved educational outcomes once you know students’ learning styles . . . although knowing a child’s learning style often requires buying the book they want to sell you, or attending a workshop they are conducting.

The main cost of learning styles seems to be wasted time and money, and some worry on the part of teachers who feel that they ought to be paying more attention to learning styles, for it appears that most teachers don’t do much with them. The cost of other scientifically inaccurate beliefs has been more substantial. Consider this example. Before about 1920, the way to teach children to read seemed obvious. You start by teaching them the sound associated with each letter or letter combination (Figure I.10).

FIGURE I.10: For many years, students learning to read were first taught to associate the shape of letters with associated sounds, as in this image, reproduced from the New England Primer, published around 1760.

In the first quarter of the twentieth century, another theory of reading rose to prominence.⁸ In essence, it argued that children should be taught to read the way adults read. Adults seem to read entire words or even phrases all at once. (Watch the eyes of someone reading, and you’ll see that they do not dwell on each word, but rather stop a few times as they scan each line.) Adults read silently, which is much faster than reading aloud. And adults read what interests them. Children, in contrast, are taught to read sound by sound (not whole words), aloud (not silently), and out of boring primers (not engaging material).

In what became known at the look-say or whole-word method, children were encouraged to memorize entire words. Books for reading instruction used a limited set of words so as to make memorization possible.⁹ Students were encouraged to guess at a word based on the surrounding context and pictures if they did not recognize it. This method also emphasized that phonics drills—memorizing letters and their associated sounds—are boring and that using them was likely to make children hate reading. Instead, the whole-word method suggested that children be surrounded by real books, not drill materials, and that the stories be ones that they can understand and identify with. The whole-word method became dominant in American education during the 1930s and 1940s.¹⁰

Two factors might have tipped off educators that this approach to reading instruction was suspect. First, written language is a sound-based system, not a meaning-based system. Seeing the three letters d, o, and g doesn’t tell you meaning. Letters signify sounds. If that weren’t true, then when I showed you an unfamiliar word—for example, mielesta—you wouldn’t just be uncertain of its meaning; you would also have no idea of how to pronounce it. Given that writing is sound based, teaching reading with a method that ignores sound seems risky.

Second, the theory encourages the teaching of reading based on the way adults read. On the one hand, you can see the logic: if you want to learn something, find someone who is good at it and then try to do what he or she does. On the other hand, there’s no guarantee that the expert did it that way when he or she was a beginner. An expert basketball player no longer needs to think about the basics of ball handling and footwork because that has been so extensively practiced. The expert thinks about play making and strategy, but the beginner needs to think about fundamentals. Copying an expert reader is not necessarily a good strategy for beginning readers.

In 1955, the book Why Johnny Can’t Read was published.¹¹ It argued that if the direct teaching of the sounds that go with letters were omitted, reading was not being taught. The book was a strident, take-no-prisoners invective, and it became a best seller. The book was negatively reviewed by many education professionals, however.¹² Professors who studied reading argued that the book was ill-informed and that the author was simply wrong. Over the next several years, arguments over how to teach reading raged, later dubbed the Reading Wars.

In 1961, the Carnegie Corporation sought a scholar to comb through all of the scientific studies and to draw a conclusion: Was phonics-based or whole-word instruction superior? Jeanne Chall, a professor at the Harvard Graduate School of Education, was selected to conduct the review. In her 1967 book, she said that the relevant research showed that the phonics method was superior.¹³

That sounds clear enough, right? Education rides off the rails briefly (well, actually for thirty years or so), but science comes to the rescue. So we would expect that post-1967, the whole-word approach to reading instruction would be relegated to the dustbin. Well, we’d be wrong. The basic idea behind whole-word reading resurfaced in the mid-1980s.¹⁴ Renamed whole language, the pitch was familiar: phonics instruction deadens interest and is unnecessary. Learning to read is as natural as learning to speak. Just surround kids with authentic materials, and they will learn to read on their own. The Reading Wars began again. Some districts and even entire states (notably California) adopted curricula based on the whole-language method of teaching reading.

In 1997, Congress asked the Department of Education to draw together a panel of reading experts to sort through the scientific research on reading instruction. Their conclusion, published in 2000, matched that of Chall in 1967.¹⁵ Phonics instruction is a critical part of learning to read. In its absence, some kids figure out on their own the sounds that go with letters and letter combinations. But some don’t. And those kids will end up disliking reading, and some of them will end up being labeled as dyslexic.

The first phase of the Reading Wars was understandable. Someone had an ill-conceived theory about reading instruction. It sounded good, so people tried it. It’s somewhat more difficult to understand why it took as long as it did—some thirty years—for the scientific evidence to influence public opinion and public policy. It is nearly inconceivable that the same mistake was made again twenty years later, sparking the second phase of the Reading Wars.c

When science is brought to bear on education problems carelessly or underhandedly, perhaps the worst damage is inflicted on kids with disabilities. (As the parent of a child with Edwards’ syndrome—also called Trisomy 18—I have personal experience here.) Many developmental disabilities do not have effective treatments, and parents are desperate. They are willing, even eager, to try unproven alternative treatments—anything that might work, anything that holds out some hope. Furthermore, there are a lot of disabled kids in this country—estimates are that about 13 percent of kids have some disability, ranging from very mild speech impairments to chromosome disorders that affect nearly every aspect of intellectual and physical development.¹⁶

Scam artists go where the money is, and the parents of kids with autism spectrum disorder (ASD) are some of their preferred targets because there are a lot of them. Kids with ASD show a fairly broad range of characteristic behaviors, but they tend to have these in common: (1) difficulty with communication, both verbal and nonverbal (that is, pointing, gesturing); (2) problems in social relations, especially in perceiving the emotions and thinking of others; and (3)