Why We Shouldn't Trust the Tests

Endangered Minds: Why Children Don't Think And What We Can Do About It
by Jane M. Healy, Ph.D.

(Page 4 of 5)

This fiasco only illustrates what educational psychologists already realize; strange goings-on sometimes occur in the name of "testing." Test results, in fact, can be quite misleading estimates of just how well, or how poorly, children can read. Perhaps the NAEP results really were accurate. They probably appeared so surprising because other current reading tests — believe it or not — actually make students' abilities look considerably better than they really are! Here are several reasons why most test scores should be taken with a large grain of salt:

1. What Is Reading?

How do you define "reading"? I have described in my first book an unusual group of children called hyperlexics, who teach themselves to read as early as age two and continue to read obsessively from any written material they can get their hands on. One five-year-old hyperlexic boy whom I tested brought the New York Times to my office and proceeded to read it aloud with flawless élan. Not surprisingly, he also scored at the level of an average high school senior on a commonly used reading test that measured how well he could sound out and pronounce words he had never even seen before! With scores like this, the child must be a gifted reader, right? Wrong. Unfortunately, he could not understand the meaning of even a first-grade story. Like others afflicted by this strange syndrome, he could "word-call," but he comprehended little.

The ability to "bark at print" is not reading, but many people, including well-meaning parents, think it is, Tests which show that young children's scores are rising may simply be focusing on the "lower level" skills of word reading while neglecting the real heart of the matter: How well do they understand what they have read? Can they reason — and talk, and write — about it?

2. How Do We Test It?

When testing children on reading skills, it is relatively easy to cheek out "phonics" and other word-reading abilities. It takes much longer to find out how well students have understood a passage. Because it is time-consuming to sit down with each child and do a thorough job, most standardized tests used today are given to large groups of children and scored by machines. They are poor vehicles for assessing comprehension because the student is not required to formulate (say or write) anything, merely to fill in "bubbles," to check off one of a given set of answers. Such multiple-choice tests receive a lot of well-justified criticism because they tend to concentrate on "lower-order" literal questions. Sometimes you don't even have to read the passage to get the right answer:

What color was John's wagon?
green
black
red

"It's testing for the TV generation — superficial and passive," commented Linda Darling-Hammond, director of education for the RAND Corporation. "We don't ask if students can synthesize information, solve problems, or think independently. We measure what they can recognize. But this is very different from what actually goes on in our information society. No one goes to work and finds a checklist on their desk."

Even poor readers may manage to answer "little red wagon" questions, but they start to flounder when the language, the texts, and the questions grow up. One effective way to probe a reader's understanding is to ask him to "tell what happened," give a summary or a paraphrase. Many students today have particular difficulty with such questioning, perhaps because they have never been required to synthesize or talk about texts in this way; they've been too busy filling in the bubbles.

3. "Dumbed-Down" Tests

Most people are unaware that there has been a major "dumbing-down" of reading tests since the 1960s. It is a shocking fact, considering their poor scores, that our children are taking tests drastically more simple than those of only two decades ago. The evidence suggests that test-makers are making children look better than they really are by manipulating the level of difficulty of both the reading and the types of questions asked.

When discussing tests, I often think back to the mid-seventies, when I was principal of a primary school and we switched to the brand-new, updated form of a nationally normed achievement test. Every child's scores magically rose because the new test was so much easier than the previous one. By simply using the new form, we could raise scores significantly without even teaching anything! Educators went around at professional conferences that year telling each other, "If you want your school to look really good, switch to the new form of Brand X achievement test."

What a wonderful discovery! If scores continue to decline — why, just keep changing the tests.

Reading abilities of contemporary children cannot easily be compared with those from past decades because most of the tests have been changed every eight or ten years. In 1978, one college professor in Minnesota gave students in his classes the same reading test that had been used in 1928. Their scores were more like those of the high school students of fifty years earlier. Such comparisons are not terribly valid for a number of reasons, including differences in standard-vocabulary and usage from one generation to another, yet there is every indication that reading abilities have undergone even more accelerated declines since he did this research in 1978. At the same time, we have seen increasingly frequent revisions of the major tests. Do these more frequent changes reflect a greater need for a fix-up?

In 1987, Dr. James Cannell blew the whistle on test-changers. In an incendiary report he charged that the degree of difficulty in the reading comprehension section of the widely used California Achievement Test for second and third graders was a full grade level below that of the 1977 version of the same test. The equally popular Stanford Achievement Test, said this report, "showed a profound drop in expository reading difficulty between 1972 and 1982." Despite noisy protests from the testing establishment, the essential truth of Cannell's findings was subsequently confirmed by a federally sponsored analysis.

Are the test-makers really at fault? "Norms," by definition, vary according to the abilities of the group of children used to develop the scoring system for any given test; if overall abilities decline, so do the standards of the test. If sixth graders in the 1980s are poorer readers than sixth graders were in the 1960s, the 1980s test has to be easier in order to get a "normal distribution" of scores, with many children receiving average scores and only a few out on the extreme high or low ends.

Moreover, because administrators tend to shun tests that make their children look stupid (and themselves incompetent), publishers are naturally pressured to produce tests to make kids look good. They appear to have done exactly what Cannell claimed. When I compared the 1964, 1972, and 1982 forms of a typical, widely used reading test, I was shocked to observe the differences. Each successive edition was so much easier than the previous one that it was hard to believe they were actually given to children of the same grade level! As just one example, Figure 1 shows comparable items (the last page) from the 1964 and 1982 forms of the test for fourth graders. You don't need a master's degree in reading to notice the increasing simplification of content, vocabulary level, sentence length, etc. This test, incidentally, is advertised as "the standard by which all other achievement tests will be measured."

The most scary of all is a new "Advanced" form, designed for ninth graders and published in 1988 (Figure 2), which calls on such complex skills as reading a menu in a fast-food restaurant. This entire test is demonstrably easier than what fourth graders were expected to read in 1964.

Is the publisher's advertisement of this last instrument as "Testing Today's Curriculum" an unconscious irony? Personally, I find it incredible that this is called a "reading" test, yet it is one of the major instruments by which "competency" is evaluated.

4. Teachers and Administrators Can Cheat, Too

When the pressure is on for better test scores, administrators may report falsely inflated results to make their schools or districts look better. Cannell's study found, in fact, that all fifty states were above the national average, although no one knows quite how this apparent miracle occurred. Teachers, too, are susceptible to pressure. When one's evaluation — and maybe one's job — is on the line, even a responsible teacher may slide into a seductive practice called "teaching the test." When the same test is used for more than one year and teachers become familiar with the questions, they tend, perhaps even unconsciously, to focus instruction on the items ("Remember this word — you just might see it again...") that will make their students shine statistically.

There are other clever little ways to manipulate test scores. One group of elementary teachers from Michigan told me they always give the pretest (in September) late in the afternoon and tell the children they can go out on the playground as soon as they finish. For the "posttest" (by which the "gains" from their teaching are judged at the end of the year), they give the students orange juice and a healthful snack first thing in the morning; then when blood (and brain) sugar are at peak level, they hand out the test and encourage the class to take their time and stay in their seats to check answers if they finish early.

Why We Shouldn't Trust the Textbooks

"Johnny is only in third grade, but he's already in a fourth-grade reader!" carols a delighted mother. Unfortunately, she should not assume this accomplishment proves Johnny to be other than a mediocre reader, since many textbooks have also undergone "dumbing down." For some time, textbook publishers have been under pressure to make texts more "readable," unfortunately defined as having shorter sentences, less complex vocabulary, and more pictures. Elementary school textbooks ("basal readers") have increasingly contained short, unnatural sentences and awkward prose that can hardly be expected to endear to students the cadences of good language and literature.

Quality has also been jeopardized by superficial standards of reading "competency." According to a 1988 report of The Council for Basic Education, "Editors are increasingly organizing elementary reading series around the content and timing of standardized tests." The result? "A thin stream of staccato prose winding through an excessive number of pictures, boxes, and charts."

High school textbooks (in science, history, etc.) have been pruned in response to complaints by teachers that students cannot understand books with traditional levels of complexity. Given the caliber of prose "infecting" current history texts, laments history buff Jack Valenti, they "would all fail the essential test: Was it read, enjoyed and remembered?"

In a scathing critique published in Education Week, Arthur Woodward of the University of Rochester took textbook publishers to task for the new stress on visuals that drastically weakens texts. In many cases, he wrote, "instructional exposition takes second place to the design characteristics, which generally resemble those of a coffee-table picture book." He blames the high proportion of pages devoted to illustrations, often quite unrelated to the material at hand, for "the difficulty publishers face in handling given topics with sufficient substance."

Even college-level texts have suffered by becoming more "homogenized," less academic, longer, easier, and more superficially glossy, claims Dr. Diana Paul of the University of Massachusetts and Harvard. These changes came about, at least in part, because "increasing numbers of college students were reading at a level that made it difficult for them to cope with traditional college textbooks," she explains.

Overall, the state of reading points up fundamental changes, not only in skill levels, but also in the way today's students approach thinking and learning. Is it possible that reading is, indeed, an unnecessary relic of a passing culture? Could new habits possibly be more adaptive for today's kids or for society? While these are notions we will consider in the final chapter of this book, most educators see trends away from literacy as overridingly negative. Not only do they put students into direct conflict with the stated goals and methods of education, but they also render them less able to compete in the practical world of work in an information-processing society where verbal and problem-solving skills are in high demand.

Moreover, the expanded mental and human perspectives gained from reading may be a particular imperative for a generation destined to live — and provide leadership — in a technological culture. Do we want policymakers who are untroubled by the weighty realities of history because they have never read — or reflected — about them? Or business leaders who never heard of the likes of Babbitt? Or voters who have never peeked around the corner of their own thinking?

Copyright © 1990 by Jane M. Healy

About the Author

is an educational psychologist and teacher who has worked with students from preschool through graduate school. She consults and lectures worldwide, helping teachers and parents understand the educational implications of current brain research. She has appeared on national media such as the Today show, Nightline, Good Morning America, CNN, and NPR. A mother and grandmother, she currently lives in Vail, Colorado.