States distort school test scores, researchers say

States distort school test scores, researchers say
Critics say California among those that lower standards for No Child Left Behind
- Carrie Sturrock, Chronicle Staff Writer
Friday, June 30, 2006

California and some other states have inflated test outcomes by lowering the achievement standard students need to meet to be proficient in reading and math under the federal No Child Left Behind Act, university researchers say.

It amounts to a dumbing down of how the states calculate student progress, the researchers concluded.

Under No Child Left Behind, individual schools and school districts can be punished for repeatedly failing to meet the federal standards, including restructuring schools and possibly closing them in extreme cases.

Researchers with Policy Analysis for California Education (PACE) studied 12 states and found nearly all reported results significantly higher than those gathered through the federal government's own testing.

Their study confirms long-held suspicions that many states have set a low bar on what it means to be proficient, distorting the true extent to which kids are learning, said Bruce Fuller, a UC Berkeley professor of education and public policy and co-director of PACE.

In the process, he said, parents have no clear idea of how their children are progressing.

"Parents and citizens can't rely on the state test scores to know whether No Child Left Behind is working -- they can't rely on state test scores to find out whether students are learning more or less over the last 10 years," Fuller said. "We think the states are inflating the percentage of kids who are proficient."

In California, for example, state officials in 2005 estimated 50 percent of fourth-graders were proficient or better in math on the California Standards Tests, compared with 29 percent on the federal National Assessment of Educational Progress or NAEP.

California education officials questioned the validity of comparing an individual state's test scores to a national test that's intended to gauge the relative progress among states. And just because California's test scores outperform the NAEP scores doesn't make them invalid -- it might be instead that NAEP scores underestimate kids' proficiency, said Rick Miller, a spokesman for Jack O'Connell, state superintendent of public instruction.

"The NAEP tests are not aligned to our standards, so (California students are being tested) on information they're not learning in class," he said. "The NAEP scores are less valid -- they're a less-sensitive gauge of the progress of our schools. ... It makes sense to put your stock in the test of what your kids learned."

But U.S. Rep. George Miller, D-Martinez, who helped champion the bipartisan No Child Left Behind legislation, considers the PACE study comparison valid and likened the discrepancies to law school graduates boasting they passed all their tests even though they failed the state bar examination. No Child Left Behind (NCLB) aims to force states to honestly describe the quality of education provided to all students, he said, adding that he had not yet read the PACE study.

"There are a lot of people who can't break the habit of gaming the system," he said of states in general. "They want to appear they are doing right by the children, and the fact is, they're not. NCLB shines the light, and that's why there's so much resistance. It shines the light on a lot of practices where districts and states were conning the parents about the quality of education the children were getting."

PACE, an independent research center at UC Berkeley, UC Davis and Stanford University, compared federal and state fourth-grade test score data between 1992 and 2005 from 12 states: Arkansas, California, Illinois, Iowa, Kentucky, Massachusetts, Nebraska, New Jersey, North Carolina, Oklahoma, Texas and Washington.

The final report, titled "Is the No Child Left Behind Act Working? The Reliability of How States Track Achievement," comes as the House of Representatives' Committee on Education and the Workforce begins hearings on reauthorizing No Child Left Behind in 2007.

No Child Left Behind mandated in 2002 that all 50 states -- many of which already had their own testing programs -- participate in a federal testing plan to improve education for all children. Every state had to create a blueprint for ensuring that every student scores proficient in math and English by 2013. To reach that goal, the federal government requires that school test scores increase by set amounts each year -- called adequate yearly progress -- and threatens sanctions against schools and districts if they don't.

Fuller said states have found it challenging to make adequate yearly progress and face political pressure to permit the numbers to float upward artificially by, for example, allowing teachers to closely tailor their lessons to the test. The study found large gaps between various states' estimations of students' progress and the federal government's.

In Oklahoma, for example, the discrepancy between state and federal findings since 1994 averaged 48 percentage points in reading and 60 percentage points in math since 1996. In Texas, that gap averaged 55 points in reading and 51 points in math.

In California, the gap averaged 19 percentage points in reading and 24 points in math over several years. Massachusetts scores most closely aligned with NAEP -- there was a 10 percentage-point average annual gap in reading scores and a 1 percentage-point gap in math.

The study outlines several reasons for these large differences. One is that states sometimes lower their standards for what they deem proficient.

Another is that once states establish a test, similar test questions may be used each year. Some teachers circulate the tests, making it easier for students to answer the questions, thereby inflating scores. An additional problem in determining student progress, the study notes, is that states commonly change testing companies and/or the actual tests. As a result, scores often drop when a new test is used. States could equate the new and old tests to bring scores into alignment, but many don't, Fuller said. In 2002, Texas reported that 91 percent of fourth-graders were proficient in reading, a number that fell to 76 percent the following year after the state used a new test. According to NAEP, 29 percent of Texas fourth-graders were proficient.

"Parents and citizens, if they look at a graph like this, it's bewildering," Fuller said. "It doesn't help them determine whether students are learning more or less over time."

The study found that overall, federal reading scores have stayed flat, and federal math scores have improved a little since No Child Left Behind. At the same time, states boast significant gains. California has reported a 3.7 percent average annual gain in reading scores while NAEP has showed no improvement. The Golden State did a little better in math, boasting a 4.3 percent average annual gain while NAEP reported a 2.3 percent gain.

"Does NCLB further raise achievement?" Fuller asked. "So far, the evidence is pretty thin."

The study recommends several ways of addressing the disparity between federal and state test scores, including:

-- State policymakers could enact more challenging standards that involve more analytic and writing skills and higher order thinking.

-- The federal government could provide financial resources to help states link old and new tests and compare the scores.

-- The federal government could call for state tests to be released simultaneously with NAEP scores so the public can better gauge how students are doing.

-- Federal officials could encourage states to reach a consensus on where their respective proficiency bars are set.

California education officials insist the state has tougher standards than many states and notes that California students are making real progress.

"Where this report does a disservice is that it implies that real reforms and real improvements made in student achievement over the last five years are not real, when in fact we know that kids are doing better and achievement is up," said Rick Miller, the spokesman for California's superintendent.

On the Web
To read the report, go to pace.berkeley.edu/testscoretrends.html

A big difference

California boasts much greater progress in fourth-grade reading and math scores on its California Standards Tests than the federal government does with its National Assessment of Educational Progress tests. A new study by Policy Analysis for California Education concludes that many states, including California, exaggerate the extent of students' educational achievement.


Fourth grade results, percent proficient or above:


Reading

CST

2002: 36%

2005: 47%

NAEP tests

1992: 19%

2005: 21%


Math

CST

2002: 37%

2005: 50%

NAEP tests

1992: 12%

2005: 29%


Source: Policy Analysis for California Education


E-mail Carrie Sturrock at csturrock@sfchronicle.com.

Page A - 4
URL: http://sfgate.com/cgi-bin/article.cgi?file=/c/a/2006/06/30/MNG28JN9RC1.DTL

©2006 San Francisco Chronicle