Analysis shows TAKS cheating rampant

Here is the first of three parts. See UT-Professor Julian Vasquez Heilig's recent publication titled "Urban Minority Students on the NCLB Bubble" in the TCEP Review. He critiques the theory of action undergirding the present accountability system. What these pieces refer to as "cheating," Vasquez Heilig treats more generally as "system gaming." In any case, systemic issues are advanced as the primary culprit.

-Angela

Analysis shows TAKS cheating rampant
State says it's addressed the problem, but News uncovers more than 50,000 cases
05:29 PM CDT on Sunday, June 3, 2007

By JOSHUA BENTON and HOLLY K. HACKER / The Dallas Morning News
jbenton@dallasnews.com; hhacker@dallasnews.com


Tens of thousands of students cheat on the TAKS test every year, including thousands on the high-stakes graduation test, according to an in-depth data analysis by The Dallas Morning News.

Also Online

See test score results, see where the cheating schools are and take a quiz

Estimated number of cheaters might be low

Bill would stop cheating by eliminating test

At TEA, years of inquiry, few concrete results

Common questions about analyzing tests for cheating

How the TAKS scores were analyzed for evidence of cheating

Read about comparing tests to find suspicious results

The analysis – among the first of its kind on this scale – found cases where 30, 50 or even 90 percent of students had suspicious answer patterns that researchers say indicate collusion, either between students or with school staff. Perpetrators go almost entirely undetected and unpunished by state officials.

The study contradicts the Texas Education Agency's stance that cheating on the TAKS is extraordinarily rare and that the agency has done a good job of policing it. Many schools with big cheating problems, including some in North Texas, have officially been cleared by recent state investigations – in most cases simply by proclaiming their innocence on a state questionnaire.

The findings also show that on a high-stakes test like the TAKS – which can determine a school's reputation, a teacher's salary and whether a student walks across the stage on graduation day – some people will seek whatever advantage they can find.

"What we have here in many of the schools, particularly charter schools, is rampant cheating involving many students," said David Harpp, a professor at Montreal's McGill University who studies cheating and reviewed the analysis.

What the study found

The study examined statewide scores from 2005 and 2006 on the all-important Texas Assessment of Knowledge and Skills – the state test given in grades three through 11. Some of the key findings:

PROBLEM SCHOOLS
Here are some of the Texas schools with the strongest evidence of substantial TAKS cheating, according to a Dallas Morning News analysis:

Forest Brook HS, North Forest ISD

Forest Brook High is in the long-troubled North Forest district, where eight of its 11 campuses have state ratings of unacceptable and state officials have appointed an overseer for the district's finances. Forest Brook had the highest rate of apparent cheating of any large noncharter school in the state. Across all grades, just over 13 percent of the school's answer sheets were flagged in 2005.

Forest Brook is also home to one of the most extraordinary performances in Texas: the 11th-grade science test in 2005. Half of the students' answer sheets were flagged for cheating: 93 of 186. Forest Brook's passing rate jumped from 54 percent the previous year to 95 percent in 2005.

Worthing HS, Houston ISD

The suspicious answer sheets are spread wide at Worthing. There were eight different TAKS subject tests on which more than 10 percent of Worthing's answer sheets were flagged as likely cheaters. The most suspicious: last year's 11th-grade science test, on which one-third of the school's answer sheets were flagged. That's 47 out of 141. Worthing's passing rate in that subject was up 23 percentage points from the year before.

Sam Houston HS, Houston ISD

In 2005, Sam Houston High had more suspicious answer sheets than any other school in Texas – 468 in all. Every version of the TAKS test taken at the school had at least a dozen answer sheets flagged for cheating. The school did better in 2006, with the number of suspicious answer sheets dropping to 161. But that still left it with the 15th-highest total in Texas.

South Oak Cliff HS, Dallas ISD

South Oak Cliff had 439 answer sheets flagged over the two years The News examined. That's almost one out of every 10. The cheating was greatest on the 11th-grade science and social studies exams, where 18 percent of the answer sheets were flagged.

Note: Charter schools are not included.

• The test scores of more than 50,000 students show evidence of cheating. Some of those students were the innocent victims of others copying their answers. But experts say most were likely either deliberately copying answers or had their answer sheets doctored by school staff.

• That total is a small percentage of all Texas students. (Two-thirds of Texas schools showed no evidence of cheating.) But the suspicious scores are focused on the state's 11th-grade tests. Those are the ones students must pass to earn a diploma.

At more than 100 high schools, at least one in 10 juniors was flagged for having extremely suspicious answer patterns on the TAKS graduation tests. Many of those students graduated last month.

• Cheating is concentrated in the state's two largest districts – Dallas and Houston – and in charter schools.

Even after accounting for their larger size, cheating is more than three times as common in Dallas and Houston as it is in the state's other large urban school districts. In Dallas, one out of every six high school juniors was flagged for cheating in 2006.

And in the state's lightly regulated charter schools – which are funded with tax dollars but run by private companies or groups – cheating was detected at almost four times the rate of traditional public schools. Cheating was more common at underachieving schools, where the pressure to boost scores is the highest.

• Most of the cheating appears to be driven by students copying off of each other, in pairs or small groups. But at a handful of the most flagrant schools, cheating is systemic. On several subject tests, one Houston charter school had 80 percent or more of its answer sheets flagged for cheating – a scale that seems difficult to contemplate without the passive or active involvement of an adult.

"The evidence of substantial cheating is beyond any reasonable doubt," said George Wesolowsky, a professor at McMaster University in Canada who studies cheating on multiple-choice tests like the TAKS. He worked with The News on the analysis, which used his methodology to identify pairs of student answers that were, statistically, too similar to each other to be the result of chance.

Officials at the Texas Education Agency have consistently argued that statistical analysis can't prove cheating and that they must rely on other forms of evidence – like getting teachers to confess to misbehavior – in their investigations. TEA decided not to use data drawn from student answer sheets – even with evidence of widespread copying in a classroom.

That approach has not been fruitful. The agency has cleared 98 percent of the schools in its recent round of investigations, in most cases because school officials did not volunteer knowledge of improprieties. Many of those schools were found to have widespread cheating in The News' analysis.

State officials have said they are willing to reverse course and consider using statistical methods in the future. "I'm certainly open to the idea," said Criss Cloudt, the TEA associate commissioner who recently assumed oversight over the state's testing program.

Different school officials had different reactions to The News' findings.

"I'm not going to dispute the methodology," Dallas Superintendent Michael Hinojosa said. "Your study came to the conclusions on what seem like reasonably objective measures."

He said that after suspicions were raised about TAKS cheating last year, Dallas instituted new test-security policies for this spring's tests. There must now be two adults in every classroom, and their doors must be kept open during testing. Extra monitors were assigned to schools with suspected problems. Those reforms and others, he hopes, will reduce incidents of cheating from the levels found in 2005 and 2006.

"We've had issues regarding our assessment program," he said. "That's why we decided to change our protocols."

Houston school officials, in contrast, issued a statement calling the analysis part of a "continued effort by The Dallas Morning News to dismiss the real academic progress in Texas schools." The statement said there is "absolutely no evidence" of cheating in Houston schools.

Researchers say The News' study raises serious questions about the legitimacy of the state's methods of evaluating schools, which in some cases have given public praise – and promised hard cash – to schools with major cheating problems.

They say that on a test with such high stakes – for every level of the educational system – confronting cheating honestly can be difficult. The incentives for improved scores are strong; those for vigorously fighting cheating are weaker.

"People often don't want to know what's happening," said Robert Frary, a professor emeritus of educational measurement at Virginia Tech who has studied cheating for more than 30 years.


Established methods

The News' analysis was based on a well-established method for detecting answer-copying developed by Dr. Wesolowsky. Research in these methods dates back more than 80 years; variations of them are used to detect cheating on tests like the SAT, the ACT and some college final exams.

"Some of the methods work better than others, but they all work pretty well," Dr. Frary said. "Wesolowsky's is one of the best, maybe the best." Dr. Frary is considered by some to be the modern godfather of the field, having studied it since the 1970s.

Methods like Dr. Wesolowsky's look for pairs of students who share unusually high numbers of uncommon answers. A few shared answers won't trigger any alarm bells. But extreme cases – when the run of identical and unusual answers stretches so long as to move past the boundaries of mere chance – lead to the pair of students being flagged.

Take the case of "Sara" and "Joe," two students at Dallas' South Oak Cliff High. (They're real students, but those aren't their real names.) In 2005, as juniors, Sara and Joe took the science portion of the graduation TAKS. Out of 55 questions, they answered 51 the same way.

That might not be unusual if they were answering them correctly. After all, the correct answer to a TAKS question is almost always the most popular response. But they consistently gave the same unusual wrong answers. In Sara and Joe's case, their answers are so unusually similar to each other that their pattern would appear naturally among the school's innocent students fewer than once in 277 million cases.

"I can't think of any other plausible explanation except cheating," Dr. Wesolowsky said. Dr. Harpp agreed, calling it "an extreme case of arrogant collusion."

Sara and Joe weren't the only South Oak Cliff juniors flagged for cheating on the science test. So were 31 others, The News' analysis found. In all, 77 answer sheets on the 2005 graduation tests were flagged. (The graduation test is also given in English language arts, math and social studies.)

Administrators at several Dallas high schools, including South Oak Cliff, referred questions to district headquarters. But students at South Oak Cliff and other area high schools said they definitely hear about cheating on the TAKS, especially by kids who haven't studied or have missed lots of classes. That's not shocking, since in national surveys a majority of teenagers routinely report they have cheated on tests in school.

"It actually makes me angry that they do that, because other people work hard and they study and all that stuff," said Alma Gonzalez, who just graduated from South Oak Cliff.

She said students would whisper answers to each other on TAKS day, especially when a teacher left the room for a moment. A few times when classmates asked her for answers, she said, she gave wrong ones on purpose.

"They think they can get their way through high school by just cheating, and they're not really learning anything," she said.


Caveon analysis

In 2005, after a series of articles in The News about TAKS cheating, TEA hired the test-security firm Caveon to analyze that year's test scores. Caveon identified 700 schools whose scores it considered suspicious for one or more reasons, including having too many students with answer sheets suspiciously similar to one of their schoolmates.

In that analysis, South Oak Cliff was one of the schools Caveon flagged. But TEA announced in December that it had cleared South Oak Cliff, along with nearly 600 other schools, of any wrongdoing. That decision was based on the contents of questionnaires filled out by each school's administrators about their test-security practices.

"It is with great pride and pleasure that we are now able to exonerate a large majority of the schools flagged by the Caveon report," state Education Commissioner Shirley Neeley said in a prepared statement at the time. "It is imperative that Texans trust our test results and have confidence that they are valid and reliable."

One member of the TEA panel that reviewed those questionnaires said they were not useful in determining whether cheating had actually taken place.

"That's basically what this questionnaire process was about: asking schools, 'Did you cheat or not?' " the panel member said. "We weren't given anything else to go on – no statistical data."

Currently, only 12 of the 700 schools Caveon identified remain under investigation. No schools, so far, have been cited for even a single incident of cheating.

Meanwhile, according to The News' analysis, cheating continued at South Oak Cliff. In 2006, 36 percent of the school's juniors had at least one answer sheet flagged as suspicious on the graduation TAKS.


Forest Brook

South Oak Cliff isn't the only school cleared by TEA despite an apparent cheating problem.

The school that Caveon found the most suspicious was Houston's Forest Brook High School, in the North Forest school district. Caveon's analysis flagged the school in 52 ways, in every TAKS subject area. In particular, Caveon flagged Forest Brook repeatedly for having lots of students with answer sheets very similar to those of their peers.

After site visits and a paperwork review, TEA cleared Forest Brook. The state's report on its investigation states that TEA did not examine any student answer sheets or use the data produced by Caveon's analysis. Instead, agency officials interviewed district officials about whether testing procedures were followed on test day. Forest Brook's leaders denied any wrongdoing.

TEA officials also accepted Forest Brook's explanation for drawing Caveon's attention: Teachers had made a "concentrated effort" to prepare students for the TAKS test, and the school had boxed student answer sheets in such a way that they believed it could have triggered a Caveon flag.

Caveon's work contract does not allow company officials to comment on their findings in Texas. But The News' analysis found rampant answer copying on the graduation test at Forest Brook – and on a scale unmatched in Texas.

On the 2005 science test, for example, 93 of the 186 answer sheets were flagged for copying. That's the highest number of tainted answer sheets on a single test at any school in the state. An additional 56 sheets were flagged on the graduation exams in the three other subjects tested.

Forest Brook – a historically poor-performing school – got a passing rate of 95 percent on the science test that year. That was up from 54 percent the year before.

Or, to put it another way: Forest Brook jumped from 23 percentage points below the state average to 14 points above it.

Dr. Harpp did his own analysis of the North Forest data – using a method different from Dr. Wesolowsky's. He said the data "clearly shows that massive collusion took place" and that North Forest's explanation was "completely unconvincing."

"To dismiss this mountain of evidence merely on the word of a few teachers saying they did everything by the book defies all logic," he said. "In effect, the TEA is certifying that it is more reasonable to believe that nature has completely deviated from its course than that someone has told a lie."

For some schools, TAKS scores mean money. In recent years, a number of state programs have begun to reward schools and their teachers for good test scores. Forest Brook received a $165,000 state grant this year; the school's eligibility depended in part on its 2005 TAKS scores.

North Forest representatives did not return multiple phone calls seeking comment last week.

In all, The News' analysis found 112 schools where at least 10 percent of the answer sheets on a 2005 TAKS test were flagged for cheating. Of those, four are still under investigation by state officials. Another 33 were never flagged by Caveon in the first place, and thus were not part of the TEA investigation. The remaining 75 have been declared cheating-free by state officials.


Sophisticated tactics

Not all cheating schools are created equal.

At many schools, the students identified in The News' analysis are in isolated pairs – the sort of pattern you might expect if the adults in a school are trying their best, but still don't keep a close enough eye on each student to prevent one from sneaking answers off a neighbor.

More serious is the pattern common in many Dallas and Houston high schools on the graduation test. In those schools, there still isn't the pattern one might expect if adults were actively doctoring answer sheets. But the amount of answer copying is large enough that it appears test proctoring is loose.

In 2006, for instance, 17.6 percent of Dallas juniors were flagged for cheating. So were 13.3 percent of Houston juniors. (The statewide average was 4.1 percent.)

"There's always cheating going on, even when it isn't the TAKS test," said Priscilla Ramirez, a rising senior at Adamson High School. She, like all the students interviewed, said she doesn't cheat. But she hears about students who do.

"It's crazy how smart people are about cheating," she said. "If it's not one way, it's another."

Many Dallas-area high school students said they knew of no cheating. But many others said the tools of prospective cheaters have grown beyond the traditional to include text messaging and other electronic forms. Some tactics sound like urban legends – such as kids signaling question numbers and answers with prearranged finger codes – but students swear they're real.

"It's getting good enough where the teachers don't notice it," said Krysha Bluitt, who just finished her sophomore year at A. Maceo Smith High in Dallas.

Students say there is enormous pressure to do well on the TAKS. Performance on the test can have major impacts on the lives of students, teachers, and administrators. For adults, it can mean bonuses or raises. For schools, too many bad scores can mean permanent closure.

"From day one, when you get there, you're there to pass the TAKS," said Ulysses Hauxwell, who just graduated from North Dallas High.

Stephanie Westbrook, acting principal at A. Maceo Smith High, said that the school takes test security seriously and that she knows of no cheating on the TAKS. "Honestly, we try to teach our students about integrity, and it is made very clear to our teachers that [cheating] does not happen under their watch," she said. District officials also send staffers to campuses on test days to provide "an extra set of eyes," she said.

Other Dallas principals contacted by The News either denied there was any cheating on their campuses or declined comment. A Fort Worth official said that district is unaware of any cheating at its schools flagged by The News' study.

The News' analysis found 67 cases where a Dallas ISD high school had at least 10 percent of its answer sheets flagged for cheating. Those cases included nearly all of the district's nonmagnet high schools. (A high school typically gives 10 TAKS tests, and The News looked at scores for two years.)

But in a small number of schools, the answer patterns were so off-kilter that Dr. Wesolowsky had to adapt his methodology to properly examine them, since his method is based on the assumption that most students are being tested honestly.

"This is completely outrageous," Dr. Harpp said of the most extreme cases. "This is so mind-boggling – it requires a new language to describe."


'A useful tool'

TEA officials said they did not feel comfortable evaluating The News' analysis without examining it more thoroughly. But they expressed somewhat less skepticism about the use of statistical analysis than agency officials have over the past year.

"Statistics can be a very useful tool to point you in the right direction," said Michael Donley, TEA's inspector general.

Despite that, he said he felt the agency had been correct to rely on interviews – and to exclude statistical evidence – from their recent investigations.

"I couldn't prove it," he said of accusations at Caveon-identified schools. "I tried. We talked to everyone we could think to talk to. Our investigators are pretty good at getting at when people are telling the truth."

But after repeatedly saying that statistical analysis was not a legitimate tool in investigating cheating, officials said they would now consider using a methodology similar to The News'.

"If it works, we would absolutely look at it," said Dr. Cloudt, the TEA associate commissioner. She assumed oversight of the state's testing program earlier this year after the state's assessment director, Lisa Chandler, was forced out.

No matter how the agency moves ahead, Dr. Harpp said there is no doubt in his mind that the cheating found in Texas is real and, in some places, systemic.

"At some point you have to stand up and say, 'This runs in the face of common sense,' " he said.

GLOSSARY
Accountability system: The state process that assigns ratings to Texas schools. Ratings are based almost entirely on test scores. From best to worst, those ratings are exemplary, recognized, acceptable and unacceptable.

Answer string: The responses a student gives to all of the questions on a multiple-choice test, such as the TAKS. On the TAKS, the answer choices alternate between questions: A, B, C or D, then F, G, H or J.

Caveon: A test-security firm that helps organizations improve the security of their exams. The Texas Education Agency hired Caveon in 2005 to analyze its TAKS scores; when Caveon identified 700 schools with suspect scores, agency officials said Caveon's results were unreliable.

Charter school: A publicly funded but privately run school. Charter schools face fewer regulations than traditional public schools.

Collusion: When students work together improperly during a test. There is strong support in the academic literature for the statistical detection of collusion; other types of cheating detection methods, such as those that count the number of erasures on answer sheets, have less support.

Flagged pair: Two students whom a statistical analysis has identified as having answer sheets extremely similar to one another. Cheating detection methodologies look for cases where the similarity is so great and so unlikely that the chances of it occurring naturally are very small.

Frary, Harpp, and Wesolowsky: Drs. Robert Frary, David Harpp, and George Wesolowsky, three cheating researchers who assisted with The News' analysis. Dr. Wesolowsky's methodology and computer program were used to perform the analysis. Dr. Harpp, using a different detection method, did a separate analysis of several dozen Texas schools. Dr. Frary examined the results.

© 2007 The Dallas Morning News Co.
http://www.dallasnews.com/sharedcontent/dws/dn/latestnews/stories/060307dnmetcheating.433e87c.html#