Efforts to stop TAKS cheating often fall short

Here is the last of the three parts. -Angela

Efforts to stop TAKS cheating often fall short
More emphasis has been placed on test, not on catching copiers
11:47 PM CDT on Monday, June 4, 2007

By JOSHUA BENTON and HOLLY K. HACKER / The Dallas Morning News
jbenton@dallasnews.com and hhacker@dallasnews.com
Last of three parts

In 1975, a social scientist named Donald Campbell came up with the idea that would eventually be called Campbell's Law. He wrote like an academic, but you could boil the concept down to this:

The higher the stakes, the more likely people are to cheat.

It makes intuitive sense. The Dallas Morning News' analysis of TAKS scores found that cheating is almost twice as common on the 11th-grade test – which is required for graduation – as on the 10th-grade test.

But experts say that Texas has missed the lesson of Campbell's Law. Over the past two decades, the state's tests have become the dominant force in Texas education. But they say the state's test-security system – the rules and tools officials use to prevent cheating – hasn't kept up with the increasing importance the TAKS test now has in students' and educators' lives.

Also Online
Interactive: See test score results, see where the cheating schools are and take a quiz

Graphic: A pattern of cheating (.pdf)

Part One: Analysis shows TAKS cheating rampant

Part Two: Cheating's off the charts at charter schools

Archive: More coverage of TAKS cheating

As a result, they say, the TAKS is given in a far more permissive environment than other high-stakes tests like the SAT, bar exams or graduate-school admissions tests. And much of the cheating on the TAKS could be prevented with a series of reforms based on what those other tests do.

"It seems to me like terribly bad leadership," said Robert Frary, a professor emeritus at Virginia Tech who has studied cheating for more than 30 years. "The people in charge have to make it their priority."

Testing in Texas is leagues away from where it was in the late 1970s, when statewide tests first came into prominence.

Back then, results weren't filtered into school ratings or published in the newspaper. Tests were only given in a few select grades. Realtors didn't pitch houses based on how the neighborhood elementary school had fared.

In the early 1970s, a young man named Jim Impara helped develop Florida's first statewide testing program. He started attending national conferences on the subject, and there was one subject that was almost never discussed.

"I can't remember security ever being a topic," he said. "There wasn't really any incentive to cheat."

But in the decades since then, Texas, Florida and the entire nation have begun to take tests much more seriously. Texas' alphabet soup of tests – first TABS, then TEAMS and TAAS, and now TAKS – evolved into the dominant force in public education. Some teachers' salaries are now tied to their students' performance on test day. School ratings are a mark of public pride or shame. And decisions like whether a teen graduates or a third-grader gets promoted now hinge on test scores.

As Campbell's Law would predict, with higher stakes came a greater temptation to cheat, at all levels of the system. (As Dr. Campbell himself originally put it: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.")

There are a variety of potential penalties on the books for cheating in Texas, all the way up to an entire school being shut down. But punishments are exceedingly rare.

Teachers bear burden
The security of the TAKS depends, in large part, on the many thousands of teachers and other school staffers who handle the exams. For the vast majority, of course, honesty isn't an issue. But recent years have shown there are exceptions.

"You can't have the same kind of control you have when you have everyone come to a neutral location and have outside proctors," said H.D. Hoover, a testing legend who was the primary overseer of the Iowa Test of Basic Skills.

Texas sets no standards for the physical arrangement of students on test day. It has no rules for how far apart students should sit from one another. It also has no rules on seating arrangements, so that in many schools, best friends are allowed to sit within easy view of each other's answer sheets. (Some school districts voluntarily set tighter rules.)

During tests with higher levels of security, such as the SAT, there are usually firm standards on the distances between students, and students are generally not allowed to choose where they sit. Would-be cheaters are sentenced, at least, to eyestrain.

Texas also does not require schools to keep charts recording where students sit. If two students have identical and highly unlikely answer sheets – but there is no way to know whether they sat near each other – it can be difficult for an investigation to proceed. Texas doesn't even require schools to record what classroom a student was in when he or she took the TAKS.

"Knowing where they sit can take the evidence from a statistic and make it stronger," said Dr. Frary, a retired professor of educational measurement.

Some local students said their teachers let them sit pretty much wherever they wanted during TAKS testing. Others said their teachers were more aggressive about assigning seating. But even when teachers take preventive steps, students find ways around them.

Priscilla Ramirez, a student at DISD's Adamson High School, said that on one test day in her class, students were assigned numbers as they walked in. Each number corresponded to a desk, which was intended to randomize seating. But, she said, some students just traded their numbers to sit next to their friends.

Cheating "is sort of like an epidemic," she said. "It's not going to stop unless you really, really, really try."

Perhaps the single most effective cheating-prevention move the state could make, experts say, would be to produce multiple forms of the TAKS. Currently, on each TAKS test, every student gets all the same questions in the same order. That makes cheating easy, since one child's Question 12 is also his neighbor's Question 12.

Scrambling the order of questions in each test booklet makes copying much harder. Many high-stakes tests produce four different versions of exams and distribute them in such a way that no test-taker is seated next to anyone with the same version of their exam.

Diane Birdwell, a teacher at Bryan Adams High in Dallas, said she started producing multiple versions of her classroom tests when she realized kids in her first-period class were sharing answers with their friends in her third-period class.

"It's a great deterrent," said Ms. Birdwell, a board member of the teachers' group NEA Dallas. "You tell the kids, look, there's no point in looking at Johnny's No. 14 because his isn't going to look like your No. 14."

When engineering professor Charles Bernardin first started teaching at the University of Texas at Dallas, he wasn't particularly proactive about preventing cheating on his exams. But he quickly noticed that some of students were taking advantage of his lax security.

"If you leave a hundred dollar bill on the floor, people are going to pick it up," said Dr. Bernardin, who is also the university's director of academic assessment. "There's going to be a temptation."

But then he started producing multiple versions of his exams and mandating where students sit. Now, he said, "I never have any cheating problems."

Switching to multiple forms does come with a few drawbacks.

For example, printing different versions of test booklets can cost more money and make the test harder to administer. But Texas already prints multiple versions of its TAKS test booklets. Each TAKS contains a number of "field test" questions that don't count toward a student's score, but are instead evaluated for inclusion on future editions of the TAKS. Those questions differ from student to student. Only the questions that are identical in every test booklet count toward a student's score.

For Texas, state officials said, the primary concern would be the added expense and complication. More versions of a test would slow down the grading process and increase the chance, however small, of a grading error.

"Certainly we have the ability to make things more and more complex and generate more forms of the test," said Criss Cloudt, a TEA associate commissioner who oversees the state's testing program. "But there's a price you pay in complication."

Scrambled versions of the same test would also have to be statistically equated so officials can ensure they are all of equal difficulty. That can take precious time, considering that in Texas, a legislative mandate requires TEA to score the entire TAKS within 10 days of test day.

Still, Dr. Impara said that despite the potential difficulties, he recommends multiple forms for statewide assessment programs. So does the company he helped found, years after he left Florida – Caveon, the test-security firm TEA hired last year to examine the veracity of its TAKS scores.

"It has drawbacks, but the one benefit is better security," he said.

Dr. Cloudt – who has overseen testing only since February – and other TEA officials defended the state's test-security system. Traditionally, the state has seen its role as providing training to district testing coordinators through manuals, seminars and other avenues. Those district coordinators are then left to define many of the specific rules for test day. That leaves some districts – like, as of this year, Dallas – with relatively strict rules about things like adult supervision and who can proctor tests. Other districts can have less stringent standards.

"We feel we do a lot to keep testing secure," Dr. Cloudt said. "We're certainly open to looking at other ideas."

A role model for Texas?
When it comes to security on state tests, Mississippi is one of the gold standards. It requires at least two adults in the testing room at all times – and in some cases, even more. Violators of state test-security rules can, in the most extreme cases, be fined or face jail time.

State auditors make at least one unannounced visit to every school district each year on testing days to inspect the security procedures in place. If students are seated too close together to meet state guidelines, for instance, schools are told to spread them out. Seating students at cafeteria tables so they face each other – which is allowed in Texas – is prohibited in Mississippi.

And each year, before scores on the graduation test are released, the state runs statistical checks to look for suspicious scores. One method, similar to The News', looks for pairs of students with too-similar answer sheets. School districts are told to investigate the most unusual scores; in about half of those cases, according to state testing director Cindy Simmons, the suspicious scores are invalidated. If state officials disagree with a district's findings, the state Board of Education can overrule the decision.

"There are big consequences attached to these tests, and we want to make sure they're fair for all students and that no one has an unfair advantage," Ms. Simmons said. School districts have been cooperative, she said. "It's not, 'We're trying to catch you.' It's, 'We want you to know.' "

Mississippi officials have discussed switching to multiple forms of the test to increase security even more, she said. But so far, the state has decided not to, citing the cost. "We are not a state that is very wealthy," Ms. Simmons said. "We feel good about our security system as it is."

Little to gain
What keeps state exams from having a system as secure as other big-time tests? For cheating researcher George Wesolowsky, the key question is one of will.

With so much to gain from high test scores, it's not always in a school's or district's best interest to find cheating, he said. Reports of cheating bring schools embarrassing negative publicity. Students and parents become upset.

"Instructors have no encouragement or incentives to actively look for cheating, and prosecuting cheaters is a time-consuming and unpleasant activity," said Dr. Wesolowsky of McMaster University, who worked with The News on its analysis of TAKS scores.

Among researchers who have devised methods of detecting cheating, the biggest roadblock often is resistance from school officials who either deny that cheating exists or think it's not a problem worth fighting.

Dr. Wesolowsky said he's given his cheating detection program to several private and governmental testing organizations. But those groups are also aware of the negative consequences of discovering cheating.

"Usually when they get the program up and running, an iron curtain goes up," he said. Concerns about falsely flagging individual students can lead educators to ignore the even more important evidence a statistical analysis can provide: information on which schools have serious test-security problems.

Dr. Frary has faced similar obstacles. "It's so difficult to institutionalize it," he said. "It's a very hard, uphill battle to convince people to go after the problem."

Even with the wide availability of security measures, some researchers are pessimistic that cheating can ever be completely eliminated as long as someone has something to gain from a high score.

"The basic ground rule is, whatever security measures go in, somebody is going to find out how to evade them," said David Berliner, a professor of education at Arizona State University and a critic of high-stakes testing. "Banks are still broken into."

In any event, Dr. Hoover – father of the popular Iowa test given in many Texas schools – has a different recommendation for how to stop cheating: Reduce the high stakes attached to tests.

"That's what leads to the problems with security: the pressure to raise scores," he said. "It's led to cases where teachers, rightly or wrongly, feel that they are being judged on the basis of these instruments. And some of them cave in and do things they shouldn't."

POSSIBLE REFORMS
MULTIPLE FORMS

Print slightly different versions of the TAKS for different students, keeping all the same questions and answers but arranging them in slightly different orders.

Pro: Would almost completely eliminate casual cheating, where one student sneaks glances at the bubble pattern on a neighbor's answer sheet. Would make it much harder for a teacher or another adult to doctor answer sheets after an exam.

Con: Added complication could increase the time and cost of grading exams. Could raise the risk of grading errors. Would require a statistical procedure to make sure all versions of the test are equally difficult.

PHYSICAL STANDARDS

Require schools to maintain adequate space between students on test day; require students to be seated in some set order with a chart showing where each student sits.

Pro: Reducing a student's ability to sit next to a best friend would make cooperative cheating more difficult. Keeping students farther apart would make casual cheating more difficult. Seating charts would aid investigations into suspected cheating.

Con: TEA officials say some schools don't have facilities big enough to avoid seating students close together.

PROCTORS

Prohibit teachers from proctoring the exams of their own students.

Pro: Would reduce incidence of teachers helping their own students improperly on test day or doctoring their answer sheets after the fact.

Con: Would do little to prevent the much more common phenomenon of students cheating off of one another. Could be logistically difficult in smaller schools.

SOURCE: Dallas Morning News research

State may switch to online tests
One possible path to cutting cheating on exams is digital: switching to computerized testing.

The state has conducted several experiments with online tests, and officials have said some state tests – although probably not the TAKS – could be administered solely online within two years.

Without a physical answer sheet, copying answers from a neighbor would be substantially harder. Officials could easily reorder questions or answer choices for each student – or even slightly alter the important numbers in math problems.

That's how the national exam for certified public accountants is administered, according to Joel Allegretti, spokesman for the American Institute of Certified Public Accountants. Questions are selected for each test taker from a pool of thousands of possible items – meaning no one sees the same questions as his neighbor.

On a computerized TAKS, results would be reported instantly to state servers, giving adults no opportunity to change student answers after testing. And without statewide shipments of test booklets, ill-intentioned teachers wouldn't have access to questions before test day.

"A lot of these security issues become nonproblems with computer testing," said Jim Impara, senior director of test security at Caveon, the Utah firm Texas hired to examine its test scores.

Of course, computerized testing would also open up new avenues of cheating. Most schools don't have enough computers to test an entire grade at once, which could mean testing some students on different days. That could allow answer swapping if questions aren't tweaked from day to day.

To beat cheats, mix up seats, tests
David Harpp, a chemistry professor at Montreal's McGill University, was alarmed when he first discovered that around 5 percent of his students were cheating on his final exams. He and a colleague named Jim Hogan devised a method for detecting the dishonest, but his primary goal wasn't detection – it was prevention.

So he began creating multiple versions of his exams and started a mandatory seating system. The result: Answer copying almost completely vanished. And when McGill instituted his policies universitywide in 1990, detectable cheating dropped to almost imperceptible levels.

Researchers have found there is a magic solution to answer copying by students. It's the cocktail of two specific reforms: defined seating patterns and multiple test versions.

The McGill reforms would stop all but the most determined cheater. A student would have to find a way to communicate with a friend whom he was not seated next to – since all of his neighbors would have a different version of the test.

And if he can get in touch with that distant friend – perhaps via text messaging – he would have to hope the friend had the same test version he did. That would only be a one in four chance.

"It makes cheating more difficult in a dignified way," Dr. Harpp said.

© 2007 The Dallas Morning News Co.