Excellent read in the April 15, 2022 edition of The Chronicle of Higher Education. Errors, ambiguity, and the pressure to look good make the annual U.S. News and World Report's rankings suspect. The first mistake is to believe that rankings are objective as cited herein by Akil Bello, from FairTest:
"[Bello] has noticed more critiques of rankings methodologies making it into the mainstream. 'There are some cracks being created in the belief that there is an objective foundation to the creation of the rankings,' he said.”
The critique herein falls along the same lines as that of school rankings from our high-stakes accountability systems that rely on high-stakes, standardized exams. In short, we make too much of these as Bello also states herein:
“How do you unring the bell of the socially accepted rankings?” FairTest’s Bello said. “That’s the biggest challenge right now — is that the ‘These colleges are good’ and ‘These colleges are bad’ has entered the ether of the higher-ed admissions landscape.”
To Bello's question, all I can say is that a lot of bells are getting unrung right now. The very act of bringing this sacred cow to question is an advance.
This critique raises the obvious questions of whose interests are served by these rankings, as well as how this approach has become time-honored, reflecting cultural values that attach to social rewards and hierarchies?
Again, not unlike K-12 education, before us is a culture of measurement that manifests along narrow delimited dimensions of what we may think of as "school quality."
However cultural or globalized these ranking systems may be, I don't need any convincing that they're unfair.
-Angela Valenzuela
obert Morse, the lead designer of U.S. News & World Report’s rankings methodology, speaks at the professional conference for college data-crunchers every year. And every year, attendees say, his workshop is packed.“Every time I’ve gone to that session, it’s been standing-room-only and people leaning in the door,” said Jeffrey A. Johnson, director of institutional research and effectiveness at Wartburg College. Conference-goers always ask tough questions, said Todd J. Schmitz, assistant vice president for institutional research for the Indiana University campuses. But Morse’s audience is rapt: “He has this room of 300 people hanging on his every word,” Johnson said.
The scene captures the complicated relationship between colleges’ data submitters and U.S. News, the best-known college-ranking system in the United States. Many resent the time and oxygen the ranking takes up. After The Chronicle asked to interview her, Christine M. Keller, executive director of the Association for Institutional Research, conducted an informal poll of the group’s members about their views of U.S. News. One major theme: Answering the magazine’s survey requires too many resources, a situation they see as taking away from internal data projects that contribute more to student success than rankings do. Yet they know that responding to the survey is an important part of their jobs, and often campus leaders are paying close attention.
Recently that relationship has undergone renewed scrutiny, as rankings-data controversies have piled up. First, a former dean of Temple University’s business school was sentenced to 14 months in federal prison for leading an effort to inflate statistics his school had sent to U.S. News. Then a Columbia University mathematics professor publicized his belief that his institution is sending inaccurate data to Morse and his colleagues, a contention Columbia has denied. Finally, the University of Southern California pulled out of the rankings for its graduate program in education because it discovered it had submitted wrong data for at least five years.
The headline-grabbers are the latest in a decades-long history of scandals about colleges gaming U.S. News and unintentionally sending inaccurate data to it. Every few years, it seems, another incident comes to light. Criticisms of the rankings have also been longstanding, but there’s been some fresh attention since the popular journalist Malcolm Gladwell covered them on his podcast last year.
Willis Jones, an associate professor at the University of Miami who studies higher-education leadership, has noticed more of a social-justice bent to rankings criticisms lately. Increased societal awareness of historically Black colleges and universities highlighted that rankings are “one of the many things that were creating disparities among HBCUs versus predominantly white institutions in state funding and things like that,” he said.
Another observer, Akil Bello, director of advancement for the National Center for Fair and Open Testing, an advocacy group known as FairTest, has noticed more critiques of rankings methodologies making it into the mainstream. “There are some cracks being created in the belief that there is an objective foundation to the creation of the rankings,” he said.
As the college staffers typically responsible for gathering and submitting the high-stakes data, institutional researchers are on the front lines of this much-scrutinized process. And while outright lying may be relatively rare, there’s always human error, plus ample room for interpretation in the U.S. News questions. That ambiguity can create incentives to finesse the data in a way that makes one’s institution score better in the magazine’s rubric.
“Not one of them had a clean audit,” Larmett said. Colleges were making unintentional mistakes, often as a result of software systems not working well together, the timing of data pulls (at what point in the year enrollment is counted, for example), and human errors in data entry.
“High risk” statistics, where Baker Tilly auditors often saw problems, included the number of applicants, admitted students’ test scores and GPAs, and faculty-to-student ratios. Test scores can be problematic if colleges rely on numbers shared by applicants rather than by testing companies. And who counts as a faculty member or a student can be defined in many ways, depending on who’s asking.
The differences between what colleges reported and what Baker Tilly found were generally small, Larmett said. But you never know what will be enough to give an institution a lower or higher ranking than it deserves, she said, given that U.S. News doesn’t disclose exactly how it weights survey answers in its rankings.
Even with perfect quality control, two institutions may still count the same number in different ways.
Continue reading here.
