Assessing Accountability

Inside Higher Ed
July 1, 2009

Most states don’t have systems in place to measure college students’ learning outcomes, and rare is the state that actually uses accountability data to drive policy decisions, a new report says.

Education Sector, a think tank promoting education reform, analyzed accountability systems across the nation and found varied results in its report, "Ready to Assemble: Grading State Higher Education Accountability Systems." The group’s survey determined that 38 states have little if any system for measuring learning outcomes, adding that 36 states have yet to develop a method for linking college funding to performance.

“Accountability isn’t just about gathering information; it’s about doing something useful with it,” said Kevin Carey, policy director for Education Sector.

“There’s a lot of innovation for states to learn from,” he added. “The bad news is I don’t think any state has put together a complete package.”

Education Sector measured states in 21 categories of accountability, analyzing any systems that might be in place to assess areas like affordability, degree production, research and scholarship. States that promote or require the use of assessment tools, and take steps to publicize the information, were given the highest marks. Those that had few tools for assessment and did little to spread information were graded lower.

Education Sector graded 50 states, Puerto Rico and the District of Columbia on a three-grade scale. The highest grade, “best practice,” was given to 10 states with well developed reporting mechanisms. The second ranking, “in progress,” was given to 27 states that have less complete efforts underway. The lowest category, “needs improvement,” went to 13 states, D.C. and Puerto Rico, where little is being done in the way of accountability, according to Education Sector.

To assess the level of accountability, Education Sector examined whether states use new assessment tools like the National Survey of Student Engagement (NSSE) or the Collegiate Learning Assessment (CLA). States have used these tools, developed by nonprofit organizations, in part to answer critics who say higher education hasn’t held itself accountable. Some fear that if colleges and universities don’t develop their own standards to measure effectiveness, federal standards could be implemented. Carey said he would not be interested in a large government accountability system akin to No Child Left Behind.

“Nobody I think wants a No Child Left Behind for higher education; we certainly don’t,” he said. “But if you’re not going to have direct regulation, which we don’t think we should have, then really accountability will only work if we create strong incentives. Incentives have to be tied to what institutions care about.”

In other words, states need to develop systems to reward colleges that show improvement in areas like student engagement, graduation rates and research production, Carey said.

While there is great room for improvement, there are some bright spots in the accountability universe, the report acknowledges. South Dakota, for instance, uses the Collegiate Assessment of Academic Progress (CAAP) -- developed by the makers of the ACT college-entrance examination -- to see if students are making satisfactory progress in their first two years of college. If students fail to meet standards three times, they are not permitted to re-enroll in state institutions. The provision affects about 2 percent of students each year, according to the report. While that may not sound like much, it's one of the few examples in the report of a state setting a data-driven standard that has consequences.

While several states use the NSSE, institutions vary when it comes to publicizing what students say about them. Vermont, which was ranked as a “needs improvement” state, puts the results of all 80 NSSE survey questions into a searchable database accessible to the public. While the state got high marks for transparency, Vermont could still be more proactive in informing the public about the performance of its institutions, the report said.

Even states that do a good job of collecting data often fall short when it comes to publicizing the findings, Carey said.

“The average student and parent can’t be expected to sift through mountains of PDF files and an obscure spreadsheet,” he said.

The states with the lowest grades were often cited for failing to compile information by race and gender, something many states do regularly. On the other hand, the report sought to find out which states collect less commonly used data. Only five states, for instance, were given the “best practice” designation for measuring the way colleges improve their community’s quality of life through arts and cultural programs. Connecticut, which was rated “best practice” in this category, is the only state to track the number of artistic and creative products attributable to state institutions, according to the report. The state keeps tabs on how many plays, compositions, paintings and other cultural contributions can be traced back to colleges and universities.

Overall Grades for States on Education Sector's Accountability Measures