"We find negative average impacts of attending a charter school on math
scores for all three grade levels, and on science scores for middle and
high schools but not for elementary schools. We don’t find any evidence
that charters have much of an impact on reading scores at any grade
level."
-Angela
Mixed Results for Arizona’s Charter SchoolsCharter schools are more popular in Arizona than in any other
state. In the 2012-13 school year, 13.3 percent of Arizona students
attended charter schools, almost three times the national average of 4.6
percent. That same year, Arizona’s 530 charter schools accounted for
nearly a quarter of all public schools in the state. But student-level
data on Arizona’s schools have not been made widely available, so the
state’s charters have not been subject to the kinds of impact
evaluations that have been completed in states such as Florida and North Carolina and cities such as Boston and New York.
We provide the first recent, comprehensive look at Arizona’s charter schools in a new paper released last week, which will be published in a forthcoming issue of the peer-reviewed journal Educational Evaluation and Policy Analysis. Our
analysis is based on statewide, student-level longitudinal data
obtained from the Arizona Department of Education (AZDOE) that contains
information on test scores, school enrollment, and student
characteristics for the 2005-06 through 2011-12 school years.
Many prominent studies of charter schools take advantage of admission
lotteries to compare students who were equally interested in attending a
charter, but only some of whom were given the opportunity. That method
is the best way to measure the impact of those schools, but these
studies have a potentially important shortcoming: they can only examine
charters that are so popular that they have more applicants than
available seats. Our study examines a wide range of charter schools in
Arizona using methods that have been shown to best replicate lottery-based results.
We focus our analysis on charter middle schools, because we are able
to compare charter and traditional public school students who had
similar entering test scores and demographic characteristics and even
attended the same elementary school. We also examine high schools,
taking into account students’ academic performance at the end of middle
school. It is not possible to use this methodology to examine elementary
schools because testing begins in third grade, so for those schools we
compare test-score growth in traditional public schools and charter
schools while taking into account student characteristics such as race,
age, and special education status.
On average, charter schools in Arizona do no better, and sometimes
worse, than the traditional public schools. Figure 1 shows our estimates
of the average impact of attending a charter school on test scores,
expressed in standard deviation units. To put these units in context,
the average middle school student gains about a quarter of a standard
deviation per year; for elementary students, the average gain is between
a third and a half of a standard deviation. For example, the negative
impact of charter middle schools on math scores of 0.02 standard
deviations translates into about 10 percent of a year of learning.
We find negative average impacts of attending a charter school on
math scores for all three grade levels, and on science scores for middle
and high schools but not for elementary schools. We don’t find any
evidence that charters have much of an impact on reading scores at any
grade level.
Click to enlarge
charter school will perform better, on average, than a similar student
attending a traditional public school. But only comparing averages
likely misses a lot of nuance. There are good and bad charters, just as
there are good and bad traditional public schools. But do charters vary
more in terms of their ability to promote student achievement than
comparable traditional public schools? Our data allow us to answer this
question empirically, something few, if any, prior studies have done.
We find that charters vary more in their impact on student
performance on state tests than traditional public schools. In other
words, even though the average charter has a zero or negative impact on
test scores, there are more charters with very large positive or very
large negative test-score impacts than there are traditional public
schools with such extreme outcomes. We also find that the negative
impacts of charters are concentrated in non-urban areas (Figure 2),
which is consistent with a lottery-based national study finding that charter middle schools deliver better results in urban areas.
The greater variability in the quality of charters is consistent with
the idea that they are laboratories for innovation and experimentation,
some of which succeed and some of which fail. We investigated further
whether certain types of charters are likely to succeed or fail by
separating charter schools into categories based on their mission
statements. Figure 2 shows that schools with missions emphasizing
academic rigor had positive effects on math scores, whereas those with a
progressive (e.g., focused on the “whole child”) or more general
mission statement had negative impacts. Schools focused on the arts also
had negative impacts, perhaps because their focus is on an area other
than core academic subjects.
Click to enlarge
the freedom to innovate they will be held accountable for results. Thus
far, the system of accountability in place in Arizona has not produced a
charter sector that produces better outcomes, on average, than the
traditional sector. But we do find that the charter sector is better at
weeding out poorly performing schools than the traditional sector.
Figure 3 shows that the charter elementary and middle schools that
closed were significantly less effective in math, reading, and science
than traditional and charter schools that remained open. The same was
not true for traditional public schools that closed, which barely
differed from traditional schools that remained open.
Click to enlarge
operate for 15 years before coming up for review. Because the most
rapid expansion of the Arizona charter sector occurred around the turn
of the 21st century, many charters are poised to come up for review in
the next few years. This provides an opportunity for rapid improvement
through careful attention to quality in the reauthorization process, and
the fact that lower-quality charter schools have been more likely to
have their charters revoked in recent years is encouraging in this
regard. But our evidence also suggests that a 15-year period with little
oversight of academic quality may be too long to wait to intervene and
potentially close schools that are producing subpar results. A shorter
authorization period accompanied by vigorous efforts to measure quality
along the way may strike a better balance between autonomy to innovate
and accountability for results.
The Arizona State Board for Charter Schools, which serves as the
authorizer for more than 90 percent of charter schools in the state, has
already taken important steps in this direction. In October 2012, it
adopted a new Academic Performance Framework that
subjects each of its schools to an annual review based primarily on
student achievement levels and growth and clarifies that charters may be
revoked well before the 15-year contract expires. Although standardized
test scores should not be the only metric to judge quality, especially
among schools with certain specialized missions, the framework appears
to provide a sound basis for taking timely action to address consistent
under-performance.
The mediocre performance of Arizona’s charter sector as a whole
should not overshadow the impressive work being done in some individual
schools. The same should be said for the traditional sector, in which
there is also substantial, albeit less, variation in quality. But part
of what makes the charter idea compelling is that it provides
opportunities for schools to innovate, while not tolerating persistent
failure. For this ideal to be realized, policies that drive continuous
improvement of the sector over time need to be in place.
—Matthew M. Chingos and Martin R. West
This post originally appeared on the Brookings Brown Center Chalkboard blog.
Really helpful for students, thanks for sharings.
ReplyDelete