Do Impacts on Test Scores Even Matter? Lessons from Long-run Outcomes in School Choice Research American Enterprise Institute - AEI

"...For the past 20 years, almost every major education reform has rested on a common assumption: Standardized test scores are an accurate and appropriate measure of success and failure.This study is a meta-analysis on the effect that school choice has on educational attainment and shows that, at least for school choice programs, there is a weak relationship between impacts on test scores and later attainment outcomes.Policymakers need to be much more humble in what they believe that test scores tell them about the performance of schools of choice: Test scores should not automatically occupy a privileged place over parental demand and satisfaction as short-term measures of school choice success or failure..."Michael Q. McShane, “Do impacts on test scores even matter? Lessons from long-run outcomes in school choice research,” American Enterprise Institute, March 19, 2018, https://www.aei.org/research-products/report/do-impacts-on-test-scores-even-matter-lessons-from-long-run-outcomes-in-school-choice-research/Do impacts on test scores even matter? Lessons from long-run outcomes in school choice research Executive Summary Almost every major education reform of the past 20 years at both the state and national level has rested on a common assumption: Standardized test scores are an accurate and appropriate measure of success and failure. It has followed that programs or policies that increase student scores on standardized tests are “good” and programs that fail to do so are “bad.” This way of thinking was central to the No Child Left Behind Act. But the same logic was applied elsewhere. The past 20 years has seen explosive growth in school choice programs, and these programs have largely been evaluated based on their impacts on student test scores.1 Reading and math tests measure basic skills that almost everyone believes are important. Test scores are convenient to collect. Yet even the most fervent believer in the power of standardized tests agrees that test scores are merely an interim measure. There is no point in increasing test scores for test scores’ sake. Increased test scores are supposed to indicate progress toward more important long-term results. Perhaps the most convincing evidence of this supposed truth comes from Raj Chetty’s seminal work that found connections between changes in students’ test scores and the likelihood that they would graduate from high school or have children as teenagers and between changes in students’ test scores and their earnings in their late 20s.2 But other education research, especially involving school choice, sows doubt with respect to using test scores as the primary measure of program success. A growing number of studies are finding that school choice programs can improve high school graduation rates, college attendance, and earnings—without producing gains in test scores. Conversely, studies of other school choice programs have found large short-term test score gains but no lasting benefits in terms of graduation rates or college attainment. Improving test scores appears to be neither a necessary nor sufficient condition for improving the later-life outcomes that truly matter. We have attempted to collect every experimental and quasi-experimental study of school choice in the US that examines attainment impacts. Most of these studies also examine impacts on test scores. We use an expansive definition of school choice, including private school voucher programs, charter schools, early college high schools, magnet schools, and vocational schools. We compare impacts on test scores to impacts on later attainment outcomes. Our question is, across all studies, do program impacts on test scores predict impacts on later outcomes? This review is one of the most thorough ever done of the school choice literature. We review every known study that contains participant-effect estimates for both student achievement and attainment. We exclude studies that look only at achievement scores. We take a simple analytical approach. We collapse findings into four categories: significantly positive, insignificantly positive, insignificantly negative, and significantly negative. We then map achievement findings against attainment findings. Using such vote-counting methods, we find that, among these studies, program impacts on achievement are inconsistent, perhaps on balance weakly positive, thus replicating the school choice achievement findings of more sophisticated meta-analyses of the test score effects specifically of vouchers3 and charters.4 However, impacts on attainment are much more consistently positive. This pattern itself implies that some programs have produced clearer attainment impacts than achievement impacts, but the pattern of findings is actually more complicated. Programs that produced no measurable positive impacts on achievement have frequently produced positive impacts on attainment.5 And on the other hand, null effects on high school graduation and college attendance have been reported from programs that produced substantial test score gains.6 Across these studies, achievement impact estimates appear to be almost entirely uncorrelated with attainment impacts. A school choice program’s impact on test scores is a weak predictor of its impacts on longer-term outcomes. Our findings are based on 39 unique impact estimates across studies of more than 20 programs. In the coming months and years, more studies of school choice will be released. Perhaps over time a stronger connection between achievement and attainment impacts will emerge. We suspect not. This pattern of findings is not unique to choice policies. The growing literature on early childhood education has found that short-term impacts on test scores are inconsistent predictors of later-life impacts. Some of the preschool programs that produced the most impressive improvements in later-life outcomes did so without producing lasting gains on test scores.7 Studies of teacher impacts on student outcomes show a similar pattern of results. As with school choice, it has been argued that teacher impacts on test scores should be used for policy purposes: Teachers who produce gains should be rewarded and promoted, and those who do not should be remediated or fired. “Value-added” methods have been developed that produce estimates of teacher-level impacts on test scores. Some researchers have used value-added methods to assess teacher impacts on other noncognitive outcomes. It turns out that teacher impacts on test scores are almost entirely uncorrelated with teacher impacts on student classroom behavior, attendance, truancy, and grades.8 Likewise, teacher impacts on test scores are uncorrelated with teacher impacts on self-reported noncognitive skills such as grit.9 Teachers who possess higher noncognitive skills boost the noncognitive skills of their students but not student test scores.10 In short, the teachers who produce improvements in student behavior and noncognitive skills are not particularly likely to be the same teachers who improve test scores. Our findings suggest that the same appears to be true of schools of choice. Our findings beg serious questions about using standardized tests as the exclusive or primary metric on which to evaluate school choice programs. If test score gains are neither a necessary nor a sufficient condition for producing long-term gains in crucial student outcomes, then current approaches to accountability for school choice programs are questionable at best. Our findings suggest that focusing on test scores may lead authorities to favor the wrong school choice programs. Focusing on test score gains may lead regulators to favor schools whose benefits could easily fade over time and punish schools that are producing long-lasting gains. Introduction A careful read of the existing literature on school choice presents a paradox. In 2010, a federally funded evaluation of a school voucher program in Washington, DC, found that the program produced large increases in high school graduation rates after years of producing no large or consistent impacts on reading and math scores.11 Conversely, a recent evaluation of Boston charter schools found no effects on high school graduation and null effects on college attendance after previous evaluations had found remarkably large impacts on reading and math scores.12 These findings appear to go against the grain of the current logic model of education policy. Much of the federal and state education policy of the past two decades has been driven by the assumption that test scores are a meaningful and important measure of what children need to know. Test scores are by far the most popular short-term outcome used in education research and program evaluation.13 Using short-term outcomes is understandable—stakeholders do not want to wait years, even decades, to know whether a program is effective. In school choice research, the number of studies that examine test score impacts far outnumber studies that examine later-life outcomes such as high school graduation, college attainment, and employment income.14 Test scores have become easier and cheaper than ever for researchers to use. Standardized test scores can be administered to any child from third grade on up, efficiently, on a large scale. If the tests are well designed, they reliably measure skills that we think are important, including language arts and mathematics. In the wake of No Child Left Behind, virtually every state already collects test scores on students between third and eighth grade and at least once again during high school. Gaining access to state-collected data is far less labor intensive than collecting one’s own data. The ready availability of test scores makes research on math and reading skills convenient—but is such research important? For research on test scores to actually be meaningful, the following should be true: The impacts that schools have on math and reading skills will change the trajectories of children’s lives. Otherwise, why would policymakers and researchers put such emphasis on “student achievement” and “student growth”—measures that are based on test scores? This assumption seems uncontroversial. It is well-known that childhood test scores and later outcomes are strongly correlated.15 It seems sensible that boosting reading and math skills would boost later educational attainment. Children who know more will go further in school, right? Yet recent findings from prominent school choice studies present a puzzling picture. Test score effects are disconnected from attainment effects. These seemingly paradoxical findings motivated this study. To find the best evidence on the question of the connection among school choice, test scores, and later-life outcomes, we have carried out what we believe to be the most expansive review of the scholarly literature on the impact that school choice programs in the United States have had on educational attainment. We have done so to determine the frequency with which the same school choice programs are found to have different impacts on test scores and educational attainment. We use as broad a definition as possible for school choice. We do so for two reasons. First, we wanted to gather the largest number of studies possible to examine the relationship between achievement and attainment impacts across studies. Second, school choice is bigger than voucher programs and charter schools. Many large districts have embraced a portfolio model of school choice governance, which intentionally offers a wide array of public (and sometimes private) school choices to parents. The diversity of the studies we collected mirrors the diversity of choice options that portfolio school districts attempt to offer. In the following section, we briefly describe our search strategy for gathering studies and our screening methods for including studies in our overall review. In the section thereafter, we describe the studies included by school choice type: private school voucher programs, open enrollment programs, charter schools, selective enrollment schools, career and technical schools, inclusive STEM schools, early college high schools, and small schools of choice. We then present our aggregate findings and discuss technical considerations to keep in mind. Finally, we discuss the important implications of our findings for current policy and future research. Notes William G. Howell et al., “School Vouchers and Academic Performance: Results from Three Randomized Field Trials,” Journal of Policy Analysis and Management 21, no. 2 (2002): 191-217; Joshua M. Cowen et al., “School Vouchers and Student Attainment: Evidence from a State-Mandated Study of the Milwaukee Parental Choice Program,” Policy Studies Journal 41, no. 1 (2013): 147-67; John F. Witte et al., “High Stakes Choice: Achievement and Accountability in the Nation’s Oldest Urban Voucher Program,” Education Evaluation and Policy Analysis 36, no. 4 (2014): 437-56; and Jonathan N. Mills and Patrick J. Wolf, “Vouchers in the Bayou: The Effects of the Louisiana Scholarship Program on Student Achievement After Two Years,” Education Evaluation and Policy Analysis 20, no. 10 (2017): 1-21. Raj Chetty, John N. Friedman, and Jonah E. Rockoff, “Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood,” American Economic Review 104, no. 9 (2014). M. Danish Shakeel, Kaitlin P. Anderson, and Patrick J. Wolf, “The Participant Effects of Private School Vouchers Across the Globe: A Meta-Analytic and Systematic Review” (working paper, Economics Research Network, Department of Education Reform, College of Education and Health Professions, University of Arkansas, Fayetteville, Arkansas, 2016). Julian Betts and Y. Emily Tang, “A Meta-Analysis of the Literature on the Effect of Charter Schools on Student Achievement,” Center on Reinventing Public Education, August 2014. Patrick J. Wolf et al., “School Vouchers and Student Outcomes: Experimental Evidence from Washington, DC,” Journal of Policy Analysis and Management 32, no. 2 (2013): 246-70; and Ruth Curran Neild, Christopher Boccanfuso, and Vaughan Byrnes, “The Academic Impacts of Career and Technical Schools: A Case Study of a Large Urban School District,” Everyone Graduates Center, January 2013. Joshua D. Angrist et al., “Stand and Deliver: Effects of Boston’s Charter High Schools on College Preparation, Entry, and Choice,” Journal of Labor Economics 34, no. 2 (2016): 275-318. Diane Whitmore Schanzenbach and Lauren Bauer, “The Long-Term Impact of the Head Start Program,” Brookings Institution, August 2016. Seth Gershenson, “Linking Teacher Quality, Student Attendance, and Student Achievement,” Education Finance and Policy 11, no. 2 (2016); and C. Kirabo Jackson, “Non-Cognitive Ability, Test Scores, and Teacher Quality: Evidence from Ninth Grade Teachers in North Carolina,” National Bureau for Economic Research, December 2012. David Blazar and Matthew Kraft, “Teacher and Teaching Effects on Students’ Academic Behaviors and Mindsets,” Mathematica Policy Research, 2015; and Matthew A. Kraft and Sarah Grace, “Teaching for Tomorrow’s Economy? Teacher Effects on Complex Cognitive Skills and Social-Emotional Competencies,” Brown University, 2016. Albert Cheng and Gema Zamarro, “Measuring Teacher Non-Cognitive Skill and Its Impact on Students: Insight from the Measures of Effective Teaching Longitudinal Database,” Economics of Education Review (forthcoming); and Albert Cheng, “Like Teacher, Like Student: Teachers and the Development of Student Noncognitive Skills,” University of Arkansas, College Department of Education Reform, 2015. Wolf, “School Vouchers and Student Outcomes.” Angrist et al., “Stand and Deliver.” Mark W. Lipsey et al., “Translating the Statistical Representation of the Effects of Education Interventions into More Readily Interpretable Forms,” National Center for Special Education Research, November 2012. Betts and Tang, “A Meta-Analysis of the Literature on the Effect of Charter Schools on Student Achievement”; and Shakeel, Anderson, and Wolf, “The Participant Effects of Private School Vouchers Across the Globe.” Chetty, Friedman, and Rockoff, “Measuring the Impacts of Teachers II.”