Are High Quality Schools Enough to Close the Achievement Gap? Evidence from a Social Experiment in Harlem

Will Dobbie and Ronald Fryer, "Are High Quality Schools Enough to Close the Achievement Gap? Evidence from a Social Experiment in Harlem," National Bureau Of Economic Research, November 2009, https://www.nber.org/papers/w15473

“…Harlem Children’s Zone is effective at increasing the achievement of the poorest minority children. Students enrolled in the sixth grade gain more than four fifths of a standard deviation in math and between one-quarter and one-third of a standard deviation in English Language Arts (ELA) by eighth grade. Taken at face value, these effects are enough to close the black-white achievement gap in mathematics and reduce it by about half in ELA. Students in the HCZ elementary school gain approximately four fifths to one and a half a standard deviation in both math and ELA by third grade, closing the racial achievement gap in both subjects. HCZ students are also less likely to be absent. These results are robust across identification strategies, model specifications, and subsamples of the data. Surprisingly, students of all ability levels obtain roughly the same benefit from attending HCZ charters. ….HCZ is enormously successful at boosting achievement in math and ELA in elementary school and math in middle school. The impact of being offered admission into the HCZ middle school on ELA achievement is positive, but less dramatic..Figures 3-5 provide a visual representation of our basic results from HCZ Promise Academy - Middle School. Figure 3A plots yearly, raw, mean state math test scores, from fourth to eighth grade, for four subgroups: lottery winners, lottery losers, white students in New York City public schools and black students in New York City public schools. Lottery winners are comprised of students who either won the lottery or who had a sibling that is already enrolled in the HCZ Promise Academy. Lottery losers are individuals who lost the lottery and did not have a sibling already enrolled. Technically, these represent ITT estimates. In fourth and fifth grade, before they enter the middle school, math test scores for lottery winners, losers, and the typical black student in New York City are virtually identical, and roughly 0.75 standard deviations behind the typical white student. Lottery winners have a modest increase in sixth grade, followed by a more substantial increase in seventh grade and even larger gains by their eighth-grade year. The TOT estimate, which is the effect of actually attending the HCZ Promise Academy Middle School, is depicted in panel B of Figure 3. As we directly observe test scores of lottery winners who enroll and we have a TOT estimate, we can estimate the mean test score for those in the control group who would have enrolled if they had won the lottery (Kling et al., 2007). The TOT results follow a similar pattern, showing remarkable convergence between children in the middle school and the average white student in New York City. After three years of “treatment,” HCZ Promise Academy students have nearly closed the achievement gap in math - they are behind their white counterparts by 0.121 standard deviations (p-value = 0.113). If one adjusts for gender and free lunch, the typical eighth grader enrolled in the HCZ middle school outscores the typical white eighth grader in New York City public schools by 0.087 standard deviations though the difference is not statistically significant (p-value = 0.238). Put differently, the typical student before entering the HCZ public charter school was at the 20th percentile of the white distribution. After three years, the typical student is at the 45th percentile before adjusting for gender and free lunch and the 54th percentile after adjusting for gender and free lunch…”

“…. We use two separate statistical strategies to account for the fact that students who attend HCZ schools are not likely to be a random sample. First, we exploit the fact that HCZ charter schools are required to select students by lottery when the number of applicants exceeds the number of available slots for admission. In this scenario, the treatment group is composed of students who are lottery winners and the control group consists of students who are lottery losers. This allows us to provide a set of causal estimates of the effect of being offered admission into the HCZ charter schools on a range of outcomes, including test scores, attendance, and grade completion. There are two important caveats to our lottery identification strategy. First, the HCZ middle school was not significantly oversubscribed in its first year of operation, and the HCZ elementary schools have never been significantly oversubscribed, making it more difficult to estimate the effect of being offered admission for these groups. Second, results from any lottery sample may lack external validity. The counterfactual we identify is for students who are already interested in charter schools. The effect of being offered admission to HCZ for these students may be different than for other types of students. To complement the lotteries, our second identification strategy uses the interaction between a student’s home address and her cohort year as an instrumental variable. This approach takes advantage of two important features of the HCZ charter schools: (1) anyone is eligible to enroll in HCZ’s schools, but only students living inside the Zone are actively recruited by HCZ staff; and (2) there are cohorts of children that are ineligible due to the timing of the schools’ opening and their age. Our identification is driven by the between-cohort comparison of outcomes within the Zone, using the outcomes of children outside the Zone to control for natural year-to-year variation in test scores. If the interaction between a student’s address and cohort only affects his or her outcome through its effect on enrollment in the charter school, this provides another set of causal estimates. Both statistical approaches lead us to the same basic story. Harlem Children’s Zone is effective at increasing the achievement of the poorest minority children. Students enrolled in the sixth grade gain more than four fifths of a standard deviation in math and between one-quarter and one-third of a standard deviation in English Language Arts (ELA) by eighth grade. Taken at face value, these effects are enough to close the black-white achievement gap in mathematics and reduce it by about half in ELA. Students in the HCZ elementary school gain approximately four fifths to one and a half a standard deviation in both math and ELA by third grade, closing the racial achievement gap in both subjects. HCZ students are also less likely to be absent. These results are robust across identification strategies, model specifications, and subsamples of the data. Surprisingly, students of all ability levels obtain roughly the same benefit from attending HCZ charters. ….HCZ is enormously successful at boosting achievement in math and ELA in elementary school and math in middle school. The impact of being offered admission into the HCZ middle school on ELA achievement is positive, but less dramatic. High-quality schools or community investments coupled with high-quality schools drive these results, but community investments alone cannot....Taken at face value, the achievement gains of HCZ students will translate into dramatically improved life trajectories. Our middle school lottery estimates - the most modest estimated - suggest a 9.7 to 12.5 percent increase in….a 2.8 to 3.7 percent decrease in the probability of committing a property or violent crime….and a 12.5 to 18.8 decrease in the probability of having a health disability…..If HCZ affects educational attainment as dramatically as achievement, the implied benefits are enormous….The public benefits alone from converting a high school dropout to graduate are more than $250,000.54...”

However in2013he did look at the impact of the same school on Non-Test outcomes and found HCZ had additional positive effects, “…Six years after the random admissions lottery, youth offered admission to the Promise Academy middle school score 0.283 standard deviations higher on a nationally-normed math achievement test and are 14.1 percentage points more likely to enroll in college. Admitted females are 12.1 percentage points less likely to be pregnant in their teens, and males are 4.3 percentage points less likely to be incarcerated....”

Fryer found (after attempting to control for selection bias) that Harlem Children's Zone has been effective in closing black-white achievement gap, note he hasn’t undated this since 2009 to see how robust the results remain (it came out in 2009, was published in 2011)