Teaching with the test: Experimental evidence on diagnostic feedback and capacity-building for public schools in Argentina (with R. de Hoyos and P. A. Holland)

World Bank Economic Review (Accepted)

We examine the impact of two strategies to use large-scale assessment results to improve school management and classroom instruction in the Province of La Rioja, Argentina. We randomly assigned 104 public primary schools to: a diagnostic-feedback group, in which we administered standardized tests at baseline and two follow-ups and made results available to schools; a capacity-building group, in which we also conducted workshops and school visits; or a control group, in which we administered tests at the second follow-up. After two years, diagnostic-feedback schools outperformed control schools by .33σ in math and .36σ in reading. In fact, feedback schools still performed .26σ better in math and .22σ in reading in the national assessment a year after the end of the intervention. Additionally, principals at these schools were more likely to use assessment results for management decisions and students were more likely to report that their teachers used more instructional strategies and rated them more favorably. Combining feedback with capacity building does not seem to lead to additional improvements, but this might be due to schools assigned to receive both components starting from lower learning levels and participating in fewer workshops and visits than expected.

Hard cash and soft skills: Experimental evidence on combining scholarships and mentoring in Argentina (with F. Barrera-Osorio, M. L. Biehl & M. A. Cortelezzi)

Journal of Research on Educational Effectiveness (Conditional acceptance)

We evaluate a long-standing program run by one of the largest education foundations in Argentina that offers scholarships and non-academic mentoring to secondary school students. We randomly assigned 408 grade 6 students within 10 public schools in the Province of Buenos Aires to either receive the program throughout secondary school or to not receive it. After three years, the program improved students’ academic behaviors (e.g., studying before an exam or catching up on missed work), but we find little evidence that these changes translated into broader improvements in students’ academic mindsets (e.g., self-beliefs about performance and efficacy), perseverance (e.g., grit), or learning strategies (e.g., metacognition). The program also improved students’ performance in school during the first year (e.g., grades, attendance, and passing rates), but we do not find similar gains in subsequent years. This may be due to a large share of treatment students being expelled from the program for not meeting its requirements. The program did not improve student learning or personality traits (e.g., conscientiousness). Finally, we find some heterogeneous effects for female students and students from low socio-economic status.

Disrupting education? Experimental evidence on technology-aided instruction in India (with K. Muralidharan & A. Singh)

American Economic Review, 2019, Vol. 109, No. 4, pp. 1-35

We present experimental evidence on the impact of a personalized technology-aided after-school instruction program on learning outcomes. Our setting is middle-school grades in urban India, where a lottery provided winning students with a voucher to cover program costs. We find that lottery winners scored 0.36σ higher in math and 0.22σ higher in Hindi relative to lottery losers after just 4.5-months of access to the program. IV estimates suggest that attending the program for 90 days would increase math and Hindi test scores by 0.59σ and 0.36σ respectively. We find similar absolute test score gains for all students, but the relative gain was much greater for academically-weaker students because their rate of learning in the control group was close to zero. We show that the program was able to effectively cater to the very wide variation in student learning levels within a single grade by precisely targeting instruction to the level of student preparation. The program was cost effective, both in terms of productivity per dollar and unit of time. Our results suggest that well-designed technology-aided instruction programs can sharply improve productivity in delivering education.

More than words: Expressed and revealed preferences of top college graduates entering teaching in Argentina (with M. Alfonso & A. Santiago)

Comparative Education Review, 2017, Vol. 61, No. 3, 581-606

School systems are trying to attract top college graduates into teaching, but we know little about what dissuades this group from entering the profession. We provided college graduates who applied to a selective alternative pathway into teaching in Argentina with information on what their working conditions and pay would be if they were admitted into the program. Then we observed whether they reported that they wanted to go into teaching and whether they did so. We found that individuals who received information about working conditions or pay were more likely to report that they no longer wanted to pursue their application to the alternative pathway but no more likely to drop out of the program’s selection process. This could be due to prominence effects. Students with higher GPAs were more likely to drop out if they received information on working conditions but not if they received information on pay.

The barking dog that bites: Test score volatility and school rankings in Punjab, Pakistan (with F. Barrera-Osorio)

International Journal of Educational Development, 2016, Vol. 49, pp. 31-54

This paper is the first to explore the implications oftest score volatility for school accountability policies in a lower-middle-income country. Using two new datasets from the province of Punjab, Pakistan, we find that rankings based on school-level averages of students’ test scores fluctuate considerably from one year to the next due to factors unrelated to school quality. First, most variation in test scores is within, not between schools, allowing noise to play a large part in determining school rankings. Second, variation in school test scores is partially attributable to fluctuations in the characteristics of student cohorts. Third, an even larger share is explained by one-time shocks. Fourth, these problems are more pronounced in small schools. As a result, rankings that rely on test score levels or changes often fail to identify top-performing or rapidly improving schools, since they are influenced by factors beyond schools’ control.

Improving educational outcomes in developing countries: Lessons from rigorous evaluations (with R. J. Murnane)

Review of Educational Research, 2016, Vol. 86, No. 3, pp. 719-755

In this article, we reviewed and interpreted the evidence from 223 rigorous impact evaluations of educational initiatives conducted in 56 low- and middle-income countries. We considered for inclusion in our review all studies in recent syntheses that have reached seemingly conflicting conclusions about which interventions improve educational outcomes. We grouped interventions based on their theory of action. We derived four lessons from the studies we review. First, reducing the costs of going to school and expanding schooling options increase attendance and attainment, but do not consistently increase student achievement. Second, providing information about school quality, developmentally appropriate parenting practices, and the economic returns to schooling affects the actions of parents and the achievement of children and adolescents. Third, more or better resources improve student achievement only if they result in changes in children’s daily experiences at school. Fourth, well-designed incentives increase teacher effort and student achievement from very low levels, but low-skilled teachers need specific guidance to reach minimally acceptable levels of instruction.

Why do some school-based management reforms survive while others are reversed? The cases of Honduras and Guatemala

International Journal of Educational Development, 2016, Vol. 47, pp. 33-46

In the 1990s, El Salvador, Nicaragua, Guatemala, and Honduras enacted school-based management (SBM) reforms that allowed communities to make key decisions about their schools that were previously reserved for state-appointed officials. Yet, these reforms have recently begun to slide back. What explains this trend toward reform reversal? This paper argues that two factors determine the likelihood of the reversal of an SBM program: the scope of the reform and the level of national investment in it. Using the cases of Honduras and Guatemala, I contend these two factors determine the extent to which an SBM reform is vulnerable to events that can bring about its termination, such as changes in government, union strength, or parental pressure.