Reports and Publications
Working Papers
Other Cool Stuff
Did the Striving Readers Comprehensive Literacy Program Reach Its Goals? An Implementation Report (2024 NCEE evaluation report, with Mike Garet, Kerstin Le Floch, Joanne Carminucci, and Barb Goodson). Boosting literacy among school-age children remains a national priority. Nearly one third of students in the United States have not developed the foundational reading skills needed to succeed academically, with students living in poverty, students with disabilities, and English learners especially at risk. Starting in 2010, Congress invested more than $1 billion for state literacy improvement efforts through the Striving Readers Comprehensive Literacy (SRCL) program. SRCL was intended to focus funding on disadvantaged schools, encourage schools to use evidence-based practices, and support schools and teachers in providing comprehensive literacy instruction. This report assesses how well SRCL implementation was aligned with these goals, using information collected from states, districts, and schools in all 11 states awarded three-year grants in 2017.
A Randomized Controlled Trial of the City Year Whole School Whole Child Model’s Tier 2 Services in Five Urban School Districts (Journal of Research on Educational Effectiveness 2024, with Juliette Berg, David Osher, Max Pardo, Marie-Andree Somers, and Zeest Haider). The paper reports results from an evaluation of City Year's Whole School Whole Child (WSWC) model's Tier 2 services in 22 middle schools in five urban school districts before the COVID-19 pandemic. Student Success Coaches provide universal holistic services and Tier 2 services to students based on assignment to one or more “focus lists” of students recommended based on one or more early indicators that are associated with whether students will stay in school and graduate from high school. The effects of Tier 2 services on student math and ELA achievement, attendance, social-emotional skills, and suspensions were evaluated using a two-year, individual randomized controlled trial (RCT). The sample included 1,755 Grade 6 students randomly assigned to receive Tier 2 services or to the control group. There were no significant effects of Tier 2 services on student outcomes. Limitations of the study, including disruptions due to COVID-19 and the challenges of implementing an RCT as a method for studying a complex intervention, are discussed.
Massachusetts Early Literacy Tutoring Study: Summative Report (Massachusetts Department of Education 2023, with Katherine Gross, Ji Hyun Yang, Aleksandra Holod, Emily Ackman, and Emma Wilson; reports for other years also available). This summative evaluation of the Early Literacy Supplemental Services Tutoring program aimed to gather information on program models and predictors of student growth. The information in this evaluation came from data made available to the evaluation team for the period from September 2022 to June 2023.
Texas Teacher and School Leader Project: Summative Report (Texas Center for Education Excellence 2023, with Dioni Garcia Piriz, Dong Hoon Lee, Michael Vaden-Kiernan, Jingyan Xia, and Tara Zuber). For the past 5 years, the Texas Center for Educator Excellence (TxCEE) has leveraged federal grant funding to support three partner districts in implementing a multi-pronged human capital management systems approach with strategies that incentivize and support teachers and school leaders and, ultimately, improve teaching and learning. With funding from the 2017 Teacher and School Leader (TSL) Incentive Program, TxCEE developed the Texas TSL Framework. The primary goal of Texas TSL is to build on districts’ existing human capital management system, including the educator evaluation systems and systems for effective educator development, to improve student achievement. The American Institutes for Research® (AIR®), the independent evaluator of Texas TSL, has completed an implementation and impact study of the program over the last five years. The study assessed formative and impact research questions, including participants’ perceptions, understanding and fidelity of implementation of key TSL program components, the validity and reliability of participants’ evaluation measures, as well as impacts of the program on teacher and student outcomes. This report summarizes the findings.
Children’s Literacy Initiative: Final Report of the i3 Scale-Up Study (IES 2020, with Katie Drummond, Natalie Tucker-Bradway, Deeza-Mae Smith, John Meakin, and Terry Salinger). This i3 scale-up study examined the implementation and effectiveness of a 3-year literacy intervention developed by the Children’s Literacy Initiative (CLI). The study was a school-level cluster randomized controlled trial conducted in 55 elementary schools from four states. Implementation results showed a high level of fidelity of intervention implementation across the treatment schools in the four urban school districts that had not previously worked with CLI. Intent-to-treat analyses of observation data showed that the intervention’s effects on classroom literacy environment and teacher practices were not statistically significant in Year 1. By Year 3, however, treatment teachers who had taught in intervention schools for all 3 years were rated significantly higher in both the quality of classroom environment and the quality of literacy instructional practices than their peers in control schools. Analyses of student achievement data indicated that the CLI intervention had no statistically significant impact on students’ reading achievement in any of the 3 intervention years. This held true for the overall student sample and for grade-specific subsamples, based on the overall literacy test score and for literacy subtest scores.
More Gains Than Score Gains? High School Quality and College Choice. Most measures of high school quality concentrate on how students perform while they are in high school. If schools are intending to prepare their students for later challenges, such as college and careers, a school that produces good high-school students isn't much help if what they learn doesn't help them after they leave. I create a test-score value-added measure of high-school quality and then match it to transcript data from public colleges in Michigan, exploring the relationship between how well high schools raise test scores and how their graduates perform in their first-year college coursework. Overall effects are positive and highly significant, even after accounting for a litany of selection concerns. This may reassure observers who are worried that schools achieve high value added by teaching to the test, reallocating resources toward high-stakes subjects, or even cheating.
The Effect of Labor Market Shocks on College Attendance: Evidence from Plant Closings in Michigan. Human capital theory suggests that students should pursue more education when the local labor market is struggling, particularly if job losses are disproportionately concentrated among high school graduates, because these circumstances are associated with lower opportunity costs of additional schooling. However, credit constraints and incomplete information about financial aid options can prevent students from receiving their optimal level of education. Using WARN Act data on business closings and mass layoffs in Michigan in 2002-2011, I exploit the exogenous variation in local job losses in the year that students graduate from high school, examining changes in college-going rates and substitution between two-year and four-year colleges. A 1-standard deviation increase in per capita job losses is associated with a small but statistically significant 0.2-percentage point (0.3%) increase in the probability of attending college, driven entirely by attendance at two-year colleges; using the WARN job losses as an instrument for local unemployment rates produces similar results. In some subgroups, local job losses are associated with a small but significant decrease in attendance at four-year colleges. This result implies that both standard human capital theory and credit constraints have roles in guiding students' college-going decisions in the face of a turbulent labor market.
Estimating the Effects of a Large For-Profit Charter School Operator (with Sue Dynarski, Brian Jacob, and Silvia Robles). In this paper, we leverage randomized admissions lotteries to estimate the impact of attending a National Heritage Academy (NHA) charter school. NHA is the fourth largest for-profit charter operator in the country, enrolling more than 56,000 students in 86 schools across 9 states. Unlike several of the other large for-profit companies that operate virtual charters, NHA only has standard bricks-and-mortar schools. Our estimates indicate that attending a NHA charter school for one additional year is associated with a 0.04 standard deviation increase in math achievement. Effects on other outcomes are smaller and not statistically significant. In contrast to most prior charter school research that finds the largest benefits for low-income, underrepresented minorities in urban areas, the benefits of attending an NHA charter network are concentrated among non-poor students attending charter schools outside urban areas. Using data from a survey of school administrators in traditional public and charter schools, we document several aspects of school organization, culture and instructional practice that might explain these positive effects.
Bridges Collaborative Interactive Map (with Megan Sambolt, Crystal London, Chad Henry, Trenita Childers, and Katherine Gross). The Bridges Collaborative provides a unique opportunity to explore desegregation and integration efforts in diverse contexts and address some of the gaps in the research. An initiative of The Century Foundation (TCF), the Bridges Collaborative is a coalition of various collaborators, describing itself as a grassroots initiative to advance racial and socioeconomic integration and equity in U.S. schools. AIR has worked with the Century Foundation and the member organizations of the Bridges Collaborative (the Collaborative) to learn more about how the school districts, housing organizations, and charter management organizations involved in the Collaborative have been working to counteract the impact of racial segregation in their schools and neighborhoods and share best practices. Connecting many facets of our work is an interactive map that describes the environments in which the member organizations operate, contrasts those environments with the environments in surrounding states, and allows readers to interpret the descriptive data in the context of a theory of change and interpret qualitative data from the indicator reports and member profiles.
What Works Clearinghouse Procedures and Standards (with dozens of colleagues at AIR, IES, and academic and policy institutions around the country). For more than a decade, the WWC has been a central and trusted source of scientific evidence on education programs, products, practices, and policies. We review the research, determine which studies meet rigorous standards, and summarize the findings. We focus on high-quality research to answer the question “what works in education?” AIR worked with IES to update the Procedures and Standards handbook used to evaluate studies, as well as the Study Review Protocol that defines outcome domains, provides guidance about measures, and clarifies areas of uncertainty in the handbook.
Florida Teacher Value-Added Student Learning Growth (with Eric Larsen, Susan Ward, Paul Bailey, and many others at AIR, Cambium Assessment, and the Florida Department of Education). Section 1012.34, Florida Statutes requires that educator evaluations incorporate measures related to student performance into educator evaluations. It further requires the Florida Department of Education to develop student learning growth models that districts may or may not use as part of their educators’ evaluations. Since 2011, the department has made Value-Added Model (VAM) learning growth results available to districts. AIR and Cambium Assessment computed these student learning growth models for the Florida Department of Education from 2011 to 2021. Technical reports are available upon request from accountability@fldoe.org.
