What we're asking in education
What are the benefits of after-school programming for children?
Effective after-school programming can bring a wide range of benefits to youth, families, and communities. Programs have the potential to develop pro-social skills, improve academic performance, reduce risky behaviors, promote health and wellness, and provide a safe, structured environment for the children of working parents. Understanding conditions that are necessary to improve outcomes for children through after-school programming is critical in effectively filling service gaps for children and families in need. For over fifteen years, PRG has conducted outcomes research and annual evaluations of an after-school program in Baton Rouge, Louisiana.
Results from a First-Year Evaluation of Academic Impact of an After-School Program for At-Risk Students
This paper, published in the Journal of Education for Students Placed at Risk in 2007, presents the research findings of an evaluation of the academic impacts of 21st Century Learning Centers (CCLC) in Louisiana. Using quasi-experimental design, the article operationalizes academic achievement as core and subject test performance on nationally standardized pre- and posttests (Iowa Test of Basic Skills; ITBS). Based on previous research and evaluation requirements, the article (a) employs outcomes of interest to policymakers (standardized test scores); (b) uses program attendance as a key independent variable; (c) uses efficient methods to control for extraneous impact on the dependent variable; and (d) focuses the evaluation on a specific group of students—at-risk children in Louisiana. Findings indicate that the 21st CCLC program is having a positive academic impact on participants who attend the program for 30 days or more. Further, impacts are shared across specific grantee programs, specific subjects, and subgroupings of students. Finally, the study finds that intensity of attendance is positively related to academic impact.
Jenner E., Jenner L., Results from a First-Year Evaluation of Academic Impact of an After-School Program for At-Risk Students, Journal of Education for Students Placed at Risk (2007), doi: 10.1080/10824660701261144
Student Perspectives on the Influences of Participation in an Afterschool Program
At the end of each academic year, PRG research analysts conduct focus groups with roughly thirty elementary and middle school children to explore how an afterschool program is influencing school performance, attitudes toward school, and social-emotional capacities. Focus groups consistently suggest that the afterschool program positively impacts academic skills, by providing dedicated time for students to complete homework in a structured environment and time to review class lessons outside of school, often with the support of peers and instructors. Each year, conversations touch on several themes concerning participants’ understanding of social-emotional skills such as cooperation, empathy, and managing conflict. Although participants are not always able to clearly define concepts related to these social skills, they are often able to provide examples of why these skills are important or instances in which they have used these skills during afterschool programming.
How can innovative, problem-based learning improve improve STEM engagement and achievement in Elementary and Middle School?
Workers in science, technology, engineering, and math (STEM) occupations have higher employment rates and higher median earnings than those non-STEM jobs. Elementary, middle, and secondary education lay the foundation for postsecondary STEM-related majors and occupations. Persistent academic engagement is a well-established predictor of academic achievement. While research has documented the decline in STEM interest among students from elementary to high school, little evidence exists to demonstrate why student perceptions of STEM curricula and careers change during this time. Opportunities to interact with problem-based, immersive, or simulation-based learning in elementary and middle school may motivate students to engage with and pursue opportunities in STEM subjects. Employing approaches that range from large-scale cluster-level RCTs to in-depth qualitative interviews with students and teachers, PRG’s research examines the potential impact of innovative, classroom-based science learning experiences on students’ engagement with science, math, engineering, and data science. PRG’s work has also explored the classroom conditions under which innovative learning experiences are most effective and why.
Data Adventures. PRG is conducting a five-year evaluation of Data Adventures, a classroom-delivered data science and digital storytelling learning initiative designed by Santa Clara County Office of Education (SCCOE), WestEd, and San Jose State University. PRG is conducting an iterative design-based research formative evaluation as well as a cluster randomized controlled trial (RCT). This evaluation will assess the program’s readiness for rigorous evaluation (Phase I) followed by its impact on data literacy outcomes and STEM engagement of students (Phase II) across 10-20 middle schools in Santa Clara County, California.
ENpowered. PRG conducted a five-year evaluation of Project SYNCERE’s ENpowered engineering curriculum to determine its impact on STEM achievement and engagement in middle school students. ENpowered is a 10-week classroom-based engineering design curriculum that incorporates a cross-school design competition that aims to build a passion for STEM learning among underrepresented youth in Chicago. PRG’s impact evaluation of ENpowered was a QED that aimed to draw causal inferences about the effects (impact) of ENpowered on middle school students’ achievement in mathematics and science. PRG also conducted an implementation evaluation to understand how ENpowered works in practice, interpret its efficacy, provide feedback for program improvement, and identify features necessary for sustainability and replication. This program was implemented in 22 schools in Chicago, Illinois.
EngiLearn. PRG collected data from over 2,500 students across 30 schools for this Investing in Innovation (i3)-funded RCT. Findings from the study did not indicate that the simulation-based science learning experience improved engagement and self-efficacy in science, however, qualitative analysis from teacher interviews provided useful feedback for intervention improvements to better align with classroom instruction that would be enjoyable for a diverse population of students.
Classroom Adventures. The findings from the i3-funded research on Engilearn provided a foundation for a subsequent National Science Foundation (NSF)-funded pilot study that examined the potential effectiveness of a series of STEM simulation-based learning experiences on students’ engagement with science, self-efficacy in science, and interest in pursuing a STEM career. Through this funding, Challenger Center developed a package of Classroom Adventures that focus on earth, ocean, and life sciences for fourth and fifth grades. In addition to assessing the intervention’s effect on these outcomes, PRG’s evaluation examined the relationship between these social and emotional learning outcomes and effective methods for measuring them.
Evaluation of the ENpowered Program
This final evaluation report summarizes results from an impact study of a 10-week classroom-based engineering design program designed to increase STEM engagement and achievement in middle school. The study employed a quasi-experimental difference-in-difference approach and included 1,016 students across 19 middle schools in Chicago. The impact evaluation examined the ENpowered program’s impact on science and math grades. The implementation evaluation aimed to explore the extent to which the ENpowered model was implemented as intended at each study site and gathered formative feedback from participating teachers and students to provide continuous feedback for improvement. The confirmatory analyses indicate that participation in ENpowered did not have a discernable effect on middle school science or math grades. The implementation study found that teachers and students reported a high level of satisfaction and enjoyment with the program and that the program provides valuable exposure to the engineering field.
Lass, K., Walsh, S., Burgess, K., Jenner, E. 2025. Evaluation of ENpowered: Final Evaluation Report. Report submitted for publication on the Education Resources Information Center.
Evaluation of Engilearn
The purpose of the EngiLearn Impact Evaluation Report is to report and explain the results of the impact study and provide formative feedback that can be used to further develop the intervention. EngiLearn consists of a hands-on, experiential science simulation for fifth-grade students and professional development training for fifth-grade science teachers. The program is an adaptation of Challenger Center’s site-based simulation technology for use in classroom settings. The impact evaluation aimed to estimate EngiLearn’s impact on students’ academic (science achievement) and noncognitive (academic engagement, self-efficacy in science, academic intentions in STEM, and digital literacy) outcomes. In addition to these primary outcomes, we also investigate the impact of EngiLearn on math achievement and environmental awareness as a secondary study. Primary impact analyses did not find that offering EngiLearn improved students’ science achievement or noncognitive outcomes as compared to the business as usual control condition. The feedback that PRG and Challenger Center received from teachers who implemented EngiLearn indicated that, although the curriculum was rigorous, and students enjoyed the intervention experience, there was not enough time built into the curriculum to get through all the components.
Jenner, E. & Lass, K. (2018, June). U.S. Department of Education Investing in Innovation Grant Program: Challenger Center for Space Education.
Teachers’ Perspectives on Improving Simulation-Based Science Learning for the Classroom
EngiLearn, a computer-simulation based ocean science curriculum and professional development program, was piloted in fifth-grade classrooms in Virginia. Through feedback questionnaires administered after the pilot, science teachers provided insight into the feasibility of the program for classroom-based teaching and the conditions under which it could be improved. Findings indicate that students were highly motivated by the new technology and readily engaged with the more innovative and collaborative aspects of the curriculum. Reading and writing components of the intervention, however, were not dynamic enough to engage students with varied literacy levels.
Can mentoring programs aimed at developing social and emotional skills improve educational outcomes?
Literature suggests that there is a window of opportunity to prevent the consequences of disengagement for students who may be on the path towards school dropout, by developing targeted interventions that focus on the transitions into middle and high school. PRG is conducting several studies to examine the efficacy of school transition and mentoring programs that aim to improve academic outcomes, social and emotional skills, and school connectedness.
Achievement Mentoring (AM). PRG investigated the efficacy of AM, a one-to-one adult mentoring intervention for 10th and 11th grade students identified as high-risk for dropping out of school due to high absenteeism, course failure, or disciplinary infractions. The project served 393 students in 18 high schools in communities in Delaware, Maryland, New Jersey, Pennsylvania, and rural North Carolina. PRG conducted an experimental evaluation to measure impacts on academic and disciplinary outcomes, as well as social and emotional learning skills. The implementation evaluation provided annual formative data on the extent to which the mentoring program was implemented as intended at each study site and qualitative feedback from mentors and mentees regarding the perceived value of the program.
Peer Group Connection – High School (PGC-HS). PRG conducted a multi-site individual-level RCT with 9th-grade students in rural North Carolina schools. PGC is a high school transition and cross-age peer mentoring program that aims to improve academic outcomes, social and emotional skills, and school connectedness. The sample consisted of 1,351 9th-grade students from six high schools.
Peer Group Connection – Middle School (PGC-MS). PRG conducted a multi-site individual-level RCT with 6th-grade students in New Jersey, Maryland and rural North Carolina middle schools. PGC-MS is a middle school transition and cross-age peer mentoring program that aims to improve academic outcomes, social and emotional skills, and school connectedness. The sample consisted of 1,902 6th-grade students from nine middle schools.
Evaluation of Achievement Mentoring
This final evaluation report summarizes the results from an impact study of an individualized adult mentoring program designed to prevent school dropout for students identified as being at risk for dropout after ninth grade. The study employed a randomized controlled trial (RCT) of 393 tenth-grade students. The impact evaluation examined the AM program’s impact on educational outcomes, as well as social and emotional outcomes. The implementation evaluation aimed to explore the extent to which the AM model was implemented as intended at each study site and the amount of programming students assigned to the treatment condition received. The confirmatory intent-to-treat (ITT) analyses indicate that the offer to participate in AM in 10th grade had a statistically significant impact on students’ self-efficacy, perceived adult support at school, and decision-making skills, as well as likelihood of being suspended in 10th grade. Analyses on the subsample of participants that were enrolled through 11th grade indicate that the program had a significant effect on students’ perceptions of adult support, academic self-image, and decision-making skills at the end of 11th grade. The implementation study found that most students assigned to the mentoring program received about half the recommended dosage of mentoring.
Lass, K., Walsh, S., Burgess, K., Levin, J., Jenner, E. 2025. Evaluation of Achievement Mentoring: Final Evaluation Report. Report submitted for publication on the Education Resources Information Center.
Effects of Cross-Age Peer Mentoring Program Within A Randomized Controlled Trial
This paper summarizes results from an impact study of a cross-age peer mentor program designed to prevent school dropout during the transition from middle to high school. The study employed a randomized controlled trial (RCT) of 1,351 ninth-grade students. Although the intent-to-treat (ITT) analyses indicate modest, yet potentially meaningful program impact on ninth-grade outcomes of disciplinary action, school attachment, and expectations of degree attainment across varying dosage levels, complier average causal effect (CACE) estimates suggest that a threshold level of program participation broadens the program’s impact on additional academic achievement and social and emotional learning outcomes. Given the adverse effects of the transition to high school, this promising evidence indicates that the cross-age peer mentoring intervention could be a cost-effective and sustainable strategy for high schools to implement. In presenting the study findings, we outline two methods for estimating the CACE, or the effect of program participation, that allow researchers to leverage RCT data, beyond ITT analyses, to provide meaningful context for how and when interventions work.
Jenner, E., Lass, K., Walsh, S., Demby, H., Leger, R., Falk, G. Effects of Cross-Age Peer Mentoring Program Within A Randomized Controlled Trial. Manuscript submitted for publication.
Evaluation of Peer Group Connection – Middle School
This final evaluation report summarizes results from an impact study of a cross-age peer mentor program designed to prevent school dropout during the transition from elementary to middle school. The study employed a randomized controlled trial (RCT) of 1,902 sixth-grade students. The impact evaluation examined the PGC-MS program’s impact on educational outcomes, including attendance in school and number of courses passed in sixth grade. The implementation evaluation aimed to explore the extent to which the PGC-MS model was implemented as intended at each study site and the amount of programming students assigned to the treatment condition received. The confirmatory intent-to-treat (ITT) analyses indicate that the offer to participate in PGC-MS did not have a discernable effect on student attendance or the number of classes passed during sixth grade. The implementation study found that most schools experienced barriers to implementing the program to fidelity with their sixth graders, who in turn did not get the recommended minimum amount of programming. The schools that were unable to offer the program to fidelity were severely impacted by the COVID-19 pandemic.
Lass, K., Walsh, S., Demby, H., Leger, R., Burgess, K., Jenner, E. Evaluation of Peer Group Connection – Middle School: Final Evaluation Report. Report submitted for publication on the Education Resources Information Center.
Impact on Antecedents of Student Dropout in a Cross-Age Peer Mentoring Program
PRG presented at the SREE (Society for Research on Educational Effectiveness) spring 2020 conference as part of a symposium entitled Unpacking the Logic Model: A Discussion of Mediators and Antecedents of Educational Outcomes from the Investing in Innovation (i3) program.
PRG researchers presented findings that suggest that a cross-age peer mentoring program, Peer Group Connection – High School (PGC-HS), has the potential to mitigate negative outcomes associated with the transition into high school by increasing students’ school engagement and expectations for the future during their ninth-grade year and, when implemented with high fidelity, reducing the likelihood students receive disciplinary referrals and suspensions. Results also suggest the program may not being working as hypothesized with regards to other targeted behavioral and attitudinal antecedents. From a practical standpoint, research such as this is of import to both program developers and educators. For the PGC-HS program developers, these findings on the mechanisms of behavior or behavior change targeted by the program provide validation about how the program is working and whether it is achieving its goals. In areas where the program does not seem to be “moving the needle,” developers plan to utilize the above information in conjunction with results gleaned from the implementation study to revisit the curriculum to better understand program theory and make adaptations to the program that will improve overall performance. Likewise, educators can use information gathered from this type of exploratory research to target their finite resources toward programs that solve a particular area of concern for their school and their students’ needs. They can utilize results to justify implementing, sustaining, and/or expanding a particular program; they can also make decisions about whether they should implement complementary intervention(s) to address unmet needs.
Funded through an Investing in Innovation (i3) development grant, The Peer Connection Study investigated the impact of PGC-HS on student outcomes related to dropout prevention. The study was an RCT with a sample of 1,533 ninth-grade students in rural North Carolina.
How can secondary schools incorporate Science of Reading-aligned practices to support high school students reading below grade level?
Reading scores have been on the decline in recent years. Elementary schools are increasingly moving toward a Science of Reading approach to literacy instruction, however, students continue to enter secondary schools reading below grade level, increasing their chances of dropping out or entering the postsecondary world ill equipped to succeed. Secondary educators typically lack training in fundamental, explicit literacy instruction and high schools often do not have structured policies supporting professional development in literacy instruction. PRG is conducting a large-scale impact study that explores whether providing structured and comprehensive professional development for secondary educators in the Science of Reading, as well as providing students with high-impact tutoring can bring students’ reading abilities up to grade level.
Literacy Plus. PRG is conducting a five-year evaluation of Literacy Plus, a hybrid professional learning and high-impact tutoring approach to improving high school students’ reading comprehension in rural high school across Northern California. The project is a collaborative partnership with Northern Humboldt Union High School District, a consortium of 9 additional rural high school districts, The New Teacher Project (TNTP), Cal Poly Humboldt University, and Century Analytics. Literacy Plus will bring TNTP’s Science of Reading Training (SoRT) for Secondary Educators, a professional development and individualized coaching model, to high school English teachers in 9th and 10th grades. The program also places a trained high-impact tutor in each classroom to provide targeted small group and individualized tutoring to students with greater need. PRG is conducting an individual-level randomized controlled trial (RCT). This evaluation will assess the program’s impact on students’ reading comprehension, course grades, and cumulative GPA at the end of 9th and 10th grades.
What policies can help address persistent equity gaps in college and career readiness?
Graduation rates among U.S. high school students are improving, however, disparities in which students are graduating prepared to enter college or the workforce persist. Students from historically underserved populations, including Black, Hispanic, and Indigenous students and those with disabilities or economic disadvantage continue to graduate at rates below the national average. PRG’s evaluations aim to explore the ways in which schools and districts can enact broad policies to help close these gaps and prepare students for college and career success.
Wisdom Project. The Baboquivari Unified School District (BUSD) implemented its Wisdom Project from 2012-2015. The project implemented a comprehensive schoolwide college readiness strategy at both the middle and high school levels that included teacher training, improvements in technology, and a series of student-directed seminars and trainings to prepare students for high education. The project also offered an elective course teaching organizational and study
skills, college tutoring, and motivational activities. PRG conducted two quasi-experimental impact studies examining the impact of the schoolwide policy and the elective course on students’ college readiness and academic achievement.
Teach, Lead, Counsel. PRG is conducting a five-year evaluation of Teach, Lead, Counsel (TLC), a school-wide, systems change model developed by Orenda Education and being implemented in high schools across California. TLC engages teachers, administrators, and counselors in a structured and coordinated professional learning model and continuous, data-driven improvement routines. PRG’s quasi-experimental impact evaluation will assess the impact of TLC on schools’ graduation rates and A-G proficiency rates, a California university system admissions requirement.
Wisdom Project. In this Impact Evaluation Report we present findings of two separate quasi-experimental studies. The primary study assesses the impact of the AVID-Core-Plus intervention – the combination of the AVID elective course and the schoolwide intervention – on participating students’ academic achievement in math and reading and on their college readiness. The secondary study assesses the impact of the schoolwide intervention on the same academic achievement and college readiness outcomes. Findings from the primary study indicated that the AVID-Core-Plus intervention positively impacted students’ college readiness, but not academic achievement in math or reading. The secondary study findings indicated that the schoolwide policy did not have a discernable effect on students’ college readiness or academic achievement.
Jenner, E., Jenner, L., Walsh, S., Demby, H. & Leger, R. (2016, November). U.S. Department of Education Investing in Innovation Grant Program: Impact Evaluation Results for Dev45 – Baboquivari Unified School District Wisdom Project.
