What we're asking in education
What are the benefits of after-school programming for children?
Effective after-school programming can bring a wide range of benefits to youth, families, and communities. Programs have the potential to develop pro-social skills, improve academic performance, reduce risky behaviors, promote health and wellness, and provide a safe, structured environment for the children of working parents. Understanding conditions that are necessary to improve outcomes for children through after-school programming is critical in effectively filling service gaps for children and families in need. For over fifteen years, PRG has conducted outcomes research and annual evaluations of an after-school program in Baton Rouge, Louisiana.
Results from a First-Year Evaluation of Academic Impact of an After-School Program for At-Risk Students
This paper, published in the Journal of Education for Students Placed at Risk in 2007, presents the research findings of an evaluation of the academic impacts of 21st Century Learning Centers (CCLC) in Louisiana. Using quasi-experimental design, the article operationalizes academic achievement as core and subject test performance on nationally standardized pre- and posttests (Iowa Test of Basic Skills; ITBS). Based on previous research and evaluation requirements, the article (a) employs outcomes of interest to policymakers (standardized test scores); (b) uses program attendance as a key independent variable; (c) uses efficient methods to control for extraneous impact on the dependent variable; and (d) focuses the evaluation on a specific group of students—at-risk children in Louisiana. Findings indicate that the 21st CCLC program is having a positive academic impact on participants who attend the program for 30 days or more. Further, impacts are shared across specific grantee programs, specific subjects, and subgroupings of students. Finally, the study finds that intensity of attendance is positively related to academic impact.
Jenner E., Jenner L., Results from a First-Year Evaluation of Academic Impact of an After-School Program for At-Risk Students, Journal of Education for Students Placed at Risk (2007), doi: 10.1080/10824660701261144
Student Perspectives on the Influences of Participation in an Afterschool Program
At the end of each academic year, PRG research analysts conduct focus groups with roughly thirty elementary and middle school children to explore how an afterschool program is influencing school performance, attitudes toward school, and social-emotional capacities. Focus groups consistently suggest that the afterschool program positively impacts academic skills, by providing dedicated time for students to complete homework in a structured environment and time to review class lessons outside of school, often with the support of peers and instructors. Each year, conversations touch on several themes concerning participants’ understanding of social-emotional skills such as cooperation, empathy, and managing conflict. Although participants are not always able to clearly define concepts related to these social skills, they are often able to provide examples of why these skills are important or instances in which they have used these skills during afterschool programming.
Can simulation-based science learning in elementary school improve STEM engagement and achievement?
Jobs in science, technology, engineering, and math (STEM) pay better on average and are projected to grow at higher rates than non-STEM jobs. Virtually all STEM occupations require post-secondary education, yet fewer than 20% of American students obtain a STEM-related degree. Persistent academic engagement is a well-established predictor of academic achievement. While research has documented the decline in STEM interest among students from elementary to high school, little evidence exists to demonstrate why student perceptions of STEM curricula and careers change during this time. Opportunities to interact with problem-based, immersive, or simulation-based learning in elementary school may motivate students to engage with and better achieve in STEM subjects. Employing approaches that range from large-scale cluster-level RCTs to in-depth qualitative interviews with students and teachers in the Mid-Atlantic region, PRG’s research examines the potential impact of simulation-based science learning experiences on students’ engagement with science, self-efficacy in science, and interest in pursuing careers in the STEM field. PRG’s work has also explored the classroom conditions under which simulation-based learning experiences are most effective and why.
Teachers’ Perspectives on Improving Simulation-Based Science Learning for the Classroom
EngiLearn, a computer-simulation based ocean science curriculum and professional development program, was piloted in fifth-grade classrooms in Virginia. Through feedback questionnaires administered after the pilot, science teachers provided insight into the feasibility of the program for classroom-based teaching and the conditions under which it could be improved. Findings indicate that students were highly motivated by the new technology and readily engaged with the more innovative and collaborative aspects of the curriculum. Reading and writing components of the intervention, however, were not dynamic enough to engage students with varied literacy levels.
PRG’s research examines the potential impact of simulation-based science learning experiences on students’ engagement with science, self-efficacy in science, and interest in pursuing careers in the STEM field. Through the implementation of a large-scale randomized controlled trial (RCT) from 2014-2018 and a pilot study that will continue through 2020, PRG is contributing to the evidence base surrounding effective interventions to improve STEM achievement and engagement.
EngiLearn. PRG collected data from over 2,500 students across 30 schools for this Investing in Innovation (i3)-funded RCT. Findings from the study did not indicate that the simulation-based science learning experience improved engagement and self-efficacy in science, however, qualitative analysis from teacher interviews provided useful feedback for intervention improvements to better align with classroom instruction that would be enjoyable for a diverse population of students
Aquatic Investigators 3.0. The findings from the i3-funded research has provided a foundation for a subsequent NSF-funded pilot study that will examine the potential effectiveness of a series of STEM simulation-based learning experiences on students’ engagement with science, self-efficacy in science, and interest in pursuing a STEM career. In addition to assessing the intervention’s effect on these outcomes, PRG’s evaluation will examine the relationship between these social and emotional learning outcomes and effective methods for measuring them.
Can a mentoring program aimed at developing social and emotional skills improve educational outcomes?
Literature suggests that there is a window of opportunity to prevent the consequences of disengagement for students who may be on the path towards school dropout, by developing targeted interventions that focus on the transitions into middle and high school. PRG is conducting several studies to examine the efficacy of school transition and mentoring programs that aim to improve academic outcomes, social and emotional skills, and school connectedness.
Peer Group Connection – High School (PGC-HS)
PRG conducted a multi-site individual-level RCT with 9th-grade students in rural North Carolina schools. PGC is a high school transition and cross-age peer mentoring program that aims to improve academic outcomes, social and emotional skills, and school connectedness. The sample consisted of 1,351 9th-grade students from six high schools.
Peer Group Connection – Middle School (PGC-MS)
PRG conducted a multi-site individual-level RCT with 6th-grade students in New Jersey, Maryland and rural North Carolina middle schools. PGC-MS is a middle school transition and cross-age peer mentoring program that aims to improve academic outcomes, social and emotional skills, and school connectedness. The sample consisted of 1,902 6th-grade students from nine middle schools.
Achievement Mentoring (AM)
PRG is investigating the efficacy of AM, a one-to-one adult mentoring intervention for 10th and 11th graders identified as high-risk for dropping out of school. The project will serve high-need students in up to 20 high schools in low-income communities in rural North Carolina and urban districts on the East Coast. PRG will conduct an experimental study to measure impacts on academic and disciplinary outcomes, as well as social and emotional learning skills. PRG aims to enroll approximately 800 students into the study, beginning in September 2020.
Impact on Antecedents of Student Dropout in a Cross-Age Peer Mentoring Program
PRG will present at the SREE (Society for Research on Educational Effectiveness) spring 2020 conference as part of a symposium entitled Unpacking the Logic Model: A Discussion of Mediators and Antecedents of Educational Outcomes from the Investing in Innovation (i3) program.
PRG researchers will present findings that suggest that a cross-age peer mentoring program, Peer Group Connection – High School (PGC-HS), has the potential to mitigate negative outcomes associated with the transition into high school by increasing students’ school engagement and expectations for the future during their ninth-grade year and, when implemented with high fidelity, reducing the likelihood students receive disciplinary referrals and suspensions. Results also suggest the program may not being working as hypothesized with regards to other targeted behavioral and attitudinal antecedents. From a practical standpoint, research such as this is of import to both program developers and educators. For the PGC-HS program developers, these findings on the mechanisms of behavior or behavior change targeted by the program provide validation about how the program is working and whether it is achieving its goals. In areas where the program does not seem to be “moving the needle,” developers plan to utilize the above information in conjunction with results gleaned from the implementation study to revisit the curriculum to better understand program theory and make adaptations to the program that will improve overall performance. Likewise, educators can use information gathered from this type of exploratory research to target their finite resources toward programs that solve a particular area of concern for their school and their students’ needs. They can utilize results to justify implementing, sustaining, and/or expanding a particular program; they can also make decisions about whether they should implement complementary intervention(s) to address unmet needs.
Funded through an Investing in Innovation (i3) development grant, The Peer Connection Study investigates the impact of PGC-HS on student outcomes related to dropout prevention. The study is an RCT with a sample of 1,533 9th grade students in rural North Carolina.
Effects of Cross-Age Peer Mentoring Program Within A Randomized Controlled Trial
This paper summarizes results from an impact study of a cross-age peer mentor program designed to prevent school dropout during the transition from middle to high school. The study employed a randomized controlled trial (RCT) of 1,351 ninth-grade students. Although the intent-to-treat (ITT) analyses indicate modest, yet potentially meaningful program impact on ninth-grade outcomes of disciplinary action, school attachment, and expectations of degree attainment across varying dosage levels, complier average causal effect (CACE) estimates suggest that a threshold level of program participation broadens the program’s impact on additional academic achievement and social and emotional learning outcomes. Given the adverse effects of the transition to high school, this promising evidence indicates that the cross-age peer mentoring intervention could be a cost-effective and sustainable strategy for high schools to implement. In presenting the study findings, we outline two methods for estimating the CACE, or the effect of program participation, that allow researchers to leverage RCT data, beyond ITT analyses, to provide meaningful context for how and when interventions work.
Jenner, E., Lass, K., Walsh, S., Demby, H., Leger, R., Falk, G. Effects of Cross-Age Peer Mentoring Program Within A Randomized Controlled Trial. Manuscript submitted for publication.
Evaluation of Peer Group Connection – Middle School
This final evaluation report summarizes results from an impact study of a cross-age peer mentor program designed to prevent school dropout during the transition from elementary to middle school. The study employed a randomized controlled trial (RCT) of 1,902 sixth-grade students. The impact evaluation examined the PGC-MS program’s impact on educational outcomes, including attendance in school and number of courses passed in sixth grade. The implementation evaluation aimed to explore the extent to which the PGC-MS model was implemented as intended at each study site and the amount of programming students assigned to the treatment condition received. The confirmatory intent-to-treat (ITT) analyses indicate that the offer to participate in PGC-MS did not have a discernable effect on student attendance or the number of classes passed during sixth grade. The implementation study found that most schools experienced barriers to implementing the program to fidelity with their sixth graders, who in turn did not get the recommended minimum amount of programming. The schools that were unable to offer the program to fidelity were severely impacted by the COVID-19 pandemic.
