From Implementation to Outcomes to Impacts: Designing a Comprehensive Program Evaluation

Abstract:

Funders are often interested in learning about the impact of program activities, yet before the impacts are determined, educational evaluations should first examine program implementation and outcomes. Implementation evaluation examines how and the extent to which program activities are delivered as intended, including the extent to which activities reached the targeted participants. Outcome evaluation is comprised of a systematic examination of the effects that a program has on program participants, such as changes in knowledge, attitudes, beliefs, values, and behaviors.

In this presentation, presenters will share insights on evaluating the implementation, outcomes, and impacts associated with an online science curriculum for K-2 students. The science curriculum was designed to provide students with access to science concepts and skills in an interactive and innovative environment, and teachers with embedded, aligned, and on-demand professional development. One of the most important—and challenging—steps in this evaluation was to select outcomes that were well-defined, measurable, and aligned to program activities, as well as relevant to program stakeholders. An additional challenge was to measure implementation given limited access to the classroom environment. This presentation will include a discussion of the process evaluators used to select appropriate implementation indicators and outcomes (teacher and student), design an evaluation approach, and craft data collection instruments. Although examples provided are specific to the K-2 science intervention, the best practices discussed are pertinent to all program and event evaluations.

Impact evaluation goes beyond implementation and outcome evaluation to inform whether a program is working or not. It requires a comparison group to inform what outcomes would have been in the absence of the intervention. As such, this presentation will also include a discussion of impacts, including how impacts are defined and measured, and some common challenges in evaluating program impact.