Evaluation of NASA's Mars Public Engagement Program

Abstract:

From 2009-2014, NASA’s Mars Public Engagement (MPE) Program developed and implemented project-level logic models and associated impacts and indicators tables using the NSF’s “Framework for Evaluating Impacts of Informal Science Education Projects” (Friedman, 2008) as a key guiding document. This Framework was selected given the national-expert-level evaluation committee who synthesized evaluation in a way that allows project-to-project comparisons in key areas of measurable change, while also allowing variation for appropriate project-specific measures and outcomes. These logic models, revisited and refined annually, provide guidance for all measures developed, tested, and implemented with MPE projects, including the Mars Student Imaging Project (MSIP), the Imagine Mars Project, and Mars Educator Professional Development. Project questionnaires were developed, tested, refined, retested, and finalized following standard procedures outlined in Converse & Presser (1986), Dillman, Smyth, & Christian (2009), Krosnick & Presser (2010), and Presser, et al. (2004). Interview questions were drafted, reviewed by project staff, and revised following established interview question development guidelines (e.g., Kvale, 1996; Maxwell, 2005; Maykut & Morehouse, 1994; Strauss & Corbin, 1998). For MSIP final projects, a rubric guided by Lantz (2004) was developed to evaluate systematically the quality and completeness of the final projects.

We will discuss our instruments as well as the important issue of nonresponse error, which is relevant to a wide range of NASA programs because most data is collected from customers who are voluntary participants, as opposed to grantees who must report as a condition of their grant. NASA programs that consider data and report results from voluntary samples must be cautious about claims or decisions based on those data. We will discuss the ways in which we consider and address this challenge.