Using Systems Thinking to Frame the Evaluation of a Complex Educational Intervention

Abstract:

InTeGrate (serc.carleton.edu/integrate/) is the geoscience component of NSF’s STEM Talent Expansion Center program. As such, it is a $10M, 5 year effort, with dual goals of improving undergraduate STEM education and addressing an important national challenge, which in InTeGrate’s case is environmental sustainability.

InTeGrate is very complicated, involving five PI’s, dozens of curriculum developers, scores of workshops and webinars, hundreds of faculty, and thousands of students. To get a handle on this complexity, the leadership team and evaluators are viewing project activities and outcomes through a system thinking lens, analogous to how geoscientists view the Earth system. For each major component of the project, we have a flowchart logic model that traces the flows of information, materials, influence, and people that are thought to result from project activities. As is to be expected in a complex system, individual activities are often influenced by multiple inputs and contribute to multiple outputs. The systems approach allows us to spot critical points in the system where evaluative probes are needed; for example, are workshops actually resulting in a flux of new people into roles of increased responsibility within InTeGrate as intended?

InTeGrate is permeated with opportunities for participants to engage in assessment, reflection and peer-review. From a systems perspective, this evaluative culture can be seen as an effort to create reinforcing feedback loops for processes that advance InTeGrate’s values. For example, assessment team members review draft instructional materials against a materials development rubric and coach developers through an iterative development cycle towards materials that embody InTeGrate’s priorities.

Of particular interest are flows of information or influence that may carry InTeGrate’s impact outward in space and time beyond activities that are directly funded by the project. For example, positive experiences during materials development may influence developers’ teaching practice such that they embed InTeGrate’s methods into their teaching of non-InTeGrate materials and advocate for InTeGrate methods on their campuses. Only if such influence pathways exist will InTeGrate be able to achieve national and enduring impact.