A Small Taste of Big Data: Strategies for Evaluating Reach, Impact, and Outcomes of Starchitect on Facebook

Abstract:

Starchitect (www.starchitect.net) is an online stellar and planetary evolution game developed for Facebook (though it can be played outside of Facebook as well). Funded through NASA and NSF, Starchitect was designed in part to explore the educational potential of a specific type of game design, in which players may only spend a few minutes with the game in a given sitting, but must come back repeatedly over days or weeks to make real progress. The intent was to determine if we could engage players and keep them "minds on" for longer periods of time, enhancing the educational value of the game. However this design leads to a variety of challenges and opportunities for the evaluation program. Players cannot easily be asked to "play through" the game while under observation, because the game is designed to be played over long periods. Even recruiting test players can be a challenge, since this particular game framework doesn’t appeal to everyone, and we need players to stay engaged long enough to have a meaningful experience. On the other hand, the game itself collects large amounts of data automatically, and can be strategically designed to collect even more. N in this case is very large (in excess of 11,000 players to date), and Facebook enhances the data by providing information about players that is not generally available to online activities (e.g., gender and age). All told this provides us with an intriguing double edged sword: challenges for traditional, in-person evaluation strategies, combined with new opportunities for online recruitment and automated data collection. This paper will discuss the strategies we've used to evaluate the project, including interviews, online surveys dynamically triggered by the game, targeted Facebook advertising, game data evaluation, embedded pre/post quizzes, and more.