Process data, the new frontier for assessment development: rich new soil or a quixotic quest?

Abstract

This paper presents the concepts and observations in the author's keynote address at the May 2019 "Opportunity versus Challenge: Exploring Usage of Log-File and Process Data in International Large-Scale Assessments" conference in Dublin, Ireland. This paper recaps briefly some key points that emerged at the December 2018 ETS symposium on process data in Washington, DC, and suggests some common terminology and concepts for facilitating future meaningful discussion. It then borrows some concepts from evolutionary biologists to take a critical look at process data and some of its ethical implications before turning to discuss major applications for and challenges of using process data and logfiles. The second half of this paper then maps out the terrain in which process data emerges as a "byproduct" of the shift from paper-and-pencil testing to digital-based assessments (DBA) and looks closely at how process data interacts with the development and operationalization of assessment systems. This interaction suggests new, fresh ways of improving national and international assessment system processes, and ultimately the measurement of cognitive skills, while venturing into the new frontiers opened with process data.