Measuring Numeracy: Validity and the Programme for the International Assessment of Adult Competencies (PIAAC)

Abstract

A tension raised in recent scholarship is that between numeracy as a social practice and numeracy as a functional skill set. Such frameworks for conceptualizing numeracy pose a challenge to assessment because what individuals do with numeracy is not the same as what individuals can do (or express) in an assessment setting. This study builds on work related to numeracy assessment through a validity examination of a portion of a well-known assessment: the OECD's Programme for the International Assessment of Adult Competencies (PIAAC). In following a path set out by standards for assessment, I ask: What does the PIAAC numeracy assessment claim to measure? What are the intended uses of the assessment? How are we to interpret scores with those uses in mind? And to what degree do evidence and theory support interpretations for those uses? The main finding from this work is that while score interpretations from the PIAAC numeracy assessment may be valid for the use of describing proficiency distributions for specific groups, the construct of interest - numerate behavior - is not what is measured. Moreover, evidence distinguishing what is measured from other constructs, such as the OECD's conception of literacy, is largely absent. This study contributes to existing literature on numeracy assessment by providing sources of evidence to consider in making judgments about validity for an assessment. It also suggests that, as scholars, we carefully hedge the ways that we talk about large-scale assessments, and in relation, what individuals can or cannot do based on results from such assessments.