How Effective Are Eye-Tracking Data in Identifying Problematic Questions?

Abstract

To collect high-quality data, survey designers aim to develop questions that each respondent can understand as intended. A critical step to this end is designing questions that minimize the respondents' burden by reducing the cognitive effort required to comprehend and answer them. One promising technique for identifying problematic survey questions is eye tracking. This article investigates the potential of eye movements and pupil dilations as indicators for evaluating survey questions. Respondents were randomly assigned to either a problematic or an improved version of six experimental questions. By analyzing fixation times, fixation counts, and pupil diameters, it was examined whether these parameters could be used to distinguish between the two versions. Identifying the improved version worked best by comparing fixation times, whereas in most cases, it was not possible to differentiate between versions on the basis of pupil data. Limitations and practical implications of the findings are discussed.