New Insights on the Cognitive Processing of Agree/Disagree and Item-Specific Questions

Abstract

In empirical social research, using questions with an agreement scale, also known as agree/disagree (A/D) questions, is a popular technique for measuring attitudes and opinions. Methodological considerations, how-ever, suggest that such questions require effortful cognitive processing and are prone to response bias, such as acquiescence. Therefore, many researchers recommend the use of item-specific (IS) questions, which are based on tailored response categories and seem to imply less response burden. In this study, we investigate the cognitive processing of A/D and IS questions in web surveys, using eye-tracking methodology. On the basis of recordings of respondents’ eye movements, we are able to draw conclusions about how respondents process survey questions and to evaluate how they process information. Our results indicate that IS questions require deeper processing than A/D questions. Interestingly, the eye-tracking data reveals that this phenomenon is only observable for the response categories but not for the question stems; this indicates that the stems do not differ in terms of cognitive effort. We therefore argue that the observed differences are directly attributable to a more intensive processing of the IS response categories. Practically speaking, this additionally indicates a more thoughtful processing of the response categories and, thus, might lead to more well-considered and appropriate responses.