Download full text
(679.0Kb)
Citation Suggestion
Please use the following Persistent Identifier (PID) to cite this document:
https://nbn-resolving.org/urn:nbn:de:0168-ssoar-58064-7
Exports for your reference manager
Are Readability Formulas Valid Tools for Assessing Survey Question Difficulty?
[journal article]
Abstract Readability formulas, such as the Flesch Reading Ease formula, the Flesch-Kincaid Grade Level Index, the Gunning Fog Index, and the Dale-Chall formula are often considered to be objective measures of language complexity. Not surprisingly, survey researchers have frequently used readability scores as... view more
Readability formulas, such as the Flesch Reading Ease formula, the Flesch-Kincaid Grade Level Index, the Gunning Fog Index, and the Dale-Chall formula are often considered to be objective measures of language complexity. Not surprisingly, survey researchers have frequently used readability scores as indicators of question difficulty and it has been repeatedly suggested that the formulas be applied during the questionnaire design phase, to identify problematic items and to assist survey designers in revising flawed questions. At the same time, the formulas have faced severe criticism among reading researchers, particularly because they are predominantly based on only two variables (word length/frequency and sentence length) that may not be appropriate predictors of language difficulty. The present study examines whether the four readability formulas named above correctly identify problematic survey questions. Readability scores were calculated for 71 question pairs, each of which included a problematic (e.g., syntactically complex, vague, etc.) and an improved version of the question. The question pairs came from two sources: (1) existing literature on questionnaire design and (2) the Q-BANK database. The analyses revealed that the readability formulas often favored the problematic over the improved version. On average, the success rate of the formulas in identifying the difficult questions was below 50 percent and agreement between the various formulas varied considerably. Reasons for this poor performance, as well as implications for the use of readability formulas during questionnaire design and testing, are discussed.... view less
Keywords
survey research; questionnaire; development; survey; pretest; data quality
Classification
Methods and Techniques of Data Collection and Data Analysis, Statistical Methods, Computer Methods
Free Keywords
survey question difficulty; survey question design; survey question testing; readability formulas; question wording; survey pretesting; readability
Document language
English
Publication Year
2014
Page/Pages
p. 677-698
Journal
Sociological Methods & Research, 43 (2014) 4
DOI
https://doi.org/10.1177/0049124113513436
ISSN
1552-8294
Status
Postprint; peer reviewed
Licence
Deposit Licence - No Redistribution, No Modifications