Assessing conceptual comparability of single-item survey instruments with a mixed-methods approach

Abstract

An increasing number of research projects and infrastructure services involve pooling data across different survey programs. Creating a homogenous integrated dataset from heterogeneous source data is the domain of ex-post harmonization. The harmonization process involves various considerations. However, chief among them is whether two survey measurement instruments have captured the same concept. This issue of conceptual comparability is a fundamental precondition for pooling different source variables to form a harmonized target variable. Our paper explores this issue with a mixed-methods approach. On the one hand, we use psychometric latent variable modeling by presenting several single-item wordings for social trust to respondents and then performing factor analytic procedures. On the other hand, we complement and contrast these quantitative findings with qualitative findings gained with an open-ended web probe. The combined approach gave valuable insights into the conceptual comparability of the eleven social-trust-related single-item wordings. For example, we find that negative, distrust-related wordings and positive, trust-related wordings should not be pooled into an integrated variable. However, the paper will also illustrate and discuss why it is easier to disprove conceptual comparability than fully prove it.