In Search of the Optimal Mode for Mobile Phone Surveys in Developing Countries: A Comparison of IVR, SMS, and CATI in Nigeria

Abstract

Mobile phone surveys are increasingly prevalent in low- and middle-income countries. The main modes include computer-assisted telephone interviewing (CATI), interactive voice response (IVR), and short message service (SMS, or text messaging). But there is surprisingly little research to guide researchers in selecting the optimal mode for a particular survey. To address this gap, this study compares cross-sectional CATI, IVR, SMS, and face-to-face (FTF) surveys of the general population in Nigeria. We ask four research questions: (1) What are production and response rates to CATI, IVR, SMS, and FTF surveys? (2) How representative (age, gender, education, marital status, literacy, household assets, urbanicity) are CATI, IVR, and SMS respondents relative to FTF respondents? (3) Can IVR and SMS provide an unbiased estimate of voting behavior? If there is bias, to what extent can weights reduce bias? (4) How does the cost and time differ across mobile phone survey modes? We find that FTF had the highest response rate (99%), followed by CATI (15%), IVR (3%) and SMS (0.2%). All mobile phone modes had substantial deficiencies with representativeness: mobile phones underrepresented women, older people, the less educated, and people in rural areas. There were differences in representativeness among mobile phone modes, but differences were relatively small and inconsistent. Both SMS and IVR produced biased estimates of voting relative to official statistics – but SMS was less biased than IVR. Weighting SMS and IVR data for demographic characteristics did not reduce bias. With regard to cost, we find that CATI is the most expensive mobile phone survey mode. For a survey of 3,000 completes, IVR is 43% the cost of CATI, and SMS is 24% the cost of CATI. SMS is significantly less expensive than IVR. We discuss the implications of these results for research and practice.