How Accurately Do Different Evaluation Methods Predict the Reliability of Survey Questions?

In: Journal of survey statistics and methodology: JSSAM, Volume 4, Issue 3, p. 362-381

Abstract

A diverse range of evaluation methods is available for detecting problems with survey questions. Ex-ante methods are relatively inexpensive because they do not require data collection. Other methods require data collection in either the laboratory or the field. This study evaluates the extent to which four ex-ante methods (the Question Understanding Aid [QUAID], the Survey Quality Profile [SQP], the Questionnaire Appraisal System [QAS], and expert review), one laboratory method (cognitive interviews), and two field methods (behavior coding and measurement of response latency) predict the reliability of answers to questions. The findings suggest (1) a multimethod approach to evaluation is appropriate given differences between the methods in their prediction of reliability and (2) a combination of a subset of the methods is most predictive of reliability: QAS, QUAID, SQP, cognitive interviewing, and expert review worked as well as the seven methods together.