Suchergebnisse

The most important national surveys of the general population for creating official statistics or public policymaking in many countries, including South Korea, are still conducted using face-to-face interviews with household members. Recently face-to-face surveys have faced threats to data quality from decreasing response rates and rising costs of in-person visits. The COVID-19 pandemic and associated lockdown exacerbated the situation for face-to-face surveys. Survey organizations suspended fieldwork or began to explore alternate means of collecting data. One alternative was a shift to telephone surveys; however, telephone interviews have encountered similar difficulties with declining response rates and increasing costs. Could a self-administered web survey be a viable alternative to interviewer-administered modes such as telephone interviews (CATI) or face-to-face interviews (CAPI)? Smartphones may offer opportunities not offered by other modes. We conducted a smartphone web survey using SMS invitations where a sample of cell phone numbers was selected by random digit dialing (RDD) and compared it with a large-scale national face-to-face survey (CAPI) where a sample of households was selected by stratified cluster sampling. The two surveys were conducted during the COVID pandemic in the second half of 2020. The coverage and sample representation of the smartphone web survey were comparable to that of the face-to-face survey. Despite the relatively small number of respondents, the quality of the smartphone web survey was sufficient to provide accurate data and compared favorably with the CAPI survey. The smartphone web survey yielded more reports of socially undesirable attitudes and behavior than the CAPI survey. The findings will guide researchers to explore new opportunities in establishing a web survey methodology that obtains data more conveniently, efficiently, accurately, and with less cost.

AbstractOwnership of a bank account is an objective measure and should be relatively easy to elicit via survey questions. Yet, depending on the interview mode, the wording of the question and its placement within the survey may influence respondents' answers. The Health and Retirement Study (HRS) asset module, as administered online to members of the Understanding America Study (UAS), yielded substantially lower rates of reported bank account ownership than either a single question on ownership in the Current Population Survey (CPS) or the full asset module administered to HRS panelists (both interviewer-administered surveys). We designed and implemented an experiment in the UAS comparing the original HRS question eliciting bank account ownership with two alternative versions that were progressively simplified. We document strong evidence that the original question leads to systematic underestimation of bank account ownership. In contrast, the proportion of bank account owners obtained from the simplest alternative version of the question is very similar to the population benchmark estimate. We investigate treatment effect heterogeneity by cognitive ability and financial literacy. We find that questionnaire simplification affects responses of individuals with higher cognitive ability substantially less than those with lower cognitive ability. Our results suggest that high-quality data from surveys start from asking the right questions, which should be as simple and precise as possible and carefully adapted to the mode of interview.

Internet-enabled smartphones and wireless communication technologies are opening new ways to conduct web-based self-administered data collection for academic or nonacademic research. Considering the relative advantages of self-administration such as the low cost, overall convenience, and collection of better data about sensitive topics, survey researchers are eager to explore conducting national web surveys of the general population via smartphones, especially if they can use probability-based random-digit-dialing (RDD) sampling methods. But questions about the feasibility of such surveys remain. We conducted an experiment using national samples drawn from an RDD wireless sampling frame to compare two administration methods: a smartphone web survey using SMS (text messages) invitations and a cell phone (smartphone or feature phone) survey through computer-assisted telephone interviewing (CATI). This study was conducted using the National Survey of Smoking and Health in South Korea, a country with a very high rate of smartphone ownership. The geographic and demographic representativeness of respondents in the smartphone web mode (self-administered mode) was similar to that of the cell phone CATI survey (interviewer-administered mode), although the completion rate in the former was nearly half that of the latter. A majority of demographic variables and measures of experiences, attitudes, and perceptions of cigarettes or smoking showed significant mode differences for both unweighted and weighted estimates. The total cost of the web survey was about one fifth that of the telephone survey. The results demonstrate the potential of a smartphone web survey as a stand-alone or primary mode of data collection, if carefully designed and implemented.

More and more respondents are answering web surveys using mobile devices. Mobile respondents tend to provide shorter responses to open questions than PC respondents. Using voice recording to answer open-ended questions could increase data quality and help engage groups usually underrepresented in web surveys. Revilla, Couper, Bosch, and Asensio showed that in particular the use of voice recording still presents many challenges, even if it could be a promising tool. This article reports results from a follow-up experiment in which the main goals were to (1) test whether different instructions on how to use the voice recording tool reduce technical and understanding problems, and thereby reduce item nonresponse while preserving data quality and the evaluation of the tool; (2) test whether nonresponse due to context can be reduced by using a filter question, and how this affects data quality and the tool evaluation; and (3) understand which factors affect nonresponse to open-ended questions using voice recording, and if these factors also affect data quality and the evaluation of the tool. The experiment was implemented within a smartphone web survey in Spain focused on Android devices. The results suggest that different instructions did not affect nonresponse to the open questions and had little effect on data quality for those who did answer. Introducing a filter to ensure that people were in a setting that permits voice recording seems useful. Despite efforts to reduce problems, a substantial proportion of respondents are still unwilling or unable to answer open questions using voice recording.

Several authors and software vendors advocate the benefits of auto forwarding in web surveys, but there is little empirical research on this approach. We experimentally tested automatic versus manual forwarding (MF) under different levels of cognitive effort. We manipulated information accessibility (IA; low vs. high) and consistency requirements (CRs; yes vs. no), along with auto forwarding (AF) versus MF in two studies conducted among students in Finland. We find that an AF survey takes less time to complete, but only for those completing a survey on personal computers or tablets; no time advantage is found for smartphone users. We also find that respondents in both AF and MF conditions return more often to items with higher cognitive burden (low IA or a CR). MF respondents change answers more often than AF respondents. AF appears to reduce straightlining slightly. We find no difference in response consistency between two behavioral items between AF and MF, but a slight advantage for AF for two attitude items. Finally, respondents reported more positive experiences with the AF version. Auto forwarding appears to be somewhat more efficient and easy to use but may decrease the quality of responses to cognitively demanding questions.

Much research has been done comparing grids and item-by-item formats. However, the results are mixed, and more research is needed especially when a significant proportion of respondents answer using smartphones. In this study, we implemented an experiment with seven groups ( n = 1,476), varying the device used (PC or smartphone), the presentation of the questions (grids, item-by-item vertical, item-by-item horizontal), and, in the case of smartphones only, the visibility of the "next" button (always visible or only visible at the end of the page, after scrolling down). The survey was conducted by the Netquest online fieldwork company in Spain in 2016. We examined several outcomes for three sets of questions, which are related to respondent behavior (completion time, lost focus, answer changes, and screen orientation) and data quality (item missing data, nonsubstantive responses, instructional manipulation check failure, and nondifferentiation). The most striking difference found is for the placement of the next button in the smartphone item-by-item conditions: When the button is always visible, item missing data are substantially higher.

"The use of open-ended questions in survey research has a very long history. In this paper, building on the work of Paul F. Lazarsfeld and Howard Schuman, we review the methodological uses of open-ended questions and verbatim responses in surveys. We draw on prior research, our own and that of others, to argue for increasing the use of open-ended questions in quantitative surveys. The addition of open-ended questions - and the capture and analysis of respondents' verbatim responses to other types of questions - may yield important insights, not only into respondents' substantive answers, but also into how they understand the questions we ask and arrive at an answer. Adding a limited number of such questions to computerized surveys, whether self- or interviewer-administered, is neither expensive nor time-consuming, and in our experience respondents are quite willing and able to answer such questions." (author's abstract)

"Text data from open-ended questions in surveys are difficult to analyze and are frequently ignored. Yet open-ended questions are important because they do not constrain respondents' answer choices. Where open-ended questions are necessary, sometimes multiple human coders hand-code answers into one of several categories. At the same time, computer scientists have made impressive advances in text mining that may allow automation of such coding. Automated algorithms do not achieve an overall accuracy high enough to entirely replace humans. We categorize open-ended questions soliciting narrative responses using text mining for easy-to-categorize answers and humans for the remainder using expected accuracies to guide the choice of the threshold delineating between 'easy' and 'hard'. Employing multinomial boosting avoids the common practice of converting machine learning 'confidence scores' into pseudo-probabilities. This approach is illustrated with examples from open-ended questions related to respondents' advice to a patient in a hypothetical dilemma, a follow-up probe related to respondents' perception of disclosure/privacy risk, and from a question on reasons for quitting smoking from a follow-up survey from the Ontario Smoker's Helpline. Targeting 80% combined accuracy, we found that 54%-80% of the data could be categorized automatically in research surveys." (author's abstract)

"This paper reports on a series of experiments to explore ways to use the technology of Web surveys to help respondents provide well-formed answers to questions that may be difficult to answer. Specifically, we focus on the use of drop-down or select lists and JavaScript lookup tables as alternatives to open text fields for the collection of information on prescription drugs. The first two experiments were conducted among members of opt-in panels in the U.S. The third experiment was conducted in the 2013 Health and Retirement Study Internet Survey. Respondents in each of the studies were randomly assigned to one of three input methods: text field, drop box, or JavaScript lookup, and asked to provide the names of prescription drugs they were taking. We compare both the quality of answers obtained using the three methods, and the effort (time) taken to provide such answers. We examine differences in performance on the three input format types by key respondent demographics and Internet experience. We discuss some of the technical challenges of implementing complex question types and offer some recommendations for the use of such tools in Web surveys." (author's abstract)

"A large number of findings in survey research suggest that misreporting in sensitive questions is situational and can vary in relation to context. The methodological literature demonstrates that social desirability biases are less prevalent in self-administered surveys, particularly in Web surveys, when there is no interviewer and less risk of presenting oneself in an unfavorable light. Since there is a growing number of users of mobile Web browsers, the authors focused our study on the effects of different devices (PC or cell phone) in Web surveys on the respondents' willingness to report sensitive information. To reduce selection bias, we carried out a two-wave cross-over experiment using a volunteer online access-panel in Russia. Participants were asked to complete the questionnaire in both survey modes: PC and mobile Web survey. The authors hypothesized that features of mobile Web usage may affect response accuracy and lead to more socially desirable responses compared to the PC Web survey mode. They found significant differences in the reporting of alcohol consumption by mode, consistent with our hypothesis. But other sensitive questions did not show similar effects. They also found that the presence of familiar bystanders had an impact on the responses, while the presence of strangers did not have any significant effect in either survey mode. Contrary to expectations, the authors did not find evidence of a positive impact of completing the questionnaire at home and trust in data confidentiality on the level of reporting. These results could help survey practitioners to design and improve data quality in Web surveys completed on different devices." (author's abstract)

"Survey researchers are making increasing use of paradata - such as keystrokes, clicks, and timestamps - to evaluate and improve survey instruments but also to understand respondents and how they answer surveys. Since the introduction of paradata, researchers have been asking whether and how respondents should be informed about the capture and use of their paradata while completing a survey. In a series of three vignette-based experiments, we examine alternative ways of informing respondents about capture of paradata and seeking consent for their use. In all three experiments, any mention of paradata lowers stated willingness to participate in the hypothetical surveys. Even the condition where respondents were asked to consent to the use of paradata at the end of an actual survey resulted in a significant proportion declining. The author's research shows that requiring such explicit consent may reduce survey participation without adequately informing survey respondents about what paradata are and why they are being used." (author's abstract)