Suchergebnisse

Near the end of a web survey respondents are often asked whether they have additional comments. Such final comments are usually ignored, partially because open-ended questions are more challenging to analyze. A random sample of final comments in the LISS panel and Dutch immigrant panel were categorized into one of nine categories (neutral, positive, multiple subcategories of negative). While few respondents chose to make a final comment, this is more common in the Immigrant panel (5.7%) than in the LISS panel (3.6%). In both panels there are slightly more neutral than negative comments, and very few positive comments. The number of final comments about unclear questions was 2.7 times larger in the immigrant panel than in the LISS panel. The number of final comments complaining about survey length on the other hand was 2.7 times larger in the LISS panel than in the immigrant panel. Researchers might want to consider additional pretesting of questions when fielding a questionnaire in the immigrant panel.

Abstract
Manual coding of text data from open-ended questions into different categories is time consuming and expensive. Automated coding uses statistical/machine learning to train on a small subset of manually-coded text answers. Recently, pretraining a general language model on vast amounts of unrelated data and then adapting the model to the specific application has proven effective in natural language processing. Using two data sets, we empirically investigate whether BERT, the currently dominant pretrained language model, is more effective at automated coding of answers to open-ended questions than other non-pretrained statistical learning approaches. We found fine-tuning the pretrained BERT parameters is essential as otherwise BERT is not competitive. Second, we found fine-tuned BERT barely beats the non-pretrained statistical learning approaches in terms of classification accuracy when trained on 100 manually coded observations. However, BERT's relative advantage increases rapidly when more manually coded observations (e.g., 200–400) are available for training. We conclude that for automatically coding answers to open-ended questions BERT is preferable to non-pretrained models such as support vector machines and boosting.

The data can be used to study the classification of open-ended questions. It is not suitable for subject matter analysis becase it is not linked to any other data.

The question asked: "Joe's doctor told him that he would need to return in two weeks to find out whether or not his condition had improved. But when Joe asked the receptionist for an appointment, he was told that it would be over a month before the next available appointment.
What should Joe do? "
The answer was categorized by two coders into one of 4 categories: destructive, passive, somewhat proactive, proactive. The coding book is supplied.
The variables include the categorization by the two coders, as well as the resolved category.

Abstract
Text answers to open-ended questions are typically manually coded into one of several codes. Usually, a random subset of text answers is double-coded to assess intercoder reliability, but most of the data remain single-coded. Any disagreement between the two coders points to an error by one of the coders. When the budget allows double coding additional text answers, we propose employing statistical learning models to predict which single-coded answers have a high risk of a coding error. Specifically, we train a model on the double-coded random subset and predict the probability that the single-coded codes are correct. Then, text answers with the highest risk are double-coded to verify. In experiments with three data sets, we found that this method identifies two to three times as many coding errors in the additional text answers as compared to random guessing, on average. We conclude that this method is preferred if the budget permits additional double-coding. When there are a lot of intercoder disagreements, the benefit can be substantial.

Text answers to open-ended questions are often manually coded into one of several predefined categories or classes. More recently, researchers have begun to employ statistical models to automatically classify such text responses. It is unclear whether such automated coders and human coders find the same type of observations difficult to code or whether humans and models might be able to compensate for each other's weaknesses. We analyze correlations between estimated error probabilities of human and automated coders and find: 1) Statistical models have higher error rates than human coders 2) Automated coders (models) and human coders tend to make similar coding mistakes. Specifically, the correlation between the estimated coding error of a statistical model and that of a human is comparable to that of two humans. 3) Two very different statistical models give highly correlated estimated coding errors. Therefore, a) the choice of statistical model does not matter, and b) having a second automated coder would be redundant.

Abstract
Asking people about their occupation is common practice in surveys and censuses around the world. The answers are typically recorded in textual form and subsequently assigned (coded) to categories, which have been defined in official occupational classifications. While this coding step is often done manually, substituting it with more automated workflows has been a longstanding goal, promising reduced data-processing costs and accelerated publication of key statistics. Although numerous researchers have developed different algorithms for automated occupation coding, the algorithms have rarely been compared with each other or tested on different data sets. We fill this gap by comparing some of the most promising algorithms found in the literature and testing them on five data sets from Germany. The first two algorithms we test exemplify a common practice in which answers are coded automatically according to a predefined list of job titles. Statistical learning algorithms—that is, regularized multinomial regression, tree boosting, or algorithms developed specifically for occupation coding (algorithms three to six)—can improve upon algorithms one and two, but only if a sufficient number of training observations from previous surveys is available. The best results are obtained by merging the list of job titles with coded answers from previous surveys before using this combined training data for statistical learning (algorithm 7). However, the differences between the algorithms are often small compared to the large variation found across different data sets, which we ascribe to systematic differences in the way the data were coded in the first place. Such differences complicate the application of statistical learning, which risks perpetuating questionable coding decisions from the training data to the future.

Open-ended questions in surveys are often manually coded into one of several classes (or categories). When the data are too large to manually code all texts, a statistical (or machine) learning model must be trained on a manually coded subset of texts. Uncoded texts are then coded automatically using the trained model. The quality of automatic coding depends on the trained statistical model, and the model relies on manually coded data on which it is trained. While survey scientists are acutely aware that the manual coding is not always accurate, it is not clear how double coding affects the classification errors of the statistical learning model. We investigate several budget allocation strategies when there is a limited budget for manual classification: single coding versus various options for double coding where the number of training texts is reduced to maintain the fixed budget. Under fixed budget, double coding improved prediction of the learning algorithm when the coding error is greater than about 20–35%, depending on the data. Among double-coding strategies, paying for an expert to resolve differences performed best. When no expert is available, removing differences from the training data outperformed other double-coding strategies. When there is no budget constraint and the texts have already been double coded, all double-coding strategies generally outperformed single coding. As under fixed budget, having an expert to solve disagreement in training texts improves accuracy most, followed by removing differences.

AbstractMany undergraduate degrees require the completion of an introductory statistics course, but it is unclear to what extent taking a statistics course improves statistical literacy. We conducted an online survey with a simple random sample of undergraduate students at the University of Waterloo, Canada. We then compared students who have completed at least one statistics course to those who have completed none and found that taking a statistics course did not improve statistical literacy on the questions asked (causation versus correlation, margin of error, and others). Introductory statistics courses may want to address statistical literacy as a learning outcome more explicitly for a better understanding of public policies.

"Near the end of a web survey respondents are often asked whether they have further comments. Such final comments are usually ignored, in part because open-ended questions are challenging to analyse. We explored whether final comments are associated with next-wave attrition in survey panels. We categorized a random sample of final comments in the Longitudinal Studies for the Social Sciences (LISS) panel and Dutch Immigrant panel into one of eight categories (neutral, positive, six subcategories of negative) and regressed the indicator of next-wave attrition on comment length, comment category and socio-demographic variables. In the Immigrant panel we found shorter final comments (55 words) with decreased next-wave attrition relative to making no comment. Comments about unclear survey questions quadruple the odds of attrition and 'other' (uncategorized) negative comments almost double the odds of attrition. In the LISS panel, making a comment (vs. not) and comment length are not associated with attrition. However, when specifying individual comment categories, neutral comments are associated with half the odds of attrition relative to not making a comment." (author's abstract)

"Straightlining, an indicator of satisficing, refers to giving the same answer in a series of questions arranged on a grid. We investigated whether straightlining changes with respondents' panel experience in two open-access Internet panels in the Netherlands: the LISS and Dutch Immigrant panels. Specifically, we considered straightlining on 10 grid questions in LISS core modules (7 waves) and on a grid of evaluation questions in both the LISS panel (150+ waves) and the Dutch immigrant panel (50+ waves). For both core modules and evaluation questions we found that straightlining increases with respondents' panel experience for at least three years. Straightlining is also associated with younger age and non-western 1st generation immigrants. Where straightlining was a plausible set of answers, prevalence of straightlining was much larger (15-40%) than where straightlining was implausible (<2% in wave 1)." (author's abstract)

"Text data from open-ended questions in surveys are difficult to analyze and are frequently ignored. Yet open-ended questions are important because they do not constrain respondents' answer choices. Where open-ended questions are necessary, sometimes multiple human coders hand-code answers into one of several categories. At the same time, computer scientists have made impressive advances in text mining that may allow automation of such coding. Automated algorithms do not achieve an overall accuracy high enough to entirely replace humans. We categorize open-ended questions soliciting narrative responses using text mining for easy-to-categorize answers and humans for the remainder using expected accuracies to guide the choice of the threshold delineating between 'easy' and 'hard'. Employing multinomial boosting avoids the common practice of converting machine learning 'confidence scores' into pseudo-probabilities. This approach is illustrated with examples from open-ended questions related to respondents' advice to a patient in a hypothetical dilemma, a follow-up probe related to respondents' perception of disclosure/privacy risk, and from a question on reasons for quitting smoking from a follow-up survey from the Ontario Smoker's Helpline. Targeting 80% combined accuracy, we found that 54%-80% of the data could be categorized automatically in research surveys." (author's abstract)

AbstractLow-wage Latina/o workers are subject to an array of workplace abuses. This study focuses on whether educational attainment may moderate inequality in knowledge or claims-making across individuals with different legal statuses. This question is motivated by research which, while highlighting the role of education in promoting civic and political engagement, has not examined the interaction between education and legal status for worker claims-making. We draw from the 2008 Unregulated Work Survey, which is representative of the 1.64 million low-wage workers in Chicago, Los Angeles, and New York, three of the largest immigrant destinations in the United States. Using the Latina/o subsample, we test whether education impacts workers' procedural knowledge of the claims process, as well as their actual claims-making behavior, across four categories of workers: U.S.-born citizens, naturalized citizens, documented noncitizens, and undocumented noncitizens. Our findings reveal that all noncitizens have lower levels of procedural knowledge about how to file a complaint with the government, compared to citizens, across educational levels. However, when it comes to claims-making, we find that education has significant positive impacts for noncitizen workers, especially the undocumented. Our results suggest that education may improve the workplace agency of even the most marginalized workers.