| The Measurement of Factual Knowledge in Surveys 1 (2009) | |||||||||||||
Abstract | |||||||||||||
| It is increasingly apparent that factual knowledge is an important determinant of heterogeneity in public opinion in a broad range of areas (Delli Carpini and Keeter 1996; Miller 1998). Presidential voting, attitude to scientific research and risk behaviour are only some of the more prominent areas of social life that have been shown to depend crucially on an individual’s level of factual knowledge. The measurement of this key construct in surveys is, however, far from straightforward. In addition to the potentially adverse effect on response rate of administering test batteries, contextual cues and personality related ‘propensity to guess ‘ have both been argued to bias estimates of between group differences in factual knowledge (Nadeau and Niemi 1995; Mondak 2001). This has led Mondak to recommend the use of “closed-ended items on which DKs are discouraged ” in surveys, following best practice in the field of educational testing (2004 p.509). This strategy is, however, problematic in at least two respects: (1) the incentive structure of test-taking in surveys is entirely different from that found in educational contexts and (2) encouraging guessing amongst the truly ignorant will bias estimates of knowledge for the sample as a whole and reduce the reliability of summated scales. In this study we use random allocation of respondents to experimental conditions in a CATI survey to compare Mondak’s proposed format with the more conventional approach set out by Delli Carpini and Keeter (1996). In the latter condition, initial ‘don’t know ’ responses are subsequently probed for a ‘best guess’. Our results show that, across items, responses given to 1 The authors wish to thank the funders of the data collection for this research, BMRB International Limited. 1 requests for a ‘best guess ’ after an initial ‘don’t know ’ fare no better than the expectation under a ‘blind-guessing ’ strategy. We conclude that opinion researchers should be cautious about adopting Mondak’s recommendations for the design of factual knowledge items in surveys. | |||||||||||||
Publication details | |||||||||||||
| |||||||||||||