Achieving quality: the importance of collecting auxiliary information

Sampling errors are measured by means of variance indicators, which in turn may be calculated analytically or, for complex sampling plans, through replication techniques such as “jackknife” or “bootstrap.” To determine variance indicators, detailed information on the sampling plan is required.

Calculating response rates

Non-response rates should be calculated with the reasons for non-response in mind (non-contact, refusal, interview not possible, etc.) and/or for each population subgroup. These calculations require information on both respondents and non-respondents, obtained from the sampling frame or acquired through the data collection process (contact order, etc.). For international recommendations on these questions, see the American Association for Public Opinion Research(AAPOR) website

Data and paradata consistency

There are several possible causes of measurement errors. Interviewers’ feedback is useful in assessing data quality as they provide information on which questions “went down well” with respondents and which raised difficulties. From a quantitative perspective, internal data consistency may be studied; collected data may be compared with outside information; “interviewer effects” may be studied by matching interviewer characteristics with the data collected, etc. It is therefore very useful to collect contextual information or paradata (contact order, interview date and time, etc.).

Technological tools should be used whenever appropriate. For example, computer-collected data (CAPI) includes a record of how long respondents take to answer difficult questions. Such variables can then be used to analyse response quality.

Interviewer influence

Lastly, to study interviewer effects it is crucial to collect information on interviewers—at least their sex, age, educational level and surveying experience. It is also important to keep a record of which (anonymously designated) interviewer conducted which interview. Interviewer effects are inevitable, but it is preferable to avoid excessive variation or atypical cases. Information on interviewers can therefore be used to identify atypical interviews and correct or eliminate them, or else to formulate recommendations for future surveys.

Survey quality

Survey quality is determined at every stage of the process, from design to implementation and analysis. There are no absolutely fixed criteria for judging that quality. Certain methodological aspects or areas of investigation (including analysis and interpretation) may be particularly effective while others are less so. To judiciously assess survey quality, it is important to have a clear record of all the different survey stages.

Lastly, the term “representative” is often used to describe sample surveys. Representativeness usually refers to a sample’s ability to correctly “reflect” the population that it is assumed to reproduce in miniature. It is on this basis that researchers can confidently extrapolate from their findings and apply them (often after data adjustment) to the target population as a whole. Here again, caution is in order: representativeness criteria are necessary but often not sufficient, and it would be wrong to claim that a sample can be perfectly representative.