Qualitative and quantitative surveys

While most INED surveys are quantitative, qualitative methods are now used regularly at different stages in research projects, usually to complement and dovetail with the quantitative approach. There are a rich variety of qualitative methods, but they are often extremely time-consuming in terms of both data collection and analysis.

Upstream of the quantitative survey

Qualitative pre-surveys

Quantitative survey preparation often involves a semi-structured interview stage aimed at facilitating and enhancing questionnaire design. These interviews provide a means to explore the research topic in a broader way that takes into account respondents’ representations, attitudes and practices.

The interviews are recorded and may later be transcribed for statistical text analysis or thematic content analysis. Untranscribed material may be listened to or summarized for the purpose of identifying the main interview components as they relate to the planned questionnaire outline. This stage also brings to light dimensions that may not have been initially planned for. The interviewee sample is small (15 to 25 respondents) and selected to represent the broadest possible range of respondent profiles.

Observation in the field

A period of field observation during which researchers get a detailed picture of the context in which the interviewing will take place is very useful for designing the data collection tools (especially the questionnaire itself), anticipating difficulties, and adapting data collection protocol and methods (sampling, respondent contact) to the different situations that will be encountered.

 Testing a questionnaire

 Cognitive interviews

Preliminary cognitive interviews can help to pinpoint any problems respondents may encounter in the interview situation. These interviews, conducted with a small, diverse respondent sample, help eliminate points of misunderstanding and any gaps between the survey designers’ intentions and the way(s) respondents understand questions.

As the cognitive process involved in answering a survey question unfolds, problems in the following areas may be identified:

•  respondent’s understanding or interpretation of a question;
• effective extraction of the requested information;
• respondent’s assessment of how to answer (self-censorship, motivation, social desirability effect, etc.);
• matching respondents’ answers to questionnaire response categories and scales.

The main techniques used in cognitive interviewing are “think-aloud” and “verbal probing.”

• In “think-aloud,” the respondent is instructed to do just that; i.e., to describe explicitly the mental process he/she is engaged in, from question comprehension all the way to how he/she arrives at the response given.
• In “verbal probing” the interviewer asks a short battery of complementary questions designed to gain insight into how the question is understood and how the respondent reaches and justifies his/her response.

Semi-structured interviews

Conducting semi-structured interviews with individuals who have already answered or will be answering a quantitative questionnaire makes it possible to compare data from the two types of questioning, which in turn may reveal flaws or oversights in the quantitative questionnaire.

Downstream of a quantitative survey

Qualitative post-surveys

Semi-structured interviews conducted after a quantitative survey provide insight into dimensions that cannot easily be covered in a closed-ended questionnaire and/or they enable researchers to obtain further details on a particular dimension of the survey topic or a particular quantitative survey finding. Usually (though not always), post-survey interview respondents are recruited from among persons who agreed to be contacted again for a further interview after answering the quantitative survey questionnaire. The sample is constructed to ensure maximum consistency with respect to the research question and respondents’ sociodemographic characteristics, and takes place in parallel with the analysis, until sample “saturation.”

Post-survey interviews are recorded and then usually transcribed for textual analysis, subject content analysis, etc.

Respondents are generally recruited using the snowball method (initial respondents indicate other possible contacts) and/or by selecting individuals from an existing database (e.g., for quantitative post-surveys) or by outsourcing the task to a specialized company. The last option is more costly than the others and precludes adjusting the sampling process to take account of changes in respondent profiles over the course of data collection and analysis.