Handling mode effects

Mode effects can generate bias and need accounting for in analyses. This guide will help you handle mode effects and prevent bias.

What are mode effects?

CLS uses several survey ‘modes’ to collect data from cohort members, including:

• face-to-face (F2F) interviews
• telephone interviews
• video interviews
• web questionnaires.

Modes can differ between sweeps, and within sweeps.

For example: the first six sweeps of NCDS were carried out by F2F interview. The Age 42 Sweep was carried out by telephone interview.

More recent sweeps of all CLS studies have used sequential mixed-mode designs. This means cohort members are initially invited to participate by one mode, such as web interview, and then offered F2F interview if they do not respond.

This can increase survey response rates. However, participants’ answers to individual survey items can differ systematically between modes due to features of each mode.

For example, in F2F interviews, interviewers may help participants answer difficult questions. Meanwhile, in anonymous modes, such as online, participants may feel more comfortable revealing sensitive information.

Differences in responses between modes due to how items are measured are termed “mode effects”.

Research shows that mode effects particularly impact items related to socially sensitive topics such as health and finances, especially when comparing anonymous and non-anonymous modes.

What is the impact of mode effects?

Mode effects can generate bias in all types of analyses of mixed-mode data. For example:

Simple descriptive statistics

Simple descriptive statistics such as means and proportions may differ in mixed-mode surveys relative to single-mode datasets.

This also includes estimates of within-person change, where different modes are used over time. For example, a person may appear to have undergone cognitive decline, when in fact different modes were used to measure cognition at different time points.

Measures of association

Mode effects also impact measures of association, including when the aim is to estimate causal effects. Two variables subject to mode effects will be correlated simply due to shared mode.

For example: if participants in an anonymous mode are more likely to reveal poor health and poor finances than those in a non-anonymous mode, then health and finances will be correlated in the data even if they were not correlated in the population.

Mode selection

Variables subject to mode effects and those related to mode selection (differences between mode due to who is being measured) will also become correlated in mixed mode data.

For example: if people with better computer skills are more likely to respond by web than F2F, and due to being interviewed in an anonymous mode, are more likely to report poor mental health, computer skills will become associated with (measured) mental health simply due to the mixed-mode design. This is likely a widespread issue: participants generally self-select into mode to some extent.

How to handle mode effects

Controlling for mode

Mode is what is known as a “collider”’” variable. This comes about when users control for mode, for example by including an indicator variable for mode in analysis or stratifying and only analysing data from one mode. This will induce an association between factors that determine selection into mode.

An example:

Imagine a sequential web-then-F2F mixed-mode survey where people respond by web if they (a) have good computer skills (determining capacity to respond via web) or (b) have good (latent) mental health (determining likelihood of early response).

If we know someone responded by web (ie we condition upon it) and we know they have poor computer skills, then they must have good mental health, otherwise they would have responded via F2F. Conversely, if we know they responded by web and have poor mental health, then they must have good computer skills. Mental health and computer skills become (negatively) correlated merely because we have conditioned upon mode.

Measured mental health may be subject to mode effects, and conditioning upon mode will remove the bias from this. However, it may also generate (collider) bias due to mode selection.

It depends on the situation whether conditioning reduces or increases bias overall, but it is unlikely to remove bias completely.

To reduce collider bias, it may be possible to condition upon factors related to mode selection, as one controls for confounders to reduce confounding bias. However, these factors may be unknown, data on them may be unavailable or their measurement may also be subject to mode effects. In any case, this strategy does not help where the factors of substantive interest are those causing mode selection.

Quantitative bias analysis

Quantitative bias analysis spans various different methods designed to determine how robust results are to plausible (or implausible) levels of bias. In particular, results of the many mode-effects experiments that have been conducted can be used to ‘correct’ observed data as if it were collected from a single mode, removing the bias from mode effects before analyses are run.

For example, imagine a mixed-mode design that includes F2F and web data collection. We are interested in analysing mental health which we expect will be more truthfully reported in the anonymous web mode. We could use findings from previous experimental work comparing mental health variables between F2F and web data collection to “correct” the values observed F2F in our analysis in a sensitivity analysis.

Get support