Response Bias: Types and Prevention Methods

What is Response Bias?

Response bias (also known as survey bias) is defined as the tendency in respondents to answer untruthfully or inaccurately. It often occurs where participants are asked to self-report on behaviors, but can also be caused by poor survey design. By following the survey design best practices, you may be able to reduce the occurrence of bias.

Respondents may or may not be conscious of how they’re answering, but your data set will be affected just the same.

Unfortunately, it’s not always possible to prevent or detect bias. But this article will help you identify them where possible. 

Types of response bias

1. Social response bias

Also known as social desirability bias, respondents affected by this will often over-report on good behaviours and under-report on bad behaviours.

Here are a few things that may be misreported:

• Abilities and skills
• Personality
• Sexual behaviour
• Religion and spirituality
• Financial earning
• Unlawful behaviour

Respondents will answer in this way to appear more socially desirable, e.g. higher salary.

How to avoid it:

The best way to address this form of bias is to assure participants that their responses will be anonymous. In doing so, you’ll encourage more open and honest feedback.

2. Non-response bias

Also known as Participation Bias, this occurs where a survey sample is non-representative of the target population. In these cases, the opinions shared by those respondents are disproportionate to that of the larger population.

This will result in a biased set of results and may impact your research outcomes.

How to avoid it:

To reduce the risk of this bias occurring, share your survey across a range of platforms and to a diverse group as possible. E.g. social media, your website, via email.

3. Prestige bias

Many use the term ‘Prestige Bias’ to mean Social Desirability Bias, but we believe it to be its own form of survey bias. It arises where survey respondents are asked about their social, educational and financial statuses, and will incur inflated responses.

Respondents may want to be perceived to have:

• more social power
• an advanced level of education
• a more favourable financial situation

Respondents affected by prestige bias will differ in their responses, as what is considered prestigious changes from culture to culture.

How to avoid it:

Neutrally worded questions will reduce the occurrence of Prestige Bias, but not eliminate it completely. Wanting to be thought of as prestigious is a staple of the human condition.

4. Order effects

Response bias can be caused by the order of your questions.

For example, if you ask employees to detail issues with their line manager before you ask how happy they are in their role, their answer to the second question will be influenced by their first response. If you reversed the order of those questions, their answer may well change.

There are two kinds of effects this type of bias has:

Contrast Effects: The order of questions results in greater differences between respondent answers.

Assimilation Effects: The order of questions results in answer selections becoming more similar between respondents.

How to avoid it:

This is one of the most difficult forms of response bias to avoid. Ordering your questions perfectly takes time, and not all researchers have it to spare. 

However, if you run a pilot or test survey, you may be able to identify Order Effects and address the issues before you launch.

Recency bias

Recency Bias concerns those respondents who simply pick the last answer they read. Most often, respondents who show this tendency have already disengaged from your survey.

How to avoid it:

The best way to avoid this is to ensure your survey design is up to scratch. But, occasionally respondents will disengage regardless of your hard work.

You can address this type of response bias by randomizing the order of your answer options. This won’t stop it occurring, but it will distribute the bias evenly between answers.

5. Hostility bias

Asking respondents about unpleasant memories or negative experiences can provoke them into becoming hostile. Examples of this would be questions concerning divorce, debt, and death.

How to avoid it:

Avoid sensitive subjects or those which may elicit negative/ hostile responses, if possible.

Alternatively, you could explain why you’re asking those questions and how you’ll use the data. This will prepare respondents for your line of questioning, reducing the shock factor of sensitive questions.

6. Satisficing

Satisficing is a combination of ‘satisfy’ and ‘suffice’, meaning: ‘what is sufficient to obtain a satisfactory outcome’.

I.e. Respondents who only meet the minimum requirements of a survey; which is to submit a response.

Respondents who display this form of response bias are likely to leave questions unanswered or to answer dishonestly.

Types of satisficing

Speed runs

Some respondents speed through surveys without paying attention to your questions.

These instances tend to occur when an incentive is being offered for participating in research. Respondents may rush through you questions just for a chance to win the prize.

However, it could also be a result of survey fatigue, where participants have become fatigued and rush through to finish.

How to avoid it:

Think carefully about whether to offer an incentive and ensure your survey design is up to scratch.

Self-selection bias

This concerns those respondents who intentionally inject themselves into a study. It’s similar to non-response bias as the respondent pool elicits a different set of responses to those who aren’t responding.

How to avoid it:

Restrict access to your survey by using a password or distributing directly to your desired respondents.

Non-differentiation (straight lining)

Where scaled questions are concerned, there’s a risk of respondents failing to differentiate between answer choices. In these cases, they may give identical, or similar, responses to all questions using the same scale.

E.g. If all your scales are measured ‘Very Bad’ to ‘Very Good’, they may choose ‘Good’ for every question.

How to avoid it:

If you need to use scaled questions repeatedly, reverse the scale for each question. This will force respondents to read the answers carefully each time, or at least evenly distribute the bias.

E.g.

Scale 1: Strongly Disagree – Strongly Agree

Scale 2: Strongly Agree – Strongly Disagree

Or you could mix up the question types being used for each question. This will keep respondents on their toes.

Acquiescence bias

This is where respondents only select positive answer choices. It is also known as ‘yea-saying’. E.g. ‘yes’ or ‘strongly agree’.

Neutral answer selection

This concerns respondents who continually select neutral answer options. These include options such as ‘Don’t Know’, ‘N/A’, and ‘No opinion’.

Extreme responding

This mostly concerns scaled questions, where options such as ‘Strongly Agree’ and ‘Strongly Disagree’ are available. Respondents may simply choose the extreme answer options for each scale.

It’s often a result of leading questions or loaded words, making people feel they need to fully agree or disagree. 

How to avoid it:

Ensure your question wording is as neutral as possible.

Primacy effects

This pertains to respondents that select the first available answer option for each question.

How to avoid it:

By randomising the order of your answer options, you’ll decrease the number of times one answer can be chosen. In doing so, you will stop your survey results being too unfairly weighted towards one option.

7. Sponsorship bias

When respondents are aware of survey sponsorship or branding, their perception of that organisation can influence their responses.

How to avoid it:

If you have to include the sponsor branding, then save it for the thank you page. This way respondents will still see the branding, but it won’t influence their choices.

8. Stereotype bias

Personal questions (such as technical skill, nationality, gender, and age) may reinforce or trigger stereotypes. This will lead respondents to fulfil that stereotype in their response.

E.g. If a question implies that younger people are better with technology, that idea will influence their responses.

How to avoid it:

Frame all questions and answers as neutrally as possible. Allow participants to explore their own ideas or opinions on the subject.

9. Recall bias

How we remember things isn’t always reliable or accurate. As the way we feel or think about something changes over time, our recollection of the past can become warped and we tend to align those memories with our current thoughts and beliefs.

E.g. If asking for event feedback too long after the event took place, attendees may not be able to remember what aspects were most useful or impactful.

How to avoid it:

If you need to enquire about past events, ensure one of two things. Either, that the event in question is frequently occurring or that it took place in the recent past.

10. Demand characteristics

This is where respondents alter their behavior to align with how they believe the ideal research participant should respond.

This is a large topic in itself, so you may want to read more about demand characteristics.

How to avoid it:

Distance participants from the hypothesis/ aims of your research if you believe it could elicit this type of bias.