Overcoming bias in research and product design

I’ve been thinking about bias and how prevalent it is in every decision we make. It’s almost impossible to avoid.

Now, I don’t think that we’d necessarily want to remove bias completely from our life, given the amount of information we are required to process every second of every day. In fact, I suspect that bias might protect us in some cases.

Maybe a better word for this “good” kind of bias is intuition.

For the most part though, I’m interested in ways to mitigate personal and cognitive biases, especially when it comes to researching and designing products. This might even be the most important role a researcher plays within a UX team. We are here to challenge our assumptions, rather than defend our ideas.

One good way to start avoiding bias is to recognize it in the first place.

I thought it might be helpful to compile a list of biases that I’ve encountered while researching new features at Buffer. It is not even close to being exhaustive, so I’d love to hear if there any other types of bias that you’ve noticed in your day to day work!

Confirmation Bias

You’ve probably heard of this one, because it’s so common. It’s a cognitive bias where evidence is consciously or subconsciously collected in order to support a hypothesis. I’ve noticed this is more likely to happen when there is pressure to do research quickly. For example, I might need to quickly validate a prototype that we’ve built, and therefore only speak to a couple of friends, asking few questions, without necessarily challenging my assumptions.

How to mitigate it: Make sure your research covers a diverse set of participants. Avoid asking leading questions. Set yourself a goal to invalidate your hypotheses and recognize when your ego is influencing your work.

Hindsight Bias

This is our natural tendency to create a false narrative to explain an unforeseen outcome. It’s problematic because our made-up explanation influences how we prepare for future events. Hindsight bias happens all the time because we’re really bad at dealing with ambiguity. As a researcher, my job is to understand “why?”, so it can feel like I’m doing a poor job when I have to admit I don’t have all the answers. The need to explain things leads to hindsight bias.

How to mitigate it: Get comfortable with admitting that you don’t know why something happened. This will also help you with the ambiguity effect.

Ambiguity Effect

The ambiguity effect leads us down the clearest path, rather than the right path, because we are afraid of the unknown. This is a really tough one, especially for researchers and product managers, when there is pressure to justify decisions with limited information.

How to mitigate it: Avoid researching just a single design or solution and gather as much evidence as possible for potential alternatives.

Loss Aversion

Loss aversion is commonly described in behavioural economics. It’s when we irrationally fear losing something that we value, even if it means gaining something of equal or greater value. It applies to design too; for example by removing a feature, button or toggle you might lose some minor functionality but enhance the overall usability of a product.

How to mitigate it: Use a lean methodology when designing products and as with mitigating confirmation bias, push yourself to invalidate new features and ideas.

Anchoring Effect and Clustering Illusion

I’ve put these biases together because they both relate to small sample sizes. The anchoring effect is our tendency to lend too much weight to the first piece of information we receive. The clustering illusion is when we mistake random coincidence in a small sample for a trend in the population at large. I’ve noticed this sometimes happens when research and testing is limited to less than five participants.

How to mitigate it: Be disciplined about research sample sizes if you want to draw robust conclusions.

Framing Effect

One thing that I’ve had to learn is to be totally objective when communicating research results to work friends, even if I’m worried that it might hurt someone’s feelings. I’m lucky to work in a team “no ego doers”, in which we are encouraged to accept feedback, good or bad, but even so; I sometimes find myself tempted to put a positive “spin” on findings. Doing so creates a “framing effect”, where research is interpreted differently depending on how it is presented.

How to mitigate it: Do your research with a scout mindset; separate yourself from your ideas and recognize when ego is affecting your objectiveness.

Psychological Distance

When you are creating something, at what point do you show it to someone else for feedback? When it is an idea in your head? When it is a working skeleton? When it is the finished product? As a researcher, I want to test our assumptions as early as possible, but this comes at a price; psychological distance. This bias occurs when we make predictions and do research based on abstract ideas that require participants to use their imagination.

How to mitigate it: Continually test assumptions and hypotheses all the way through the design process. Create working prototypes that are usable (without necessarily being beautiful!) and cross reference qualitative research with usage data, including pre-orders and revenue if possible.

Bandwagon Effect

Although established design patterns can help reduce user confusion and anxiety, the tendency to design something in a certain way just because others are doing it can also be dangerous. For example, the “hamburger menu” was very popular in mobile design for a time but has since been found to have quite a few problems.

How to mitigate it: Focus on your customers, rather than your peers.

What biases have you encountered?

I’m not a psychologist or statistician but I’m really keen to learn more about different biases and how to identify them!

Here’s a few others that have piqued my interest:

• Information bias; the tendency to seek additional information even if it is irrelevant to a decision.
• Pro-innovation bias; the tendency to favor new solutions over current solutions.
• Contrast effect; giving too much weight to something that is unexpected.

And here are some fun and interesting books to read:

• The Black Swan by Nassim Nicholas Taleb
• Thinking, Fast and Slow by Daniel Kahneman
• You Are Not So Smart by David McRaney

Feel free to share any of your own experiences or thoughts in the comments below. You can also connect with me on Twitter. I’d love to hear from you!