A story of bad research

Once upon a time at Innovatech, a team of enthusiastic product developers set out to revolutionize the social networking app by adding a new feature. Eager to gather insights, they decided to conduct their own user research.

Lacking formal training in research methods, they hastily recruited friends and family as participants, leading to a biased sample that didn't represent their diverse user base. Their survey questions and interview scripts were designed to validate their initial design ideas, inadvertently steering participants towards confirming their assumptions.

As they analyzed the data, their lack of expertise became evident — they misinterpreted nuanced responses and overlooked complexities in user behaviour and their underlying motivations. They presented their findings with confidence, emphasizing positive feedback that aligned with their expectations while neglecting dissenting voices.

Months after launch, the feature failed to gain traction. The team struggled to understand what went wrong...

This seems to be a common situation that gives research a bad rep. Most people think that it's straightforward and can be done by anyone. In the end how complicated can it be to talk to others and get feedback?

The most important aspects to look out for in research to avoid creating bad insights that lead to meaningless results and wrong decisions are:

Defining clear objectives

Begin with clearly defined research objectives. This ensures that the research is focused and targeted. You can't find out everything all at once, research is chaotic and clear goals and research questions let us make it less so.

TIPS:

Be Specific: Outline what you want to learn. Instead of broad objectives like "improve user satisfaction," aim for specific goals such as "understand the main pain points users face during the onboarding process."

• Example: If you want to improve a feature, define the objective as "Identify usability issues in the new messaging feature for first-time users."

Prioritize: Determine which objectives are most critical to your product's success and focus on those first. You can't find out everything all at once. A study with your users will let you draw different insights than a study with potential users who haven't used your product yet.

• Example: Prioritize understanding what prevents potential users from using your product, and what considerations they have while choosing a solution to their problem.

Choose appropriate UX research methods

Select research methods that are appropriate for the research objectives and the nature of the data being collected.

TIPS:

Match Methods to Goals: Use surveys for quantitative data on user satisfaction, but opt for interviews or focus groups to explore user motivations and emotions.

• Example: If you want to find out the why, and nuances that will influence the communication and overall feel of the brand then don't do a survey, talk to people by running interviews or focus groups where you will be able to gather social cause and the collective thinking.

Combine Methods: Employ mixed methods to get a comprehensive view. Use quantitative data to identify trends and qualitative data to understand the reasons behind those trends.

• Example: Conduct usability testing to identify specific issues and follow up with focus groups to discuss potential solutions and gather deeper insights.

Avoiding bias or at least being aware of it.

Bias is inherent in human cognition and perception. Every ethnographer knows it's impossible to yield a fully objective research outcome. It's impossible because it has to be understood and conceptualised by a human mind with its own beliefs, limitations and experiences that influence its pattern recognition.

TIPS:

• Acknowledge Bias: Be aware of your own biases and how they might affect your research. You hold beliefs about not only your product's purpose but also why people might be using and behaving in a certain way. Think and write down what you assume to be true about people who will be using

  • Example: If you have a strong belief that people will want to use your product because it saves them time, try to validate it - Do they speak about time-saving, what language do they use if yes? Perhaps they haven't mentioned it at all because they care more about the price or trust in the platform they're using
    
    

• Learn about the psychology of design: Understand the biases & heuristics affecting the decision-making of people related to UX and let you influence user behaviour
  You can read about them here: 106 Cognitive Biases & Principles that Affect Your UX

  • Example: Tinder's pricing table shows the most expensive subscription plan first (the anchor) so that the other plans look inexpensive in comparison
    

Setting up the right criteria for participants.

The quality of the findings depends on the quality of the participants, if your SaaS is focused on financial fitness for home buying you won't really get much use of talking to digital nomads who value the flexibility of being able to rent homes

TIPS:

• Define Clear Criteria: Identify the characteristics that your participants must have. This includes demographics, user experience level, and specific behaviours relevant to your product.

  • Example: If your SaaS focuses on financial fitness for home buying, recruit participants who are actively seeking to buy a home within the next year.
    
    

• Screen Participants: Use screening questions to ensure that participants meet your criteria.

  • Example: Include questions in your recruitment survey to filter out digital nomads if they are not your target audience.

Transparency and Reproducibility

While UX research may not yield definitive truths like academic research transparent reporting allows for informed decision-making and iterative improvements in product design and user experience. This is the most boring and tiresome part for some but it's crucial for setting up a system that will work in the long run for your organisation.

TIPS:

• Detailed Documentation: Keep a thorough record of your research process, including participant recruitment, data collection methods, and analysis procedures.

  • Example: Maintain a research log that includes dates, participant demographics, and key findings from each session.
    
    

    
    

• Report Limitations: Acknowledge the limitations of your research and how they might affect your findings.

  • Example: If your sample size is small or not fully representative, note this in your report and discuss how it might impact the generalizability of your results.
    

    
    
    

• Use Collaboration Tools: Leverage tools that facilitate collaborative research and documentation.

  • Example: Use platforms like Dovetail for storing and analyzing qualitative data, and Miro for collaborative brainstorming and mapping research insights.
    
    

    
    

• Create a Research Repository: Maintain a centralized repository of all research findings for easy reference and to track progress over time.

  • Example: Use a tool like Airtable or Notion to organize and archive past research studies, including methodologies, findings, and lessons learned.
    
    

    
    

• Share Findings Transparently: Present your findings clearly, including both positive and negative feedback. Provide context for how conclusions were drawn.

  • Example: When presenting research to stakeholders, include raw data excerpts, explain the analysis process, and discuss how findings align with or diverge from initial assumptions.

The Nature of UX Research

Coming from academic research the practices that are commonly adopted in UX research are very much the playground research practices. Unlike academic research, which aims for statistical significance and universal truths, UX research focuses on understanding user behaviours, sentiments, and motivations to enhance product design and user experience.

Therefore, the notion and understanding are that conducting user research should be lean and iterative. However, this lean approach can lead to significant pitfalls if not executed properly.

Why Bad Research Persists

Time and Resource Constraints:
One of the primary reasons for bad research is the pressure of tight deadlines and limited resources. The time mostly compromises on the planning and analysis stages which are most crucial for the quality of data attained.

Lack of Specialized Training: Not all members of product teams have formal training in research methods, psychology, or behavioural science which helps achieve meaningful results. Misinterpretation of data and confirmation bias can skew findings, reinforcing initial assumptions rather than challenging them.

Complexity of User Behaviour: Understanding the intricacies of user motivations and behaviors requires sophisticated methodologies. Behaviour can be considered from many angles such as motivation, pain alleviation, the job to be done, habits, social influence, social norms etc.

What to do and look for when conducting UX research

Conclusion

Effective UX research is not about adhering to rigid academic standards but rather about gaining actionable insights into user behaviours and preferences. Those are supposed to help you make better decisions instead of walking in the dark.

But let’s be honest — UX research isn’t rocket science, even if it sometimes feels like it. It's more like detective work, minus the trench coat and magnifying glass. Think of yourself as Sherlock Holmes, but instead of solving crimes, you're solving user experience mysteries. “Why did the user abandon the cart?” might be as perplexing as “Why did the butler do it?”—but with the right approach, the answers are within reach.

So, next time you’re tempted to skip the research phase or rush through it, remember the story from the beginning. Avoid the pitfalls of bad research by taking a thoughtful and methodical approach. As the great philosopher Yogi Berra once said, “You can observe a lot just by watching.” So even if you don't feel comfortable with running the interviews or focus groups you can still get informative findings by just meticulous and conscious observation.

So, be curious, be meticulous, and maybe, just maybe, you’ll avoid launching a feature that flops faster than a lead balloon.

In the grand game of UX, it’s not just about asking questions; it’s about asking the right questions, to the right people, in the right way. So, put on your thinking cap (preferably one that’s stylish), and dive into the wonderful, wacky world of UX research. Because in the end, making your users happy isn’t just good for business—it’s downright elementary, my dear Watson.

• Quantitative studies seem to be more analytical and are most commonly employed via surveys. Those are more structured and can be looked over and checked by someone with a research or analytical background before deploying.