How to Avoid Leading Questions in UI/UX Interviews

The most important part of user-centered design is the interview.  To create effective user interfaces, it's absolutely essential to get honest, detailed feedback from users.  The biggest pitfall, in my experience, is the leading question.  Here are tips on how to avoid leading questions and my strategy for effective UI/UX interviews.

What is a Leading Question?

A leading question is one that prompts or encourages a particular answer.  For example:

Wouldn’t your life be simpler if you had XXX product?

This question leads by implying that there is a "correct" answer.  The positive clause at the beginning of the question ("Wouldn't your life be better if...") implies that the person should agree in order to get the "correct" answer.

The same can be said for negatively-led questions:

Don’t you hate when XXX happens?

Isn’t it harder that you don’t have XXX in your life?

The structure of these questions changes the focus from the intended subject to whether or not the interviewee agrees with the interviewer.  Most people avoid conflict, so interviewees will tend to agree with the question (answering "yes").

Another type of leading question involves using a leading statement before the question:

Bright colors are important. What do you think is the weakest part of XXX?

With this structure, the interviewer has declared the focal point before asking the question.  Most people will be led by the preceding question and restrict their feedback to issues related to color.

What is a Non-Leading Question?

Non-leading questions are open-ended questions that eliminate our bias as interviewers.  They are designed to allow the freedom to answer in any way.  The most effective non-leading questions lead to full-blown discussions and can often reveal insights and ideas that the interviewer may never have considered.  Thus, our goal is to avoid leading questions.

All of the leading questions above can be rewritten in non-leading forms:

How would your life be if you had XXX product?

How do you feel when XXX happens?

How has your life been affected by not having XXX?

What do you think is the weakest part of XXX?

In the non-leading forms of these questions, the person isn't forced to agree or disagree with the interviewer, nor are they influenced by the interviewer's suggestions.

Emotion-Based Questions

People are more likely to start a conversation when there is an emotional attachment to the topic, because feelings are abstract and personal and, therefore, take more time to convey.  As such, the best interview questions are somewhat abstract and emotion-based.  If we use this insight and avoid leading questions, we can create effective conversation-starters:

When you are doing XXX, does anything frustrate you?

What do you enjoy the most about XXX?

Strictly speaking, the above questions aren't entirely non-leading (the first one suggests that there might be something frustrating about XXX, while the second nudges towards something positive), but they are open-ended within those boundaries.

A Strategy for Effective User Interviews

With emotion-based, non-leading questions as our starting point, we can create a strategy for effective user interviews:

1. Start with an emotion-based, non-leading question

First, we avoid leading questions and start off with an emotion-based question as described above:

When you are doing XXX, does anything frustrate you?

2. Ask for an example or anecdote

After the person's initial answer, immediately follow-up by asking them to provide an example or anecdote to further refine their answer.

Can you describe a specific example when you were frustrated by that?

This will provide you with a real world example - in their terms - that can  be used as a point of reference into your research.  It also pushes the discussion across the threshold from short answer to conversation, which is important in getting deeper, more honest feedback.

3. Ask the person to quantify their feedback

Next, ask the person to quantify their answer.

One a scale from 1 to 10, with 1 being very frustrated and 10 being very happy, how would you rate your experience with XXX?

Using numerical values in your interview process allows you to start assigning quantitative measures to your data gathering.

4. Dig Deeper or Start Again

At this point, you can either repeat the strategy with a new question, or use the numeric answer as a starting point for another question:

Given that your frustration in this case was 3/10, have you encountered something else with XXX that you found even more frustrating?

Ultimately, I liken this process to bringing an interviewee to a maze you have constructed.  You know at least one way out, but your goal is to avoid leading questions so that you're not showing them how to solve it.  The value comes when they show you a new way out of the maze.  If the maze (the questions) are structured well, then you should be able to follow behind, observing their progress and learning from the path they take to reach their destination.

Don't Let Dogfooding Mislead You

The first users of every startup’s product are its employees.  Starting with a few lines of code, they bring forth the scaffolding that will eventually become the MVP and, with luck, a full blown production product.  They’re also the ones who, for the first few years of a startup’s life, are amongst the most active users of the product.  But it's easy to get misled.  If you're not careful and look at the data, dogfooding can lead to the wrong conclusions.

Eating Your Own Dogfood

Many startups promote “eating their own dogfood” as part of their development process.  “Dogfooding,” as it’s commonly known, is the strategy of encouraging employees to become active users of their own product.  The benefits of this include more feedback, earlier in the development cycle, and feedback that's based on actual experience, rather than just contrived QA scenarios.

However, it’s easy to get misled by dogfooding if you don’t recognize the important fact that the population of internal users is not representative of your user population as a whole.  In general, internal users (even non-technical ones) quickly become expert users.  But that’s not the only thing.  The manner in which internal users interact with a product differs in some subtle but important ways from how equivalently-skilled external users do.  I'll demonstrate this with an example from DataHero.

Connections at DataHero

When we first released DataHero in 2013, users could import data from 6 supported cloud services or upload files from their computer.  The interface was a sidebar with square icons for all of the available services.  It acted as both a way to navigate data imported from connected services and a way to connect to new ones:

Over time, we added more connections and transitioned to a scrollable interface that included both the icon and name of the services users had connected to and a separate button for adding new services:

Eventually, we wanted to recapture some of the screen real estate used by the sidebar so that we could show larger previews of charts and dashboards.  The list of active connections took up the majority of the sidebar, but could we afford to reduce it?

What Did the Team Think?

Since we were big proponents of dogfooding, we started by looking inwards.  In early design sessions, the product team was nearly unanimous in the belief that all active connections should remain visible.  Being able to quickly navigate between datasets from different services with a single click was something they did frequently and everyone felt it would be frustrating to have to take multiple actions to go between services.  The engineers, designers and product managers each had a dozen or more services connected, so that was a lot of real estate.

But what about the rest of the team?

It turned out that sales, marketing, and customer support all felt the same way.  Almost everyone at DataHero had 10 or more cloud services connected to their account, and everyone regularly switched between them as part of their work flow.

Had we ended our design process with our dogfooding, we would have kept the sidebar and all of the real estate used by a dozen or more logos and names.  But we had data...

What Did the Data Say?

When we looked at the distribution of connected services across all users, we had more than 100 users with 10 or more connected services.  For DataHero, the number of connected services was a leading indicator of whether a user would become a paying customer, so at first glance this would argue to keep the sidebar as is.

But when we removed all of the employee user accounts from our data, the story changed dramatically.  It turned out that every single user with more than 10 connected services was an employee.  In fact, there was only one external user who had more than 5 connected services.  The average number of connected services was between 2 and 3.

Why the Disconnect?

When we went back and looked at how DataHero employees were using their accounts, it quickly became clear why everyone had so many connected services:

• Engineers, designers and product managers had almost all of the services connected for testing purposes
• The sales team had all of the services connected because they needed to be able to demo any service during a sales call
• The marketing team had all of the services connected so that they could write blogs and create other marketing content for whichever service they were promoting
• The support team needed all of the services connected so that they could debug customer issues related to any service

The Result

Instead of a sidebar with room for a dozen or more services (that, it turns out, was blank for all of our actual users!), we were able to move the connection icons to the top navigational bar and recapture a huge amount of screen real estate:

Data + Dogfooding = Better Design

Data and dogfooding go hand-in-hand.  Dogfooding is an incredibly beneficial strategy for getting early feedback on your product, but it's essential that you use data to fact-check your conclusions and make sure your internal users are truly representative of your user base.  Otherwise, you might end up designing for your employees instead of your users.

To help you put this into practice, here are 5 Steps to Use Data to Dogfood Better.

How to Set the Tone for UI/UX Interviews

The interview is one of the most critical parts of the UI/UX process.  Your goal is to solicit honest feedback, so you need to avoid leading questions and other such pitfalls.  You also need to set your interviewee at ease, as a stressed or apprehensive subject is more likely to try to appease the interviewer with their answers. Here are 5 simple steps to set the right tone for UI/UX interview success:

1. It's an Interview not an Interrogation

Try not to have more than two people in a room during an interview. Three or more people can place the interviewee in a defensive position, which can dramatically reduce the honesty of their responses. A single interviewer is best when trying to drive a conversation, but often we want to make sure there are multiple people to get different perspectives.

If you have two interviewers, try to define specific roles so that one becomes the dominant presence and the second the observer for a given question.  These roles can be swapped for subsequent questions, but should never overlap.

2. Make the Environment Comfortable

Think about the last time you had a comfortable chat with a close friend and the environment in which it happened.  This is what you want to recreate with your interviewee.

Try not to sit on opposite ends of a table, as it creates a physical barrier between the two of you.  Instead, sit on the same side of the table or at a couch or similar.  Depending on the nature of the interview, consider taking a walk around the block while recording the conversation (with their permission, of course).

3. Empower through Gratitude

Start the conversation by sharing how grateful you and your company are for their time and how much you are benefiting from their answers. You want to empower the person you are interviewing and give them a sense of control immediately. The best insights come from putting the interviewee in charge and letting them take the lead.

4. All Answers are the "Right Answer"

Emphasize from the start that there are no wrong answers. You want to eliminate any sense that the interviewee will be judged on their answer and make them feel safe.  Reinforce this periodically to create an environment where they feel that anything they say will be of value.

5. Your First 3 Questions Set the Tone

Your first three questions set the tone of the conversation, so try and make them light and easy. Ask for the interviewee’s thoughts on a topic related to your product, about their work, or even about the local sports team (just make sure you don't ask about politics or religion!).  The more honest your curiosity is, the better the person will respond.

Don’t ever jump right into the interview or start with difficult or complicated questions.  Ease into the interview and pace your questions so that you intersperse hard questions with lighter, easier ones.

Want to learn more strategies for effective UI/UX interviews?  Here's how to avoid leading questions in your interviews.