User research guide

1. Purpose of user research

What is user research?

User research is a study of who the users of a service are, how they behave and what their motivations for using the service are. Simply put, user research is a set of techniques you can use to consistently deliver exceptional customer service.

You’ll use different user research techniques depending on your current phase of service design. The purpose of a user research technique like "usability testing" ranges from trying to generate ideas (early in the product design lifecycle), to validating a prototype (towards the end of a product design lifecycle).

Why user research?

Learning about the users allows you to create better government services. You will discover the user’s motivations, struggles and goals for a service. The role of a user researcher is not to consult with, audit, and/or teach users. The role of a user researcher is to actively observe and listen without bias or assumptions.

User research will:

• discover the need for the products and services sooner rather than later
• establish a good foundation to make sure you are designing the best products, services and systems
• lower cost of projects in the long term
• improve processes to deliver services faster and more responsively
• reduce the number of user inquiries and complaints
• provide value to users, leading to better user experience

2. Preparation prior to user research

Coordinating with stakeholders and team

The benefits of having your stakeholders and team watch at least one user research session include:

• Make suggestions that are people-centred: Having your stakeholders participate in the research sessions will encourage them to prioritize user experience, rather than pursue design concepts that may not meet the user’s needs.
• Credibility: Your stakeholders will see that your design is based on research and not personal preference.
• Buy-in: Reviewing what happened in the research sessions as a group and determining priorities together (for example, which feature must be included, needs to be modified, could be implemented if there was more time, and should be removed) means you don’t have to spend a lot of time explaining your choices to stakeholders and convincing them that it’s the right approach.
• Memorability: It is easier to remember things when you experience them first hand, rather than read about them after.
• Empathy: You are able to relate to the user and understand why the service is not working for the user.

Defining a user need to focus on

Identifying a single user need will help you focus on a specific problem. It will also help you define who your users are. If you’re struggling with defining multiple user groups, read defining your users when there are multiple users involved.

Meet with your stakeholders and team to brainstorm how to go about your research. There is always a reason why users need your product or service. People don’t wake up and think I wonder what the government of Ontario is doing today? I am going to visit their website to find out. There is something that happened in their lives that drives them to use your service.

Tips for defining a user need to focus on:

• Ask around: Find the people who are already trying to make things better and understand what approach they took.
• Find what’s not working: Find failures in the service. What were the users trying to do or what were they thinking when they tried to complete the service? Document as you go.
• Use existing data: Analytics are a great way to uncover how users behave on your website and find areas where users struggle. Use usability tests to find out why users struggle.
• Consider existing government services: Think about services that are only provided by government. Are you competing with any other service providers? Are there other ways that users can meet the same goal?
• Consider new policies: Ask your policy team what is on the political agenda for the future. This is a great opportunity to plan ahead and gather feedback from users that would be affected by new policies before timelines get too tight to use the research.
• Hypothesize and prototype: Start by deciding on the questions you want to answer. Make predictions about what you think users will do, and then create tests around those assumptions and observe what happens.

Defining your users when there are multiple users involved

Narrowing down user needs when there are multiple users is hard. However, if you try to please everyone, you end up pleasing no one. You must be able to define user needs to design a usable service.

Tips to define your users when multiple users are involved include:

• Picking a primary user group: This does not mean excluding other user groups, or making services that some users won’t like. For instance, the primary user for a wheelchair ramp is a wheelchair user, but those ramps will also be used and enjoyed by people with baby strollers or people who are skateboarding.
• Focus on the scenario or situation: A specific situation will attract specific users. For example, although taking the bus is a service designed for “everyone,” there will be specific groups of users who will take the bus more often than others. From the scenario, you can start designing your study based on different groups in that specific situation.
• Focus on extreme users: World leading design firms like IDEO suggest using "extreme users" to get critical insights. Extreme users tend to have the same needs as the average users, but their needs are much more amplified. For instance, if you managed a gym, your extreme users might be ultra-marathoners, and people brand new to exercise. This is the opposite of the common approach to defining users very broadly. For example, everyone in Ontario is my user group.
• Focus on accessibility: Users with special needs, similar to extreme users, are often good at articulating the problems more than generic users might experience, but have a harder time putting their finger on it. Speaking to a small number of extreme users can give you insights into the needs of most of your users. You often find new insights, and clever workarounds that help improve services for everyone.

Identifying the most suitable user research technique with your project phase

Depending on where you are in the product development life cycle will depend on what kind of research you do. The journey from discovery to live is not about taking a long time to build a ‘grand’ product that considers as many things as possible. The goal is to build a minimal viable product (MVP) where:

• you involve users throughout the process to help you understand if you are building what will ultimately solve their problems
• user research will help your team prioritize, gather feedback, set criteria to test against what is being built
• you start small by focusing on the essentials and plan for other not-so-crucial elements for future releases
• you learn what works and doesn’t work as quickly and early as possible, rather than learning about them near launch or post-launch when it takes more resources to modify and/or fix what has already been built

Discovery phase – discovering the problem

At this phase, you focus on generative research. Your focus is discovering what the problems are.

If you have not done user research so far but already have a solution in mind, this is a great opportunity to challenge your assumptions. It is in your best interest to challenge them early to minimize unexpected surprises. If you discover areas of success and failure early on, it is easier to prioritize and plan for next steps sooner rather than later when more time and resources are dedicated to the project. You may:

• conduct interviews to discover user problems and needs
• conduct an ethnography study (shadow someone) to observe what the user is going through
• usability test an existing service to better identify user challenges and points of failure when completing tasks (this could include using an existing design from the ministry or from other public service sectors as a point of reference)

Alpha phase – planning and conceptualizing

At this phase, you focus on exploratory research. Based on problems identified in the discovery phase, you will explore multiple concepts and different options to go about tackling the problems. The goal at this phase is not to create a beautiful product off the bat, but to explore different paths to solve the problem and worry about polishing the details in later phases. You may:

• collaborate with co-creators to brainstorm and test different approaches
• guerrilla test on a few different concepts and observe the user’s reactions
• conduct preference tests to figure out the pros and cons of multiple concepts
• focus on testing with low fidelity prototypes to test your hypothesis. The reason why you use low-fidelity prototypes, rather than polished high fidelity prototypes, is because you will be exploring multiple concepts that may or may not move forward. Rather than focus on details of the prototype, you are focusing on broad concepts.

Beta phase – product testing and building

At this phase, you would have already narrowed down a concept and are focused on working out the details. You may:

• conduct card sorting exercise to structure menu items on your web page
• guerrilla test on specific elements of the product to ensure participants understand what you are asking them to do
• usability test on high fidelity prototypes to identify potential usability issues. The purpose is to take a product idea and iterate to make gradual improvements based on feedback.

Live phase – finalizing a minimal viable product (MVP) and launching

At this phase, your product is released to the public. You will be able to conduct more quantitative research as you expose the service to a larger sample of users. You may:

• create A/B tests to measure the success and failure rates of different users
• track behavioural patterns with data analytics
• work with a behavioural insights unit to conduct tests with large sample sizes to design, evaluate and implement programs and services that are more effective, efficient and human-centric.
  • For more information on how Ontario is applying this innovative policy tool, visit Behavioural science insights pilot projects or email BIU.CAB@ontario.ca.

Product versioning and/or labelling

Once your product is delivered, you may label your product as alpha, beta, or live depending on how many users have access to the information and when your product was released.

• Alpha: means that your product is still in its early stage. Although it is available for public viewing, it communicates to users that this is one of the preliminary versions so mistakes and bugs are expected.
• Beta: means that the product has been tweaked multiple times and feedback has been incorporated in the later versions.
• Live (or no label): means that the product is as close to official as it gets. Minimal problems should be expected.

Importance of ethics and risks

Every user research presents some type of risk to the participant. Types of risks may include:

• physical
• social
• psychological/emotional
• repercussions for (or for not) participating

User researchers are responsible for participant’s confidentiality and reducing the risks of conducting research. A good question to ask yourself is is the level of risk greater than what the user would encounter in their everyday lives? If the answer is a maybe or yes, please consult with the Ontario Digital Service at digital.standard@ontario.ca.

If your research is relatively low risk, you may use this guide as a starting point. Basic information on consent, recruitment, retention of information and compensation are found in this guide.

When formal approvals are needed

Approvals may be required for complex user research activities. To give context to what complex user research activities may look like, here are a few examples:

• research with vulnerable participants where they may not be able to provide consent, or may be able to provide consent but not fully acknowledge the repercussions
• participants with higher risks involving physical, social, psychological/emotional, and repercussions for or for not participating
• data with special arrangements on who can access, how it is shared, security protocols, and storage
• collecting personal information that will be identifiable to the participant
• research done outside of Canada

If you are seeking to do complex user research activities, please consult with Ontario Digital Service at digital.standard@ontario.ca for additional advice. You may also want to contact your FOI and privacy coordinator for a second opinion.

Users that need special attention (vulnerable participants)

All research, with any type of user, involves a power dynamic. If you are looking to gather insights from vulnerable participants, you may need assistance beyond this user research guide and will require more extensive review or expert guidance. Projects using participants that are vulnerable will receive greater scrutiny. Examples of vulnerable participants may include:

• children (under 18 requires parental consent)
• people who are from a historically racialized, marginalized or oppressed group
• indigenous peoples. (for example: First Nations, Métis, and Inuit peoples)
• diminished decision making capacity
• captive or dependent on others (for example: seniors in care homes)
• research projects taking place outside of Canada
• research with over-researched populations, such as people who are homeless, people who work in the sex industry, or people who use illegal drugs

If you are unsure on how to approach your user research study, always ask for a second opinion. You may contact Ontario Digital Service at digital.standard@ontario.ca or your FOI and privacy coordinator for your study.

3. Recruitment

This guide assumes that you will be recruiting participants over 18 years old who are capable of making their own decisions and are residents of Ontario. If you are looking to recruit participants that do not fit with the description previously mentioned, please review when formal approvals are needed to determine if extra assistance is required.

Identifying your users

For government services, users can come from many places. Therefore, it may be hard to narrow down a target group of users to test.

A great way to start would be to refer back to your defined topic and ask who will most likely use these services? Few key considerations to consider when identifying users include:

• Tool experience: How familiar is the user with how to use the website, application, and/or product/service?
• Program experience: How familiar is the user with programs and/or services that surround the tool?
• Domain experience: How knowledgeable is the user about this content?
• Diversity: Overall, is the mix of people you are speaking to a reflection of the diversity one might see in the overall population?

Finding users

Depending on who your target audience is, there will be multiple ways to recruit users. Consider what is most suitable and relevant for your study.

For guidance on who your users are, you may refer to preparation prior to user research.

You may recruit users from:

• existing clients from government organizations (such as the Ontario Public Service)
• stakeholder connections
• meetup groups
• social networks
• people in public spaces such as university campuses, libraries, parks, etc.
• people visiting public service offices on-site
• you may place recruitment posters (see template as guide) throughout relevant public spaces
• referrals from other users
• community service groups and non-profit organizations

Tips on recruiting: Do not mention specifics of the research such as how many tasks the users need to complete ahead of time. The users will feel pressured to complete the tasks in a timebox, rather than focusing on the tasks itself.

Recruiting with user recruitment firms

Hiring user recruitment firms is generally the last resort because of the high cost associated with each participant. If you are looking to do user research with general easy-to-reach participants (such as a university students, teachers, parents), recruitment firms may not be necessary.

Recruitment firms are good for:

• specific demographics
• special conditions
• participants who are hard to reach
• fast turnaround times for projects with tight deadlines

To effectively use recruitment firms, you should consider writing a narrowly scoped screener so the firm can better tailor to your needs. For example, government health teams have recruited in the past people who are of South Asian ancestry and have type two diabetes in their families or people who have mobility challenges associated with the loss of a limb.

What is a screener?

A screener is a list of criteria used to check whether or not your potential participant would match with the characteristics of your target audience. Recruitment firms hold large databases of people prepared to join research studies. The recruitment firm will ask "screener questions," prior to or leading up to a research session, to determine if their potential participant will be the right fit for your particular study.

4. Consent

This guide assumes that you are getting consent from participants over 18 years old who are capable of making their own decisions and are residents of Ontario. If you are looking to get consent from participants that do not fit with the description previously mentioned, please review when formal approvals are needed to determine if extra assistance is required.

Purpose of consent forms

The purpose of a consent form is for both participants and researchers of the session to understand and agree on:

• who is collecting the data
• purpose of the study, such as why you are collecting the data
• any recordings that will happen during the session and how information will be handled
• what type of information will be recorded
• how the information is going to be used
• duration of the study
• compensation for the study
• who to contact if there are any questions

Recording specifications

With informed consent from the participant, you may choose to record the test session. You must inform the participant:

• what the recording is for
• how you will protect personal information
• that their personal information, such as stories and experiences, will not be associated with them

Sharing data within the project

• Involving stakeholders live is better than sharing recordings afterwards. Showing recordings of research to stakeholders is not nearly as valuable as having them involved in the observer’s room. Whenever possible, involve the stakeholders live, rather than send recordings. Aside from being less risky from a data perspective, research is often a you-had-to-be-there story. The full experience cannot be fully captured with just a recording.
• Protect privacy of participant when storing and sharing data within the team. When sending recordings to stakeholders involved in the project, make sure they are password-protected and nothing identifiable is expressed in the recording.

Consent form

A consent form and guidelines are available on the OPS intranet (InsideOPS).

What is personal information?

The Freedom of Information and Protection of Privacy Act defines personal information as recorded information about an identifiable individual. This can include, but is not limited to:

• biographical details (name, sex, age, race)
• biological details (face, fingerprints, blood type, etc.)
• nationality
• religion
• marital status
• education
• medical or criminal history
• financial information
• identification number (SIN)
• contact details (address, phone numbers, etc.)
• personal opinions and views

5. Compensation

Purpose of compensation for user research

Each user has their own perspective and experiences to share. The best way to acknowledge their input is to compensate them; especially when the user is taking time away from work to give us their thoughts.

Here are reasons why compensation is essential:

• not everyone has the luxury of donating their time to your user research project. There may be child care considerations, long commute times, or other reasons where unpaid research is not a high priority.
• people who are able to participate voluntarily generally have time or sufficient income to attend and this leads to poor sampling.
• if you conduct free user research, most people are unlikely to take time off from work to participate. A good question to ask yourself is how likely are you to participate in a one-hour, unpaid interview, where you would have to take time off to travel to their office and provide feedback? Usually that question will illustrate why paying participants will make their effort worthwhile.

However, if it is simply impossible for your team to arrange payment, please review the following guideline on who does or does not get compensated. In this instance, it may be particularly important to run sessions outside work hours or travel to participant’s communities to make research as easy as possible.

Who gets to be compensated

As members of Ontario Public Service, we show appreciation to the user by compensating them for their time, experience, and insights provided. You may use a compensation form for proof that the participant received compensation for your study. In general, we compensate all Ontarians with a few exceptions.

We as Ontario public servants do not compensate:

• participants who are employees or contractors of the Ontario Public Service.
• participants who are part of a working group of the particular study.
• participants who are getting paid as part of their work to partake in the study.

Ask yourself Is the participant partaking in unpaid time away from their day-to-day activities to participate in the user research session? If yes, then compensation is reasonable.

Amount to compensate

The shorter and less complex a product is to test, the less compensation is provided.

Compensation should be appropriate in type and amount. The researcher is responsible for assessing the use of a particular model and level of compensation.

All compensation should be provided fairly to all participants in the same study, with equal compensation for equal participation regardless of income, age, gender, race. Additional compensation can be arranged for transportation fees if needed.

Here is an example of how much participants have been compensated in the past:

For further details on getting approvals for compensation, you may contact digital.standard@ontario.ca to request for sample business case.

6. Retention and proper management of public records

Recordings from user research sessions are considered records and should be handled like other ministry records. Please consult with your records manager for best practice.

7. User research techniques

Great design comes from iterating many times until you achieve a product or service that is exemplary.

Research is the first step to learn as much as you can about your users. You will then make use of the findings to efficiently iterate and enhance the user experience.

There are two main types of research you may conduct— qualitative and quantitative research. You may use a technique like card sorting for both qualitative and quantitative research depending on the insights you want to capture.

The following techniques listed under each type of research are recommendations based on what they are most commonly used for.

Qualitative research – answers "why?"

This type of research explores why people behave, act and interact the way they do. Qualitative research focuses on everyday life and experiences (which cannot be measured by numbers). It investigates meaning, interpretations, symbols, processes and relations to social life. The following are various techniques to conduct qualitative research:

• User interviews
• Usability testing
• Guerrilla testing
• Ethnography study
• Card sorting
• Preference testing
• Mind mapping

Roles and responsibilities

Depending on which qualitative research technique you choose to proceed with, there are key roles and responsibilities associated with each technique:

• study administrator (also known as moderator, interviewer, etc.). As study administrator, you will be responsible for the safety and comfort of the user while uncovering why the user is interacting with the product/service the way they do. You may refer to this sample script as a guide to get started.
• observers (also referred to as notetakers). Observers are responsible for taking notes. One way to take notes is to write one observation per sticky note with a marker (for clear visibility purposes). You may use a sample instruction for observers as a guide.
• live transcribers and sweeper. Having transcripts completed means you will be able to use them as reference and analysis. You may use Microsoft 360 or Confluence as a way to transcribe. There are usually three people assigned on the task.
  • person A: writes what participant says.
  • person B: writes what moderator says.
  • sweeper: tracks comments that were missed by person A or B.

Quantitative research – answers "how many and how much?"

Quantitative research uses numerical data from a population to determine behaviour patterns within groups of people. The goal of quantitative research is to get measurable data that can be analyzed with statistical procedures.

With the following techniques, data is collected with programs, software, or tools that have almost no direct contact with users. This means that while stakeholder and team participation is not applicable while data is being collected, it is beneficial for you to involve your stakeholders and team before you start the research to identify metrics and success measures, and after the research has occurred to present findings. Here are various techniques on quantitative research:

• Data and web analytics
• A/B testing
• Behavioral insights
  • To learn more about behavioural insights research in Ontario, visit Behavioural science insights pilot projects or email BIU.CAB@ontario.ca.

8. Prototyping

Prototyping is an early sample, product, or model used to simulate what the final outcome could be. Prototypes serve as a proof of concept, a way to test functionalities early, and/or a tool for visualization. It is essential to generate prototypes prior to building the final version to save costs, time, and gather feedback from users as early as possible. You may:

• create low fidelity prototypes to quickly gather feedback on general concepts or rough ideas
• create high fidelity prototypes to present users a polished version of a concept
• conduct co-creation sessions with users to generate low or high fidelity prototypes for testing

9. Turning user research into actionable insights

After conducting your research using one or more of the techniques, you are now ready to analyze and make use of the research for the following purposes:

Understanding who your users are

• Personas
• Empathy map

Building better user journeys

• Journey mapping
• Service blueprints

Communicating problems

• Problem statement

Prioritizing

• Impact vs. effort matrix
• Affinity mapping
• Dot voting

Starter kit

The method cards contain some of the concepts described above at a more general level, but does not go into precise detail on each method. Use them as a starting point and read more detail on each method as you continue to analyze more research.