The first time I had to put together a research plan to thus inform a year’s worth of budgeting was no trivial task. Considering the needs of multiple teams and the differences of where they were in their development process meant that plans could not simply be duplicated at the rate of the number of games the research team was supporting. This article can provide a few tips on how to think about the needs of development teams and how research and its intended outcomes can meet those needs even when the exact projects are up for debate. Building a learning plan to inform a research roadmap is a great way to understand what might be needed in terms of methodology, time and cost and can help make sure planned research is impactful over the course of the next six to twelve months.
Also: did you know we have our own games user research team, ready to help with anything from designing a playtest, to analyzing your video results and surveys, to creating and presenting a full user research report? Reach out to us directly using the form below to know how we can help!
Research practices within organizations show maturity when they are able to plan what research is done well in advance. In the early stages of forming a research offering, the focus is likely on playtesting and evaluation. As more studies are done, insights are discovered. Sparks of curiosity bring fresh types of questions. At first, the steady flow of discoveries is welcome. After evangelizing the value of research, studio teams begin leveraging insights to make more informed decisions.
What was once a slow trickle of requests becomes a deluge of pursuits that all seem equally important. A potential request bottleneck can be prevented however, by working collaboratively with game development teams to form something called a learning plan. Grounded in the needs and motivations of the player base, a learning plan reflects on the current knowns and unknowns and links the eventual research outcomes to key performance indicators (KPIs).
In a sense, a learning plan is like a contract for making sure research has a positive return on investment. Once a well-thought-out learning plan is formed, researchers can take the details and input them directly into a fully-fledged research roadmap. Doing this work in conjunction with finance, customer support, and any other relevant stakeholder is a great way to mitigate risk. The potential research projects can be grouped into key initiatives along with any necessary sub-projects, their budgets, and their targeted timelines. Planning these items can help all parties involved sometimes months or even years in advance.
Learning plans help teams reflect on what is and is not known about the people they are building the game for. Through that reflection, the resulting plan provides recommendations on how to acquire knowledge when critical gaps are found. Learning plans can be applied at the sub-feature, game, or even the platform level. For the purpose of this article, a single game will be the lens for understanding how to form a learning plan. With a single game in mind, a learning plan can take shape in a number of ways. Learning plans typically need to be done collaboratively. They can serve as a reflective practice tool that can be filled out in real-time, or, they can be worked on asynchronously in a shared document when the time for each collaborator is convenient. An easy way to build one is to do it in a spreadsheet format (see figure 1) covering the following areas:
- A list of beliefs about the players and a level of confidence for each of these beliefs
- Recommendations for the types of research methods that will be needed to achieve high confidence for the beliefs
- Hypotheses on player needs and motivations
- Expectations for when insights from the research will need to be obtained for the ability to take action in a timely manner
- Post-launch KPIs that the research is expected to impact (e.g. retention, engagement, revenue, user acquisition)
Relationships to KPIs
KPIs, or key performance indicators, are critical for measuring the success of games. They should be a major part of a learning plan but they are only helpful when their relationship to the outcomes is well-defined. Examples of relevant KPIs are retention, engagement, revenue and the cost of user acquisition. The KPI of retention is a great example for why KPIs should be included in a learning plan.
Retention measures if players return to the game after a previous session
Retention can be measured in many ways but typically a successful game requires players to come back daily, weekly, or within some other defined period of time. Retention can thus be thought of as a signal that people are motivated to come back and play further. On its own, the measure of retention does not reveal to game developers exactly why or how players are motivated. When building the learning plan, work backward from retention and brainstorm why a player would be motivated to come back. One approach would be to form hypotheses by finishing the following sentence: “Players will continue playing the game because…”. These hypotheses can be listed in the beliefs section. The impact of investigating these beliefs can be listed under outcomes. The KPI of retention is now directly tied to the research methods and outcomes devised in the plan to investigate the hypothesized beliefs. If the necessity of the research ever comes into question, the research team can point to the KPI in focus and revisit the learning plan to make sure the research will be moving the needle on that KPI in a positive way. Figure 1 is an example of how to fill out the different rows and columns to form a learning plan:
Profiles People that play this game will...
| Beliefs | Certainty | Notes | Methods | Outcomes | KPI |
|---|---|---|---|---|---|
| ...stream their play sessions | LOW | No data yet to validate this | Survey | Measure percentage of streamers in the genre | UA |
| ...play differently if they are a spender | MEDIUM | Known from in-game analytics | Spender Interviews | Learn habits, preferences and motivations of spenders | Revenue |
Needs This game will help players to...
| Beliefs | Certainty | Notes | Methods | Outcomes | KPI |
|---|---|---|---|---|---|
| ...be creative | LOW | Hypothesis only | Player Interviews | Uncover unmet needs of players of the genre | Engagement |
| ...form meaningful relationships | MEDIUM | Qualitative data missing | Focus groups | Understand social behaviors of guilds | Engagement |
Motivations Players will continue playing the game because they can...
| Beliefs | Certainty | Notes | Methods | Outcomes | KPI |
|---|---|---|---|---|---|
| ...socialize with my in-game guild | LOW | Do not know percentage of guild members from adjacent games | Mixed methods approach | Validate at scale if social features are motivators | Retention |
| ...experiment with different strategies | MEDIUM | Know that strategy is key to genre | Test concepts of strategic systems | Recommendations for the amount of strategic depth | Retention |
Figure 1. Example learning plan for an upcoming game, adapted from the example in User Experience Team of One by Leah Buley
From Learning Plan to Research Roadmap
Once your learning plan is filled out, you can then start to put together a research roadmap. Where the learning plan covers what needs to be done and why, the research roadmap takes a more tactical approach in thinking about how and when. The roadmap will group projects by key initiative, consider the possible projects needed to deliver the outcomes and provide details on costs, tools and timeline.
Flesh Out the Roadmap by Pulling Initiatives From the Learning Plan
Looking back at the learning plan, name initiatives by their outcomes. Take for example the retention KPI that was determined to be related to the game design of new strategic features. A title for that initiative could be Motivate Players with Innovative Strategic Features. In the roadmap, the initiative can be summarized by the key outcome and what is being designed to evoke that outcome. Initiatives should always start with a verb. With the initiative include the KPI that is intended to move and when research might be able to measure impact. Here are some basic examples:
- Motivate players with Innovative Strategic Features (Retention +20% for players over 30 days, measured by average visits across key post-launch features)
- Discover untapped markets of streamers (+10% Increase players from competitor titles that stream online, measured by surveys in Q4)
- Understand needs and behaviors of spenders (Increase revenue by 20%)
- Create a more accessible social experience (Increase number of players in guilds by 20%)
Once a learning plan is formed, the next steps are to decide on the tactical approaches to getting all the work done. These tactics form the research roadmap. Group the work to be done in the following hierarchy: key initiative, KPI, and project. Doing so allows for a number of things. Patterns will emerge for determining which kinds of initiatives are going to require what kinds of projects and what size of budgets. Having this information will allow researchers to reflect year over year on what was done previously and improve forecasting.
| Initiative | KPI | Project | Cost | Tools | Month | Quarter | Milestone |
|---|---|---|---|---|---|---|---|
| Motivation through Innovative Strategic Features | +20% 30 day retention | Single session Playtesting with post test user interviews | 36 VT | Asynch. Testing | Aug. | Q3 | Pre-Alpha |
| Motivation through Innovative Strategic Features | +20% 30 day retention | Concept Testing | 28 VT | Asynch. Testing | Sept | Q3 | Pre-Alpha |
Figure 2. Examples of items in a research roadmap covering cost, tools, timeline and names of projects
Adding Items to the Roadmap
Besides listing out the projects, a research roadmap should also include what tools or outside services might be needed. Perhaps there will be a need to onboard a new tool for the research tech stack in order to complete one of the projects. Maybe there will be a need to hire a contractor or a third party. Knowing these needs early on will allow you to start planning by setting aside or moving around budgets. You can also include a low cost or a high cost version of each project. For example, some projects may be more costly if done in-person where others may benefit from scope shrinkage through remote approaches. Some projects may need agency work depending on the size and skillset of your team. For instance, when working with licensed IP, the stakes may be extremely high with confidentiality and certain cloud based tools might not be appropriate until after the partnership is announced.
Final Thoughts
Planning research in advance is not a trivial amount of work however; putting thought into the steps outlined here will help front load planning so that the research done is valuable and impactful. Building a learning plan will help teams consider the goals of the research and how those goals can move the important KPIs tied to the business. Because learning plans take into consideration both the research outcomes and the performance metrics they will impact, researchers and the larger development team can describe their goals through initiatives that they can all agree upon.
As projects are checked off, new ones may be added as you begin to learn and pivot from the results. Let the learning plan be the guide in deciding what pivots will keep research on target. Keep track of when confidence is acquired and spread the knowledge with wider teams. Long-term planning will eventually become an expected part of the research practice.
