Think playtesting is a beast you’d rather face when your game is at its peak? Well, PlaytestCloud’s Elie Mouraud and expert games user researcher Steve Bromley strongly advise against waiting until the last second to playtest your game.
The two sat down recently to talk about the ways game builders and game studios can greatly simplify the playtesting process, and they all start with six of the most common mistakes that solo and team-based game studios make when they commit to playtesting:
To make sure you don’t fall into the same traps, below Mouraud and Bromley respond to each issue with experience from the field. They also describe actionable steps game studio teams can take to optimize playtesting and games user research:
- Playtest sooner
- Fully understand the state of your build
- Properly define current research objectives
- Maximize your playtest with small test groups
- Define your player base
- Don’t be intimidated by games user research
Each step carves a clear path on how to develop your game through games user research in response to today’s ever-changing market of potential players–even if you’re just at the ideas stage.
With their guidance, even a games user researcher new to the scene should come away feeling more confident about what to do…and what not to do…when playtesting mobile games.
1. Playtest Sooner
It can be super scary to run your first playtest, and to put your game out into the wild. That’s totally normal!
However, when teams have too many reservations about testing, they often wait until they are ready - or until they have a complete project - before committing to playtesting. Bromley says this is because game creators often have major reservations about sharing assets with players, usually out of fear of unwanted info leaks about their game.
With early-stage playtesting, should you keep it internal only? Or should you try and target an external audience?
Bromley: Really, it depends on where you are. Concerns about leaks can motivate people to try and keep things internal. They don’t want their playtest causing problems for the marketing team. The thing with internal-only playtesting is that you end up missing out on a whole bunch of important data.
As a result, teams almost always wait too long to playtest. They think: I’ll create the best possible product, test, and then make small tweaks right at the end.
Yet Bromley warns that this type of thinking can lead to some unfortunate challenges.
For example, say you’re deep into games development. At this point, you’ve likely made a ton of assumptions about what players want, what they understand, and even what they enjoy.
If at the time of playtesting, you receive (literally) game-changing feedback on needed improvements, glitches, or bugs, then they will be much harder to address since many of the major decisions about your game have already been made.
Worse yet, this leaves designers and creators feeling stuck–they’ve worked long hours on this product, only to find out it isn’t actually resonating with players, or worse, doesn’t work at all!
So, when should you playtest your games?
Bromley and Mouraud agree that the best time is right away; you don’t even need to have prototype or a minimum viable product available.
To them, playtesting can and should be done from ideation all the way through to launch; especially since the alternative is a high level of risk–and no one wants to find out that all the work they’ve put in has been for nothing.
“Teams need to move away from the idea that we need one big playtest at the end of development; there actually doesn’t need to be a huge gap between making a decision and testing a decision. You can always bring testing earlier into the development life cycle.”
In fact, Bromley says that user researchers should be playtesting across every stage of development.
Mouraud similarly notes that there are no hard lines to playtesting at the professional level. At PlaytestCloud for example, he would administer concept tests in early stages to draw initial analyses from - say - your marketing department, whereas longitudinal studies may be implemented early, and provide excellent player feedback over a longer period of time.
This kind of playtest flexibility allows you to get a better sense of who your users and players are, and how they react, no matter where you are on the road to making critical game design and research decisions.
2. Fully Understand the State of Your Build
Whether you have a team-based game studio or are working independently, it’s important to bring all parties up to speed on your game. The best way to do that is to spend more time understanding the state of your game’s build.
Your first step? Ensure there is someone (or multiple people) who you can loop into conversation on demand to give you this type of feedback.
What’s most important here is that this person (or group of people) understands at any given moment…
…the state of the team
…the state of the build (the current iteration of your game to this point)
and
…what decisions have recently been made about the game
These factors can have a huge impact on what you can test, and ultimately on your understanding of what you want to (or need to) know about your game.
Where games user research is concerned, you should always know what you’ve got available on deck - as well as what’s coming down the pipe - to ensure you’re creating realistic research tasks and objectives.
How do I improve cooperation between different teams/departments to support great playtesting?
Bromley is quick here to speak about the silos that can develop within game studios. It happens between artists, designers, developers, and producers, and can easily be mitigated by a good games user researcher–who often communicates across all of these teams.
That’s because games user researchers are well-used to forcing those tough conversations, asking questions, and have the most realistic insight on the current state of your build and team at any one point in time.
“A challenge when you’re a solo user researcher is that you don’t make software design decisions on your own, and games user research services like PlaytestCloud can for instance help you share clips and other things you’ve found between departments for more input.”
Importantly, adds Mouraud, they also know what people aren’t thinking about. They can tell you everything you need to know about your game, and are a great resource to inform you about what you need to know before submitting a build for playtesting.
They’ll ask questions like:
- Does your game really need all these start-up permissions to work?
- Are any debug commands available in the build, and should players use them if so?
- Is the in-game audio already available?
- Is the technical jargon of your game content and playtest commands legible to non-technical lay audiences?
Wondering who else should you include on this journey? You may be the main knowledge keeper, which is great! There won’t be many of those knowledge silos cropping up.
However, Bromley also encourages game builders to include other decision-makers and game designers (not just sales and marketing). Say you have a trusted designer on hand, or developer.
Getting them on board to contribute objectives, and their understanding of the state of the game, can have a major impact on your playtest’s success. Plus, they are a great (albeit somewhat biased) first ‘arm’ of playtesters as you check in on the state of your build!
3. Properly Define Research Objectives
Mouraud admits that undefined research goals or objectives can be the death of any good playtest. Either the research goal becomes too ambitious, is at the mercy of intense time pressures, or isn’t determined at all.
For example, some game builders hold onto the idea that they can bundle all their research objectives into one big playtest.
They get so ambitious and over-excited about playtesters playing their game that they end up overwhelming them, and potentially jeopardize usable playtesting data.
So the question becomes, how do games user researchers define and prioritize their research objectives?
Bromley suggests taking off the blinders: remember, the point of a playtest is to reduce risk–i.e. the risk that you release a game that isn’t fun, or the risk that your players won’t understand what they’re supposed to do.
That’s why you need to know the state of your build: to understand and define the biggest risks to your game. This knowledge will then help define your research objectives and truly understand the most productive playtest targets during games user research.
Once properly defined, you can rest easy knowing your data will reflect the tasks you set out, as opposed to having to sort through tons of data to infer questions after-the-fact. You’ll also be free to prioritize your objectives to best support games development.
How to prioritize games user research objectives
There is a way to streamline the whole process of prioritizing what objectives you’ll target with your first playtests, and it involves a basic and repeatable decision matrix.
Here’s the visual example of what Bromley says is the ‘unseen’ priority process that runs almost subconsciously for a lot of games user researchers:
Take in-game tutorials for instance. According to Mouraud, a ton of clients will come forward with a game that’s already been played and tested, but won’t have any of the tutorial elements that are critical to player retention–whatever the reason.
And while Bromley says there are a few ways to get around testing tutorials - such as asking playtesters to watch a presentation, or giving them a print out - he also believes it’s important to recreate the tutorial’s information if the tutorial isn’t ready yet, and to remember to come back and test the tutorial later.
In terms of the matrix of decision-making then, the tutorial aspect of your game would get all yeses. In fact, the tutorial should be tested early.
Not only is the tutorial a core part of the new player experience, but it is likely that the tutorial is already coming down the pipe, and, being untested, likely also doesn’t have adequate user data to determine whether something is wrong with it.
As a result, there’s a critically high risk that the end-result tutorial will be ineffective, ultimately producing all kinds of genuine usability issues with players–ones that could have otherwise been avoided with well-defined and prioritized research objectives during playtesting.
4. Maximize Your Playtest ROI with Small Test Groups
A playtest cardinal sin among professionals is to underestimate the power of small-scale, qualitative research. Yet, this is an effect still seen among large, small, and solo games studios everywhere.
The ‘bigger is always better’ mentality, it seems, has made its mark with games user research.
Bromley speaks to this common misconception, saying that even expert game builders forget the nuances offered by good qualitative research as they drive for large-scale quantitative research objectives.
There is a time and place for these surveys and quantitative studies of course. Sometimes, we need to know where players die most often, or whether difficulties are set properly. Large studies are great for that.
But when you want to know what’s going on in a player’s mind, or what they’re thinking, that’s where small-scale qualitative research objectives truly shine.
Here, Bromley speaks to the five-user approach to usability testing coined by Jakob Nielsen and Tom Landauer. They built a model which shows you can capture about 85% of your user demographic’s feedback from just five players.
This is where you can answer those nitty-gritty why questions, such as why players go the wrong way on this level, why they don’t understand fast travel, or why they click this button and not this one?
The added benefit of small user groups is that you can also engage with true-time gameplay. You can actually watch players play your game, ask them questions as they play, and pin-point the thought process of your players.
From there, you build the right interventions and fixes for each point in your game, and test again.
How do you approach player feedback when they come up with something you didn’t expect to see? Or…if they don’t react in the way you anticipated?
Bromley: I see this as a fantastic opportunity. That’s the true value of this type of research. We spend a lot of time with users as researchers, and I know from experience that we can’t channel user behavior. So, a lot of the value is in viewing those results without bias. Try your best to reduce or eliminate your personal stake in the game, or any preconceptions about what you think is true of your game. This will only result in you imposing your own bias, and adds risk to your interpretation of the results.
What else is great about small test groups? Bromley and Mouraud each take a turn emphasizing that with targeted qualitative research in small groups, you also get a better ROI for your time, and on your playtesting costs.
This is often where the value of good qualitative research is most misunderstood. Not only can players reveal things they may not have known about playtests, but you’ll hear the way they talk about the game (in real-time).
Mouraud uses the example of game builders that incorporate PlaytestCloud’s qualitative research products, who get this ‘WOW’ effect happening. Ultimately, they end up discovering a new player opportunity, or seeing a feature in a new light…all because of a simple player comment.
That’s why a good source of qualitative research is something Bromley says can be really life-changing for your game!
5.Properly define your player base
Admittedly, not a lot of new game builders or game studios know a ton about playtesting and the actual options that are out there for online and mobile games.
And while - as Mouraud says - playtest services like PlaytestCloud aim to educate new creators who want their mobile games in the app store, it can be tough for some to change the way they think about defining a player base for playtesting (as opposed to, say, advertising).
The challenge is that teams or solo game designers who haven’t done a lot of playtesting, or who don’t have a big budget, have to make use of what they have on hand. For obvious reasons, finding the right kind of players is hard when you are forced to recruit whoever is convenient.
Plus, if you don’t know how to define the player base for good games user research, you could be putting your end results in jeopardy.
Indeed, Bromley and Mouraud agree that the challenges of finding actual playtesters are made even more difficult when game builders don’t spend enough time defining who will buy their game.
“If you put the wrong people in front of your game, you’re going to get feedback, but it won’t reflect the same problems and opinions your true user-base has.”
There’s nothing against using a community to test your game, or just people on the street, but risk does increase the further you get from the people who are actually likely to play your game. That’s why Bromley says a representative player base is critical during all games user research.
How can game studios begin to create better playtest groups?
Mouraud and Bromley suggest a focus on user behavior vs. player demographics.
Many teams try too hard to base their player definition on general demographics (gender, age, etc.), when a focus on user behavior and gaming experience prior to playtesting can bring a ton of value to the player selection process.
In addition to general player characteristics, think about who your players are. Asking what other games they’ve bought, or what other games you would expect them to have played. Dig deeper, and ask questions about where they’ll buy the game, or how many hours they spend gaming per week.
Note from Steve: People often have preconceived notions about what they think a gamer looks like. This can create a ton of bias in player definitions, so a focus on user behavior over specific demographics can stop stereotypes from leaking into your games user research.
6. Don’t be intimidated by games user research
One of the biggest roadblocks to great games user research isn’t actually a lack of knowledge: it’s fear!
Too many game builders today get caught up thinking they need to be a professional games user researcher to get anything done with playtesting.
Most of the clients Mouraud sees are also under the illusion that they have to drop a lot of cash to properly playtest.
Thankfully, with the way games user research is evolving today, those fears are easy to overcome. Not only do most playtesting services like PlaytestCloud have flexible low-cost user research options for shared team insights or well-designed research studies…
…but there is a huge, FREE, open-source, online research community at the ready to help you get into a better mindset about games user research and good playtesting!
This article is just a small example of the resources that are out there to help you reduce fears about games user research. For example PlaytestCloud has an entire collection of resources just like this one to help get you started.
These sources and more will share incredible insights on how games get made, how to get involved in the development process, how to start games user research…even how you can get work in games user research!
That’s why PlaytestCloud tapped Bromley in the first place, thanks to his expertise on the subject. He has produced a ton of helpful industry content, and has written several publications including How to Be a Games User Researcher.
Truly, there is no shortage of opportunity to learn and even make mistakes with playtesting. It’s all part of the process. Whatever you choose to do - whether you hire a professional games user researcher, or take advantage of research that’s already out there - Bromley encourages everyone to feel confident in running their first playtest, and seeing how it goes.
Start Playtesting Now!
If there is one thing to take away from this article, it is that games user research is an iterative process deeply linked to the success of your game.
The idea is that after you run a playtest, you can go back and look at how you did, and see how you can do better next time. This will not only help your current game in the long-term, but potentially your career in games development.
When you do decide to begin playtesting, challenge yourself to…
…Test earlier than you are comfortable with
…Better understand the state of your builds
…Put more time into defining, prioritizing and understanding research objectives
…Know that five users can be enough to maximize ROI
…Be even more confident about how you playtest next time
We think it’s simply the best way to run a totally robust playtesting process, regardless of your technical background.
Good luck & happy playtesting!
