Social distancing in workplaces is here to stay for the foreseeable future, and that means a whole host of challenges for those of us primarily using one-to-one moderated sessions for playtesting. In order to keep players and staff safe, labs need redesigning, and research processes and pipelines need rethinking.
At the same time, these challenges are an opportunity to embrace new methodological approaches to playtesting, and unmoderated remote testing is one way to address your research needs while adhering to distancing guidelines.
Benefits and challenges
Unmoderated testing has a few inherent benefits. One is an independent and efficient data collection process: removing the researcher from the player means fewer opportunities for bias to creep in. It also means baking in more consistency for the experience of each player taking part in the study.
A common perceived challenge of using unmoderated testing is that while the data collection process is more automated, the time and effort needed for data analysis is greater. However, a well-structured and designed study can mitigate this by breaking down gameplay - and player responses - into meaningful chunks of data which can be translated quickly and efficiently into prioritised UX issues and insights.
Planning and practice
What does remote unmoderated playtesting look like in practice? Players play your game on their own devices, wherever they choose to play them, with their gameplay and verbal responses recorded. These recordings provide lots of qualitative data of course, but here's where we can get clever and use alternative methodologies to ensure that a) the data is well-organised and easy to analyse, and b) we can answer all those research questions that we would usually address in moderated sessions.
Data collection in unmoderated studies
Some of these methodologies are about better data collection. Onscreen prompts allow us to probe specific responses from players as they play our games or look at concept materials, and carefully-timed surveys allow us to elicit quantitative or qualitative assessments and thoughts from players. Surveys in conjunction with multiple playtest sessions - over the course of a week for example - can be used to gather longitudinal data, showing how players' responses and feelings change over time. Where complex or in-depth qualitative responses are required from players, PlaytestCloud offer various ways to communicate with players, either as interviews before or after the study, or follow-up questions via their messaging system.
Research design in unmoderated studies
Where unmoderated studies can really shine is when they employ smart study designs, guiding players through the tasks and content that you want them to experience for your current research aims and asking the right questions at the right times.
- You can do this with very little preparation in terms of what the game build looks like: for example, by setting challenges and goals for players to address in their upcoming session. You can vary these goals across play sessions.
- If you need players to experience different sections, features or facets of the game - perhaps you want them to play levels 3 and 11, or experience Day 1 progression followed by Day 7 - you have various options. Players can manually skip through content, progression and features using debug menus and commands. Alternatively, create slightly different game builds for different play sessions, presenting different content on different days, or perhaps with an alternative configuration or class (e.g. Mage on Monday, Fighter on Tuesday).
- Survey questions and response prompts can be tailored to the build and content that the player received on that day, allowing you to get some rich responses to a wide range of content and features, and even some comparisons between different types of features: e.g which class did players enjoy playing more? Which was the easiest to understand and use? These data can be incredibly powerful when combined with analysis of gameplay videos.
Remote unmoderated testing can take the form of single session or multi-session playtests, or in sessions over a period of time. As well as eliciting player responses to playable game builds, they can also be carried out with concept materials. By carefully structuring your study – under the guidance of PlaytestCloud’s extensive research expertise – unmoderated testing will allow you to address the same research goals as moderated testing, with the benefit of reducing the burden on staff in data collection, and leaving you and your team more time for analysis and insights.