Research goals and methods for playtesting - Part 2

(This blog is adapted from our Mobile Playtesting Playbook)

In part 1 of this blog series we covered which methods are useful for examining usability and appreciation as research goals. Here are two more types of research questions that are less common but can be just as useful during the game design process.

Balance testing

Balance testing is a typical research goal you will encounter. This could include tweaking item or currency drop rates and damage values, and looking at the short, medium and long term impact of those changes on the player experience. These goals can be examined in playtests in the short term, and are especially useful when developers want to look at the impact on the player experience that these (often minor) changes can have in practice.

They are harder to assess in the medium to long term with playtests alone, and methodological problems can arise: for example, in separating out base levels of interest in and enjoyment of the game and differences that result from the changes you have made. However, a combination of player telemetry and surveys is one option if larger numbers of players can be recruited. This can also let you carry out a form of A/B testing on how balance changes compare against the original settings. Research studies in this format would need careful planning and design to consider:

  • Comparisons of different branches
  • Number of players involved
  • Types of outcomes examined
  • Statistics that are used to measure findings

For these research studies, we would recommend speaking to experts with experience in advanced study designs, telemetry and statistics.

Discovery as research goal
Sometimes we have research goals that are not focused on the appreciation, usability or balance of our game and/or its content. Perhaps we don’t have anything to test yet! We might simply want to know more about the players who we may want to reach with our games, or the mechanics of the game that might appeal to our players.

In these circumstances, we may want to carry out exploratory and/or generative research. This type of research helps us to:

  • explore issues for our users that we don’t know much about
  • confirm or deny our assumptions
  • design products that appeal to or support users

This is often referred to as part of the discovery process in the wider user research field: that is, the research work that is carried out before the design process has even started. Don’t assume however that it is only useful at the start of a product’s life cycle. Issues may arise mid (or even late) life cycle that require us to implement new discovery phases.

Discovery research methods include:

  • Standalone player surveys.
  • Interviews.
  • Focus groups.
  • Competitor analysis.
  • Community feedback.

This is the second blog post in our three-part series introducing the Mobile Playtesting Playbook. You can read the first section by using the link provided. The Mobile Playtesting Playbook is available on Amazon.com, or you can ask your PlaytestCloud representative for a complimentary copy (while supplies last).