Case Study: Playdots

The entire team at PlaytestCloud is always interested in the current projects of our clients, and we look forward to receiving feedback - especially when playtesting makes a difference in the outcome of a game or feature launch. That is why, when we heard how playtesting with PlaytestCloud was an integral part of one of the most successful feature launches for Dots inc. We just had to share how it was done!

The Client

Playdots - a free to play mobile game - was first released in 2013, Following in the footsteps of the minimalist trend of the early twenty-teens; Their first game, Dots was created with the notion that beauty and fun are not mutually exclusive, and audiences have reacted accordingly. Within the first two weeks after its release, this simple game was downloaded more than 2 million times and was the top free app in eight countries. This success was followed on with the single-player sequel - Two Dots - released just a year after the original. Two Dots achieved the #1 rank in 71 Countries within the first 24 hours of its launch. Three years later Playdots launched Dots & Co, which quickly became a "Top 5" game in over 100 countries. With the launch of Dots & Co, the studio achieved Three back-to-back global hits.

These successes have made Playdots a trailblazer in the games market. But the team at Playdots knows you can't relax even when business is going well. Playdots must always innovate and improve in order to keep themselves relevant in the App Store — and they have a lot to lose with a poorly executed feature launch.

The Challenge

We received an email from Drew McWhorter, a product manager on the game Two Dots. His development team had been working to launch their "Win Streaks" feature designed to improve the in-app purchases (IAP) conversion rate. They were curious to see if the feature was ready to release to the public as the team at Playdots never launches a feature without playtesting it first — for any of their games. Their company testing philosophy is, if you're spending valuable resources to develop a feature it is critical to make sure the feature is ready by testing with external users prior to going live.

For new installs, the first three days of gameplay are strategically designed to help users learn how to play and enjoy the game. Because of this, these levels rarely see major changes — if ever. Playdots planned on launching Win Streaks after the third day of play for most users given the significant change to the core gameplay and advanced nature of the feature. With this playtest, they wanted to test if this new feature would help or hurt them in the overall onboarding experience. Onboarding is critical for every game; but with all of the Dots games, onboarding holds a little more weight. This is due to the unique game mechanics of the Dots franchise — so they want to make sure they get it right.

The Playtest

When Drew was ready to place the order, he contacted PlaytestCloud with a unique testing strategy. In his testing plan, he sought out to separate the players into two distinct groups. One group would consist only of new players and those players would start their gameplay on level 1. Drew would then watch these players interact with the traditional first-time user experience and learn their player type — then after players passed level 38, the new Win Streaks feature would be introduced. The second group would be players who had experience playing Two Dots from their past personal gaming background. Those players would start on level 35 immediately (three levels before the feature was introduced) and Drew would watch them play the game for three days to see how they interact with the Win Streaks feature over time.

Ordering a longitudinal playtest was the logical choice for this type of feature since longitudinal playtests are the only way to watch how players interact over many days. Drew understood that to truly understand a player's long term interactions with a game-changing feature like Win Streaks he needed to observe their behavior over multiple game sessions.

In total, Drew received 16 hours of video from 10 players. His team incorporated user testing into the tail end of their design process and needed to work quickly to analyze the results before the last development sprint began.

The Results

Originally, Playdots assumed that the Win Streaks feature would be a fairly advanced mechanic and that it had the potential to impact new users negatively. But by watching the videos it became very clear that Win Streaks were a much more powerful feature than originally thought. Time after time, Drew would watch a player become bored, frustrated, and ready to churn; but after unlocking the feature the player would suddenly change course and enjoy the game again. The power of this update was confirmed over and over again after Drew watched the players play over consecutive days.

Overall these findings were unexpected but extremely beneficial for Two Dots. It was very clear that feedback was decisively positive for the addition of Win Streaks: as evidenced by the excitement from players during the longitudinal playtest. Based on what they learned from the study they decided to push this feature earlier in the first time user experience for new installs — levels they have not adjusted or changed in a long time. By having the confidence due to having the long-term qualitative study results, they had the data to take the calculated risk.

Months later, after the feature had been fully launched, they were excited to see that the feature was even more impactful than they had originally thought. In fact, based on what they learned through the longitudinal playtest they were able to adjust and refine this feature so that it eventually became the most impactful feature launch they have had in two years — leading to the highest IAP conversion rates the game has ever had. Drew was happy to inform us that running a longitudinal playtest was one of the key reasons for the sustained success of the Win Streaks feature. He's looking forward to leveraging longitudinal studies for many future feature launches.