This article explores the potential returns a game studio can anticipate from investing in automated testing. Although the specific advantages can differ greatly depending on various elements like the chosen automation strategy, the execution quality, the degree of cooperation between Engineering and QA, the level of investment, the nature of the project, and the development phase, here are several typical outcomes.

Faster Development and Easy Pivots

Faster test execution means faster feedback and tighter iteration loops for the dev team, leading to less wasted time, and a better game. Higher “internal quality” allows fast feature development and easy pivots for the entire life of the product.

Reduced QA Costs and Increased QA Effectiveness

Automated coverage reduces the need for manual testers to execute scripted test cases. This may lead to QA cost savings, and/or free up skilled QA staff for the high value exploratory testing that will truly level up the player experience. Rare, for example, reduced the size of their in-house QA team by 66% while shipping a considerably more complex open world title to their players, weekly, at high quality.

Scales Better Over Time

Automation scales well with time; its initial cost is offset by the ability to run it forever, essentially for free, while the cost of manual testing only increases over time. A successful live service title may live for decades. If it does not have automated tests, QA costs will continue to rise, eating into profits and dev budgets.

Mitigates Risk

Games with robust test automation typically carry a very small bug load throughout development, since bugs are found and fixed quickly. Production need not worry about how long it will take to burn down bugs before a milestone or release, giving the business more options. The practice of carrying a large bug load brings considerable risk and unpredictability to the business.

At a certain level of maturity, mainline builds are always shippable, and releasing to players is a non-event, reducing the cost and risk of releases, avoiding potential reputational damage, negative reviews, and lost sales. It is also more sustainable for the dev team and helps to attract top talent.

Better Player Experience

A stable, reliable game on which the team has been able to iterate rapidly, getting releases in front of its community of players/playtesters early in development, is likely to be a better game. It will likely see better reviews, sales, engagement, and retention rates.

What Investment is Appropriate?

The upfront cost of automated testing can be significant, and depending on the situation it is likely that not every possible automation technique will provide positive ROI, and some may provide negative ROI especially if poorly implemented. The main things that can go wrong here are ignoring automation, investing in the wrong things, and executing poorly.

1. Not investing in automation at all. For most modern games this is at best a false economy and at worst a fast track to spiraling costs, delayed features, and permanent crunch. If a game is simple and is not intended to be long lived, then a full-scale automation effort may not be worthwhile. However, there are different levels of investment that achieve different results. Andrew Fray’s strategy on Rollerdrome is a great example of focusing limited resources on very high ROI automation that enabled incredible business outcomes (only one bug fix after launch, tiny carried bug load, tight iteration loops, and the confidence to make a huge physics change right before release). Every project can benefit from some level of automation appropriate to its scale (even at a 4 person studio).
2. Choosing automation techniques that don’t align with the project’s goals and constraints. While every game can benefit from some level of automation, the skill is knowing what level of automation is appropriate and will provide positive ROI. A single player indie title will have a very different needs than Minecraft. Investing too much in inappropriate techniques or too little in appropriate ones will have obvious financial consequences.
3. Executing automation poorly. To be successful, automation efforts must be treated as products, with a sharp focus on the actual value they provide. It is far too common to find unused test frameworks and tests that never run, have been disabled, or are simply ignored. Worse, it is rarely recognized that these projects have failed, so they continue to consume resources indefinitely. Successfully implementing test automation is a hard technical and cultural problem and it is all too easy to lose sight of the ROI.

The good news is that all of these pitfalls can be avoided by engaging the help of experts in the field, either internally or externally.

Conclusion

Test automation for games can offer huge benefits and ROI, and there are examples of studios having unimaginably favorable outcomes. However, it is still not a mature field and it is very easy, and common, to make mistakes that lead to a negative return on investment.

To achieve the full benefits of test automation it is important to have the expertise to know which investments are appropriate and how to implement them correctly. If you would like to partner with an expert advisor, please book a call with us today!