Assuring Quality in Today’s Complex World

Delivering consistently excellent products meeting a high quality bar takes coordinated strong efforts and careful alignment among development teams from the artists to the engineers. The specific ways in which quality assurance can effectively contribute aren’t always as well understood.

The reality is Quality Assurance is evolving both inside and outside of game development. Riot Games focuses on holistic quality ownership, where everyone in development owns their work, along with a responsibility to the value we provide players.

Managers often hire for attention to detail, a passion for products, and a high capacity for communication among different viewpoints as well as the ability to build and use dynamic checklists. Unfortunately all of those sorts of things narrowly address the types of individuals that make QA successful. It’s not a blueprint for how to develop the discipline further.

As part of our approach to evolving the discipline, we’re concentrating on a few core tenets:

Quality Assurance begins at the start of a project

From early iterations through development, release, and beyond, QA assists engineers, artists and designers to realize their best work by iterating on choices and identifying process and product defects early and often. When QA kicks off at the start of project, QA Rioters and devs form a mutual understanding of the process risks and set appropriate quality benchmarks for everyone on the team.

Everyone is responsible for the quality of their work and their team’s collective quality

Regardless of what you are working on, you’re responsible for what you put into the product and ensuring it not only functions, but meets or exceeds expectations for fun (or usability, efficiency, etc). Quality Assurance helps devs measure their success and find new and better ways to leverage technology and human intellect, to improve how quickly we can deliver an ever-higher degree of product integrity.

Systems that allow sustainable and scalable testing should be built into the definition of done

For a system or feature set to be done, it needs to work, it needs to be fun, it needs to be usable by our entire player base, and it needs to be stable enough to work next month and next year. Our definition of done each sprint needs to have a way for that sprint’s work to be validated as still working as intended in the future, regardless of what other system we pile on top. For most of Riot that involves various automation toolsets, manual effort, and strong intra-team communication.

The current biggest challenges facing QA at Riot are problems of scale. As we approach a multi-product world, how do we hold onto our core values as a discipline around providing rapid, high-quality, sustainably-delivered features to players, while maintaining a focus on quality ownership as a team?

For systems like this to work, it must be understood everyone is passionate about what they’re working on and fully engaged with the final product. For Rioters that usually means we’ve got to be core gamers, playing our product the way the audience does.

As with any guiding philosophy, our tenets are much easier to lay out in a post than practice on a daily basis. They’re part of the groundwork for a process we’re continually trying to improve.

Is Riot’s approach the right one? We certainly hope so, but maybe you can help fill in the gaps we might not even see. We want to learn more from other folks working within games and software on quality assurance.

We will be hosting an event at GDC on Tuesday, February 28th, so please reach out to GDCQA@riotgames.com if you’re interested in chatting about what works for you, swap lessons learned, or just noodle on ways to evolve the QA discipline together.