Performance Testing for Large-Scale Gaming Applications

Gamers are in awe with games that are not only engaging but also perform exceptionally well. Performance means that the game logic is able to run flawlessly under heavy load, without having much impact on the device temperature. From a game developer’s perspective, it is important to focus on performance since it plays a vital role in the user experience.

Experiences like sluggishness, crashes, freezes, etc. can irk the gamers and they might end up embracing your competitor’s product. This point holds good for native games as well as cloud-based games where data like leaderboard, achievements, etc. are updated on a real-time basis. With the advent of the COVID-19 pandemic, skill-based games^[1] like Poker, Rummy, etc. have received massive user adoption.

Momentum of skill-based games will only rise with each passing day. In this highly-competitive environment, gaming companies and game developers must leave no-stone unturned to ensure that the game experience is not deteriorated under heavy load (or stress).

Performance testing strategy for mobile (or desktop) games largely depends on the game type, graphics, multiplayer functionalities, and other such features that are unique to every game. Automated performance testing on real devices and desktops can go a long way in building a premium gaming experience.

KiwiQA has provided performance testing services to a wide range of gaming companies across varied skill & gaming volume spectrums. The end result is a robust game engine that offers unparalleled experience to the gaming audience. Now that the platform is set, let me deep dive into some of the performance testing tactics that can be applied to large-scale gaming applications.

Also Read: Key Performance Metrics for Effective Performance Testing

Different Forms of Performance Testing

For starters, performance tests are conducted to ensure that the features (or functionalities) of the app/website work as expected when it is under heavy load.

CPU load, GPU load & frequency, memory consumption, and FPS (Frames per Second) are some of the essential metrics used for performance testing.

In case of online games, the test can be used to check how the game performs when the game infrastructure is heavily stressed (or loaded) with concurrent sessions. With rigorous performance testing, game developers and/or gaming companies can unearth issues related to gameplay graphics, performance, etc. and optimize the game accordingly.

Here are some of the major forms of performance tests:

Performance Testing

Considered as the basic form of performance testing, it helps in understanding how the game play and/or gaming infrastructure can be scaled to achieve better performance numbers. Continuous monitoring of FPS, CPU, memory, and other such important hardware parameters helps game developers to scale up/scale down the game functionalities to realize good performance.

Game developers can use these tests to make their games less CPU/GPU intensive so that games can also run on low-end devices. The results of basic performance tests lays the plan for stress testing and load testing.

Stress Testing

As the name indicates, stress testing is a category of performance testing where the behavior of the gaming application/system is checked when the gaming infrastructure is subjected to heavy stress. An example of stress is a large number of concurrent sessions logged on to the server that houses the gaming infrastructure.

Stress testing gives an accurate idea about the load when the game will stop responding and how the error will be displayed to the end-user. Stress testing is helpful in games that require constant connectivity with the servers – either for game play updates, leaderboard updates, etc. Stress testing also lets you unearth the hardware failures when the device running the game and/or gaming infrastructure is put under heavy stress.

Soak testing (or endurance testing) is a major form of stress testing that is used to check the system (or game) behavior under sustainable stress. Vital game performance metrics like CPU, FPS, memory, etc. are monitored when the game and its associated infrastructure is subject to continuous load.

Spike testing is another form of performance test that is used to check the game behavior under sudden stress. The stress could be a sudden increase in the number of concurrent users, increase in the number of database transactions, etc. The main purpose of spike testing is to determine the system performance when there is a dramatic change in the load.

Load Testing

In load testing, the game and corresponding infrastructure are exposed to a certain amount of load. The purpose of load testing is to identify the performance bottlenecks when the game is subject to load. By the end of the test, team members will have a clear understanding about which particular areas require optimization.

Since load can be applied on the game and/or infrastructure, it is simulated by issuing frequent upload/download requests, querying databases, or giving a range of inputs to the gaming infrastructure. As most of the online mobile games are multiplayer games, game developers can leverage load tests to deep dive into concurrency issues related to the game (and its infrastructure).

Since the gaming device is also put under load; parameters like CPU, battery, temperature, memory, etc. are also monitored during the process. It is used to gauge understanding about how the hardware behaves when it is under differential load.

Also Read: Performance Testing vs. Load Testing vs. Stress Testing: The Key Differences

Key Focus Areas For Game Performance Testing

A majority of the online gaming companies provide game play options to users whereby they can run the game on different devices like desktops, tablets, mobile phones, etc. Hence, the performance testing strategy must encompass testing on all the different device viewports, particularly those that are used extensively by the target audience.

Here are some of the key areas that must be a part of the performance testing strategy:

• All the essential blocks of the game (i.e. device, infrastructure, servers, etc.) must be tested for stress. This helps in unearthing issues not only in the game play but also in the infrastructure related to gaming.

Integration with social media channels, third-party APIs, etc. must also be considered for performance testing.

• Like any other form of testing, all the test activities and test reports must be logged in order to check how far performance tests have impacted the game quality.

Logs must be maintained for every build, so that game developers, QA engineers, and other relevant project stakeholders can have a detailed look at the progress of key KPIs of performance tests.

• Consider performance testing on real mobile devices with different viewport sizes. Priority must be given to tests on game UI and game logic since they are pillars on which the entire game is built.

Animation flows, font rendering, and other things related to game aesthetics must also be considered in performance testing of games.

Gaming companies can onboard performance testing company like KiwiQA that help achieve best-performing gaming apps. KiwiQA uses K-SPARC – an in-house performance testing framework that traverses through every stage of the performance testing pipeline with major focus on throughput and application load.

Also Read: Jmeter Vs LoadRunner: Which One Is Better for Performance Testing?

How KiwiQA helped a major Gaming Company with Performance Testing?

A major multiplayer gaming company approached KiWiQA with a performance testing requirement – access to the product APIs developed must be consistent (in performance) even when a large number of users are simultaneously using those APIs. The response time of the modules must not be impacted even if close to 1000 concurrent users are accessing the modules.

The performance testing team at KiwiQA analyzed the aspects of the game APIs and a performance testing strategy was created. Testing was performed in two rounds. In the first round, performance tests were conducted on the production-like gaming infrastructure and the identified issues were assigned to the respective dev teams. Akin to continuous testing, the second round of testing involved verifying the game functionalities with the fixes patched from round one.

The K-SPARC framework was leveraged for running performance tests on the game. As a first step, the APIs with inherent higher response times were identified using full-blown unit tests. Once fixed, they were subjected to performance tests in order to identify & fix performance (or scale) level issues.

All the APIs were subjected to ~500 parallel virtual users and the RedLine13 tool was leveraged for analyzing performance bottlenecks. The NewRelic APM tool was used by the performance testing teams for monitoring the status of servers on which tests were executed.

Various performance metrics like CPU utilization, memory consumption, FPS, etc. were continually monitored for APIs with 3%~35% CPU utilization on database and application servers.

By performing rigorous performance tests using the combination of K-SPARC framework, RedLine13 tool, etc; the team at KiwiQA was able to identify all the lingering performance issues for the gaming company.

Conclusion

Performance testing is a must if you have to release a game that can be used by a large number of concurrent users. Gaming companies need to prioritize which form of testing (e.g. Load Testing, Stress Testing, normal Performance Testing) must be taken up for testing the game from a performance and stability point of view.

Many gaming companies partner with outsourced QA vendors that offer performance testing services so that the games can be developed and tested in a continuous fashion. Such a collaboration also helps in accelerating the time to market as well.