Comprehensive Approach to Compatibility, Security, and Performance Testing in Mobile Games

Introduction

Since 2015, the mobile gaming market has become more stable, with a focus on high-quality games becoming an essential aspect of development. In the domestic market, the growth of mobile gaming has slowed down, and the cost of acquiring users has increased. This situation means that if a mobile game's quality is not up to par, competitors with similar products will quickly take over the market. By the time developers realize and fix the quality issues, the market may already be dominated by their competition. As a result, maintaining high-quality mobile games has become crucial for developers to stand out and achieve steady growth in this competitive industry.

These Issues Negatively Impact Mobile Game Developers:

Game crashes and unexpected exits

Compatibility issues are among the most frequent problems faced by mobile games before their release. Many domestic hardware manufacturers create custom ROMs and system versions for Android devices, such as Xiaomi's MIUI, Huawei's EMUI, and Lenovo's VIBEUI. This leads to a massive number of version combinations due to multiple manufacturers and versions, resulting in various adaptation challenges.

Game cheats and security vulnerabilities

Successful mobile games can be highly profitable, and the booming market attracts numerous groups seeking illegal gains through cheats, game assistants, and studios. These activities seriously impact the game's revenue, balance, and overall lifespan. Experts from Tencent's mobile game security testing team have showcased the damage caused by cheats on multiple occasions.

To safeguard their game's revenue, many developers of popular games have established dedicated security teams to combat game cheats and protect their products.

Game lag and stuttering

Screen glitches, stuttering, and lag in-game performance may not directly stop players from gaming, but they significantly impact the gaming experience and can gradually lead to users abandoning the game. According to public data from Tencent WeTest, memory issues in mobile games are particularly severe on lower-end models, with role-playing, action-adventure, flight shooting, and online games all experiencing memory overruns. The overall performance issues of mobile games are concerning and warrant developers' attention.

Tencent Games' Solutions

Tencent Games faces similar challenges as other developers when it comes to mobile gaming. To address these issues, Tencent has set up specialized testing teams for each type of problem, creating a comprehensive testing process and mechanism to ensure quality. This process is known as Tencent's mobile game forging process, which is familiar to external developers.

Compatibility testing, security testing, and performance testing are all crucial stages before mobile games are released, highlighting the significance of these aspects. In the following, we will discuss how Tencent Games specifically tackles these issues and the standards and principles they follow.

Compatibility Testing

Before a mobile game goes live, it must undergo comprehensive compatibility testing for the following six aspects:

During version updates, it is essential to test at least four aspects - installation, startup, login, and beginner's guide - to ensure the quality of the update. The requirements for high-quality internal games are even more stringent, with key game scenes undergoing in-depth testing to guarantee a positive player experience. For instance, in a popular mobile game like "Honor of Kings", compatibility testing is repeatedly conducted in core PVP scenes during version updates. This attention to detail is a crucial factor in maintaining the market-leading position of "Honor of Kings" amidst competition from similar products.

At Tencent Games, most compatibility testing is still performed manually, leading to high testing costs. In the early stages, when the team was understaffed and facing tight deadlines, we considered using third-party automated compatibility testing services. However, after trying them out, we encountered several issues:

1. Insufficient accuracy and completeness of the problem description

Our team has provided feedback that a problem must be thoroughly verified for its inevitability. A problem is considered inevitable if it occurs 100% of the time in more than five instances. Additionally, we need to provide comprehensive information and assess the number of people affected by the current defect based on market share, suggesting a priority level for addressing the issue. However, third-party services often raise numerous concerns, necessitating extensive testing efforts to identify genuinely significant problems. This process increases the human resource cost of validation work, making it challenging for our American counterparts to understand.

2. Limited testing depth, challenging to cover essential functions

Third-party services typically test up to the beginner's guide level. However, in reality, Tencent Games' compatibility testing needs to cover at least the core functions and major scenarios. This limited testing depth makes it difficult for American audiences to understand the full extent of the testing process and its coverage.

3. Unable to perform complex operations

Professional testers have a high sensitivity to defects and are accurate in locating them. They can perform complex operation steps, such as action, racing, or random events. They are irreplaceable in testing sound effects, gravity sensing, AR/VR, and smart toys.

Overall, efficiency has improved, but the service effect has declined significantly. To ensure the quality of game testing and respond to the company's strategy of high-quality games, we ultimately chose quality over efficiency. We are willing to spend more time and cost to achieve better results.

Although compatibility testing is mainly manual, we have never given up on the pursuit of efficiency. Our team has been researching automated testing solutions. These include standard compatibility tests that can quickly verify compatibility, multi-device synchronization assistants that ensure manual testing quality while improving testing efficiency, and targeted development of the GAutomator framework for projects with heavy compatibility requirements, such as the 5v5 automated testing in "King of Glory." This framework has been recommended by Unity officials and is currently a standard framework for Tencent Unity games, applied in various testing scenarios of numerous projects such as "Naruto" and "Dragon Nest."

Currently, Tencent's mobile compatibility testing team is not only responsible for the compatibility quality of all Tencent games but also provides external developers with the same quality compatibility testing services through the WeTest platform.

Mobile Game Security Expert Testing

At Tencent, security module testing is typically integrated during the transition of a game version to functional testing. Tencent's WeTest mobile game security testing team categorizes the content to be reviewed into static and dynamic security vulnerabilities. Various types of issues require different testing concepts and analysis methods, resulting in differences in testing tools and approaches.

1. Static Vulnerability Scanning

This process involves automatically scanning the game APK's configuration files, resource files, script files, manifest.xml, and .so files to identify potential security issues. This content is integrated into the mobile game security testing plan, which covers over 120 security inspection and information security testing items.

2. Dynamic Risk Analysis

This method analyzes security risk points based on the game's specific content and gameplay. By examining protocols, functions, memory, scripts, and other technologies, the game server is assessed from multiple angles to determine if there are comprehensive validations or anti-cheat strategies in place for the corresponding risk points. This ensures that the game is secure and resistant to potential cheating or hacking attempts.

The mobile game security testing team has refined vulnerability types and identified risk points based on potential profit points within the game. For different games, specific gameplay elements are considered to derive corresponding profit methods. By analyzing these risk points, the security-related aspects of the game can be thoroughly covered. Along with checkpoints, actionable test cases are generated, forming a closed-loop process. This comprehensive approach ensures that potential vulnerabilities are identified and addressed, making the game more secure for players.

Mobile game security testing demands a high level of technical expertise. It necessitates the involvement of professional technical personnel to conduct reverse analysis and provide tool support. Without such expertise, the aforementioned testing points may remain theoretical discussions without practical implementation, thus emphasizing the importance of skilled professionals in ensuring the security of mobile games.

Client-side Performance Testing

Tencent places significant emphasis on the testing and monitoring of mobile game performance internally. Before releasing a new version, they repeatedly confirm whether the performance meets the required standards. Even a 1% deviation can result in the version being rejected and unable to be released.

To address specific issues, the internal expert team has broken down common user performance problems, such as crashes and screen glitches, into specific performance indicators. Optimizations are then made based on these indicators.

To enhance efficiency, Tencent's WeTest mobile game performance optimization team has developed the Cube tool. This tool assists users in identifying general performance, resource, and mono memory issues in mobile games.

Introducing a comprehensive solution for mobile game quality enhancement

External mobile game developers often face challenges in addressing issues that require specialized teams within Tencent Games, leading to increased time and effort.

To enhance developers' efficiency and reduce costs, Tencent WeTest presents the "One-Stop Mobile Game Quality Improvement Solution," featuring professional techniques and tools. Developers simply need to assess their game genres, follow the launch process, and select the services that best suit their needs.