Enhancing Game Quality with Tencent's automated testing platform UDT, a case study of mobile RPG game project

Automated Testing 2024-11-04 17:47 367

We are thrilled to present a real-world case study that illustrates how our UDT platform and private cloud for remote devices empowered an RPG action game with efficient and high-standard automated testing. This endeavor led to a substantial uplift in both testing quality and productivity.

As a pioneering force within the gaming sector, Tencent boasts extensive expertise in automation testing, consistently pushing the boundaries with innovative solutions. We are thrilled to present a real-world case study that illustrates how our UDT platform and private cloud for remote devices empowered an RPG action game with efficient and high-standard automated testing. This endeavor led to a substantial uplift in both testing quality and productivity.

1. Tackling Quality Challenges in RPG Games

The automated testing solution for this project encountered two major hurdles:

High DAU and Complexity:This established game lacks a dedicated test server, posing ongoing challenges for quality assurance. These challenges are particularly evident in the intricate peripheral systems, which exhibit a high level of functional coupling and integration. Any modifications could potentially trigger a cascade of issues.
Multiplayer Scenarios:Testing full multiplayer scenarios is exceptionally difficult due to the vast number of players involved.

The game team hopes to use technical means to better ensure the quality of the game in order to overcome these hurdles.

2. Choosing the Right Testing Framework

2.1 Selection of testing framework

Given the team's extensive experience with backend protocol-based testing, they initially leaned towards using it to simulate multiplayer scenarios. However, the experts at WeTest suggested a more effective hybrid approach. Because the RPG game heavily relies on frame synchronization, backend-only protocol testing fell short of accurately replicating real battle conditions. Additionally, it overlooked client-side concerns such as microphone controls, game stability, and performance under poor network conditions.

The hybrid solution combined front-end automation using real devices with backend protocol-based testing, achieving a balance between efficiency and cost.

When it comes to testing frameworks, GAutomator stands out as a stable and efficient option in most scenarios. Furthermore, GA supports the invocation of in-game interfaces, providing additional functionality. Ultimately, WeTest highly recommends that game testers primarily use GA's framework for control recognition, supplemented by calls to in-game interfaces for element identification.

2.2 Implementing Automated Task Distribution in Functional Testing

The WeTest UDT platform facilitates automated functional testing through stable device resource scheduling, aiding development teams. When device malfunctions occur, the UDT platform automatically detects the issue and notifies maintenance personnel. In situations where multiple development teams share UDT resources, WeTest experts recommend dynamically assigning tasks based on the number of available devices.

For instance, based on device availability and time priorities, a functional smoke test task containing 10 cases can be executed either sequentially on a single device or simultaneously across 10 devices, each handling a different case. This approach maximizes resource utilization. The UDT platform can leverage an automated server (svr) solution to effectively address the complexities of multi-device coordination.

2.3. Strategy Design for Multi-Device Collaborative Testing

Based on evaluations, smoke testing for the game's PVE mode requires 2-3 players to team up for battles. The testing process includes the following steps:

All devices log into the main interface.
One device creates a room.
Other devices join the room.
The team leader confirms when the room is full and initiates the battle.
After the battle concludes, the results are aggregated.
Verify that all players receive consistent settlement results (ensuring no synchronization issues).

By integrating this series of testing scenarios into the automation platform, developers can facilitate further interactions during the testing process.

For example, after all devices log into the main interface, they inform the server (svr) through an UpdateReq and continue polling. When the server detects that the number of players is insufficient, it responds with next_state=1 (continue waiting). Once the room is full, the server will reply to the other devices with signals such as next_state=2, next_state=3, etc. Each device can then perform different logical operations based on the signals received. For instance, under the next_state=2 signal, the main device will create the room, while the other devices will join the room.

2.4. Notification of Automated Testing Results

The RPG game development team has integrated the UDT platform into their daily DevOps test task management system, allowing team members to directly monitor the progress and success rates of automated testing tasks on their existing internal distribution platform. To promptly identify issues, WeTest has customized a feature that pushes test results to enterprise communication software. Below is a record of automated test results for a 2v2 battle gameplay.

* Custom Integration with Enterprise Communication Software for Instant Test Result Notifications

Team members can click on Devices in the communication software to be redirected to the UDT device testing details page. Additionally, the UDT platform offers video recording functionality, enabling developers to visually review the entire process of the automated functional tests.

3. Testing scenarios

3.1. Smoke Testing for Multiplayer Gameplay

Smoke testing focuses on a gameplay mode involving all players on the server. Players compete for fixed initial positions and then engage in head-to-head matches until a champion is determined.

Since this is a long, multi-player process, constructing a full-player scenario traditionally requires substantial preparation time, leading to high costs for each test and making regression testing difficult. Due to these cost constraints, past testing could only include a limited player scenario before events began.

With the UDT automated testing platform, these limitations have been lifted:

Automated smoke testing is facilitated through remote cloud-controlled devices, enabling multiple rounds of testing.
Full-player combat simulations can be conducted.
The UDT platform can detect crashes during testing and provide alerts through pop-up notifications.

Leveraging a custom automated testing solution with task collaboration mechanisms allows the development team to conduct tailored tests more easily. For example, if the team wants to ensure that at least n players reach the finals to cover the entire process from the top n matches to the final, WeTest provides a "minimum participation" configuration for UDT. If the number of successfully logged-in or available devices is less than n, the testing task will not proceed to the next stage under UDT control until sufficient devices are available.

If the project team wants to skip the ‘gathering' phase and allow players to join battles immediately after logging in for better simulation of real scenarios, UDT can also implement this by configuring both "minimum participation" and "maximum participation" settings.

When managing a larger number of devices, if the project team needs to quickly identify any malfunctioning devices, the automated server communication enables the team to retrieve the status of devices and the results of checkpoints at any time. Through links in notification messages, developers can directly navigate to the detail pages of problematic devices for issue resolution.

3.2 RPG Character Damage Consistency Testing

The RPG game's character and PvP modules serve as its core components. In previous versions, even minor script changes had an impact on certain characters' behaviors, leading to inconsistencies that players perceived as "unannounced changes." These inconsistencies negatively affected player experience and damaged the product's reputation.

By utilizing automated testing on the UDT platform, the team employed frame synchronization to replay battles and ensure damage consistency across different versions. Thanks to UDT's cloud-based real devices and task distribution system, they were able to harness 50 remote devices to complete over 300 battle replays in just one hour.

This process uncovered 10 critical issues, which were promptly flagged in the team's agile project management platform for immediate resolution and documentation.

4. Conclusion

The WeTest UDT platform delivered outstanding results in the following scenarios:

Automated smoke tests for multiplayer settings with full player populations.
Stability testing for core single-round features, such as the consistency of character damage in PvP battles.
Long-duration gameplay scenarios, including tutorial flows.
Randomized gameplay scenarios.
Full coverage of game instances and characters.

Are you curious about how our UDT platform and real device cloud can elevate your game testing experience? Or perhaps you'd like to dive deeper into our other cutting-edge testing strategies? Either way, we'd love to hear from you. Our expert team is here to connect and provide you with the guidance and support you need to ensure your game testing is both efficient and precise.

Book a Meeting with us!

Furthermore, we cordially invite you to try out Tencent's UDT platform, a cloud-based solution that grants you remote access to devices and seamlessly integrates with your local test devices, thereby broadening your testing horizons. We firmly believe that UDT can bring unmatched convenience and efficiency to your game testing endeavors.

About WeTest

WeTest, with over a decade of experience in quality management, is an integrated quality cloud platform dedicated to establishing global quality standards and enhancing product quality. As a member of the IEEE, approved Global Game Quality Assurance Working Group, it is recognized for its commitment to quality assurance. WeTest has served over 10,000 enterprise clients across 140+ countries.

Focusing on advanced testing tools development, WeTest integrates AI technology to launch professional game testing tools such as PerfDog, CrashSight, and UDT (Next-Gen Multi-Terminal Unified Access Management Automated Testing Platform), aiding over a million developers worldwide in boosting efficiency. Additionally, WeTest offers comprehensive testing service solutions for mobile, PC, and console games, covering compatibility, security, functionality, localization testing and other various services, ensuring product quality for over one thousand game companies globally.

Give it a try for free today. Register Now!

udt

Read Previous Post >>

Try Out WeTest UDT In-Vehicle Infotainment Testing Solution