Insight On Special Test of Explosive Games: 51CTO Interview with PerfDog Founder Wensheng Cao

Column introduction: "T Frontline" is one of the in-depth interview columns specially set up by 51CTO Content Center for technicians. These technicians including business leaders, senior architects and senior technical experts, were invited by the column group to deeply interpret the current technical hot spots, practices and trends, so as to promote the spread and development of cutting-edge technology.

When it comes to games, I believe many people think of explosive games with DAUs of tens of millions or even billions. However, after enjoying the screaming game experience, players may wonder which company or studio developed it, but they rarely mention how such excellent game performance is achieved.

In fact, in addition to careful planning, research and development, what is more important behind developing a popular game is the game performance and quality testing. To outsiders, game testing seems to be "playing games", but in fact there are very complicated technical details worth thinking about such as: How to ensure the online quality of game products and a good user experience? How are various technical special tests done? What testing techniques and tools were used? What testing technical difficulties will be encountered in different game scenarios? How to improve test efficiency and professionalism? How to coordinate the progress of testing, development, and product teams in these processes?

With these questions, "T Frontline" was lucky to interview Tencent WeTest. Relying on more than 10 years of Tencent product quality management experience, WeTest platform has been honed by thousands of Tencent products, providing high-quality quality testing solutions, which are widely used among enterprises and developers around the world.

PerfDog, as a mobile full platform performance tool launched by WeTest for developers' performance testing needs, has won good comments and recognition from game testing and developers because of its fast positioning, and no need to make any changes to mobile phone hardware, games and applications.

Today, "T Frontline" is honored to invite Mr. Cao Wensheng, the senior testing technical director of Tencent Interactive Entertainment, the performance testing expert of several large DAU products, and the founder of the star performance tool PerfDog, to share with us his practical experience and experience in the field of game testing.

Lag and memory are the main problems of game performance

The topic of performance is one of the hottest topics in the game industry, especially the mobile game industry in recent years. For players, in addition to gameplay, the most important thing that affects the experience is performance. Taking a tactical competitive game as an example, the success of this game is inseparable from the performance factors that can ensure the picture quality and interactive effect.

But for testers, game performance is a complex technical problem involving hardware, operating system, GPU-level rendering and many other aspects. Based on his years of project experience, Cao Wensheng found that the performance problems of the game were mainly concentrated in the aspects of lag and memory.

1. Lag: Frequently faced problem

At present, mobile phone hardware performance is iterating rapidly, and users' expectations are also increasing. As users have higher and higher requirements for game performance experience, their tolerance for stuttering is getting lower and lower. For example, it used to be that users could tolerate a small lag in fifty or sixty milliseconds, but now they feel that the small lag affects his experience.

At the same time, the performance requirements of game development are also increasing. It used to be a good game to reach 30 frames, but now it may be qualified to reach 60 frames. This makes lag the main problem that the game has to face.

However, in the special test of game performance, the lag is an accident. It may not show up in the first test, but in the second test. Therefore, in order to solve this problem, it is first necessary to record the on-site environment of the lag. For this reason, testers need to use professional, convenient and fast tools to repeatedly test, and at the same time, they also need to establish more accurate lag indicators through statistical algorithms, so as to more accurately identify the time and cause of the lag.

With the indicator system, we can better evaluate the power consumption of each frame, and game test specialist can give developers more suggestions on optimizing energy consumption to solve the problem of lag.

2. Memory: A Constraining Factor for Gaming

The game is different from other programs. It involves a lot of image texture loading, scene model and Shader (a technology specially used to render 3D graphics) and others that consume considerable memory.

Insufficient memory will cause the phone to fail to run, or it will crash even if it runs. Therefore, game test specialist need to use various memory tools, such as memory and other tools that come with the engine, to accurately locate which object and which module occupy too much memory, so as to solve the memory problem such as allowing a part to not load too large pictures.

Of course, memory is not as serious as lag, because lag may exist on any platform, even on flagship phones, but memory problems are usually only found on some low-end or low-memory phones.

The growth path of game test specialist: Make good use of tools and learn to review

For game specific testing, identifying and solving problems are the most critical two steps. In Cao Wensheng's opinion, game test specialist should be familiar with and master the use of various special testing technical capabilities and tools if they want to be proficient in this area. At the same time, they should learn while doing, and know how to summarize the experience of previous projects.

1. No universal tools

Cao Wensheng believes that game test specialist should have a certain understanding of the technical form of tools and realize that different forms of tools can do very different things.

The tools used in the special test of game performance can be roughly divided into two categories: embedded SDK integrated original code tools and non-embedded independent tools. The former can collect more comprehensive information, but its use threshold is high (the requirements for development teams and projects are high); the latter can be plug-and-play with a low threshold for use, but the performance indicators obtained may not be so complete .

In addition, Cao Wensheng also reminded that major engine, IDE and hardware manufacturers will have their own performance analysis auxiliary tools, and game test specialist should understand the requirements of these tools. For example, some tools require the compiled version of the game to be the development version, not the regular release version. For another example, for hardware manufacturers like Qualcomm or ARM, the tools they develop may only be applicable to their own hardware and have certain limitations, which require the attention of game test specialist.

Therefore, when conducting tests, game test specialist cannot only use a certain type of tools but combine different tools for testing.

2. Study and love are indispensable

Cao Wensheng believes that it is very important for game test specialist to make good use of previous project experience if they want to efficiently solve the problems in the testing process. How can we make good use of previous experience?

On the one hand, we must learn by doing. There are many technical points that game test specialist need to master, and it is unrealistic to master all the technologies from the beginning. Therefore, game test specialist must have a broad vision, have a certain understanding of performance, network, compatibility, and stress testing, and at the same time, they can also invest time and energy to research the field according to a vertical field that they are interested in or good at., so as to have a solid foundation in the special test.

On the other hand, if you encounter problems or make mistakes in the project, you must study it in depth, so that your memory will be deep and your ability will improve quickly. In this regard, Cao Wensheng talked about his experience in the special test of compatibility in the era of PC games.

For the so-called compatibility problem, a common example is that a game will have a problem of blurred screen, black screen, or abnormal display on many computers. When Cao Wensheng was doing test work, he found that there would be abnormal display problems on some graphics cards.

At that time, Cao Wensheng did not know the reason for this, nor did he know what kind of graphics card would have such a problem. Later, through a lot of graphics card testing, analysis, and summary, he found that there is a reason for this problem: because such games will use some 3D caps capabilities which can not be supported by some graphics cards.

So he quickly developed Bench3D, a virtual graphics card compatibility test tool, and quickly simulated which graphics cards may be problematic, which strengthened Cao Wensheng's understanding of the underlying technology of compatibility, and then quickly improve the efficiency and professionalism of special testing.

In addition to the above abilities, Cao Wensheng also mentioned another essential quality of excellent game testers: love for games.

Game testing is different from normal app testing. When do App-specific testing, you can quickly walk down all the test cases. But the game has tens of millions of ways to play corresponding to tens of millions of operation paths, and each operation path may have its problem. In this case, the tester needs to be familiar enough with the game itself. Only if the tester is interested in and loves the game, the tester can make it happen.

In terms of special technology, the requirements of game testing are also higher than that of general app testing. For example, game special testing has higher requirements in terms of weak network, engine, and security, which means that game testers only loving games enough can face these challenges.

Team spirit is indispensable

1. Identify effective needs

The work of the special testing team is more to serve the R&D personnel within the company. Usually when new work results are produced, the team needs to need to promote them to each R&D team in the company, but these teams will encounter a type of problem: each small group or individual develops a so-called usable test tool which is OK for yourself and the group. But as soon as more project teams and more people use it, they find that the tool is not useful or helpful.

For this problem, Cao Wensheng found that the root cause was that the special test team always regarded the tool developed by themself as a small tool, rather than a big tool or even a product. All aspects of the tool are limited and can only meet their own needs.

It is only when you turn the small tool into a larger tool, or even a product, that you realize that there are thousands of needs to be met. At this point you have to think calmly. Because you can't implement 10,000 requirements in one tool, you need to distinguish what are valid requirements and classify them, and then convert them into general requirements. How to assess which needs are valid?

The first is how many project groups and users have these needs, and the second is the operability and achievability.

2. It is the responsibility to feed back the truth

In addition to testing work, testers may also need to communicate with many of them such as QA, project managers, producters, developers.

Cao Wensheng said that because the testers know the progress and quality of the entire project best, they can also best feedback the real information of the project. At this time, testers may stand on the opposite side of developers in the team. For example, when project developers develop new functions, testers will "specially" find some bugs in the design of new function modules, so some opposition will be occurred, which requires more communication.

In addition to feeding back real information to developers, testers also need to feed real information to project leaders. The project leader needs to understand the progress of the entire project. At this time, the role of the tester is a "guardian" of the project leader: the tester must dare to speak out about the progress and quality of the project.

Sometimes testers play a more important role in the overall game team. In Tencent Games, quality is the last hurdle to determine whether a game can be launched. Game testers have the right to check. If the testers feel that the quality of a game is not enough, then the game will not be online.

Automation of game testing has a long way to go

Although the automations of many traditional app test have done well, Cao Wensheng found that the automation of game tests has not been very successful. According to Cao Wensheng's estimation, the current coverage rate of automated testing is only about 15%.

At this stage, automation only does a good job of releasing a single skill or a single game scene.

Why is the application scope of automation in the field of game testing so small. In Cao Wensheng's view, there are mainly two factors.

First, the real-time rendering of the screen in the game and the translucent special effect particle effect will make it difficult to automate the image. At the same time, game events and system event responses are all realized by themselves, which makes it difficult for automation to capture.

Second, the update iteration of the game version is very fast, and even a very large version update is made every day, which causes the interface to change very quickly, also makes it difficult for automation to be implemented.

Regarding the application of AI to game automation testing, Cao Wensheng's view is that AI may play a role in ordinary app testing, because no matter how the app is operated, the interface changes little and the consistency is strong. In game testing, AI may have difficulty matching the scene. For example, if a certain pixel or several pixels of the game are displayed incorrectly, it is difficult for AI to judge.

Another reason is performance. When the camera rotates a little in the game, what the player sees is completely different, which leads to completely different performance. It is also difficult for AI to find such subtle problems.

Therefore, for testers, it is still necessary to understand which ones can be automated and which ones still need to be tested manually, instead of pinning their hopes on automation.

The end

In the game industry, it is with the silent efforts of the special testing team that players can focus on the operation and gameplay instead of worrying about performance issues from time to time.

For game-specific testers, their work may rarely be recognized by the outside world, but the inner sense of accomplishment is the most important reward.

Guest introduction:

Cao Wensheng(Awen): Tencent Senior Testing Technology Director;

Founder of PerfDog

Chairman of MTSC China Internet Test and Development Conference

    Awen has 10+ years of experience in game-related engine development and engine performance, tool platform development, special technical testing and game development. In the early days, he was responsible for the performance and compatibility testing of Tencent PC game clients, and later dominated the construction of several special tool platforms such as PerfDog, PerfSight, CrashSight, Bench3D, and participated in the special performance testing of Tencent's many well-known mobile game projects. He is currently the director of Tencent Test Technology Center, mainly responsible for the construction of Tencent game quality tools and platforms, focusing on the special field of full-platform performance.