Best Testing Practices in DevOps

Introduction

In the realm of DevOps, everything can be automated through pipelines to enhance productivity. We've found that the testing phase also requires a streamlined capability to ensure stable and high-quality delivery across numerous development pipelines. This article will mainly introduce how to build comprehensive high-level testing capabilities in DevOps.

Changes in culture, process, organizational structure, and technology have brought about new requirements for testing.

 

Culture: New requirements brought by DevOps culture for testing

To adapt to the rapid changes in the market, enterprises are required to have fast product iterations and flexible responses to user needs, which give rise to DevOps.

In "Continuous Delivery 2.0," the author simplifies and summarizes DevOps into two loops: value discovery and rapid validation.

Value discovery is the rapid identification and recognition of the real needs of external customers, creating value points for them. The "rapid validation loop" requires enterprises to quickly implement and land value points after identifying business problems and setting business goals.

Testing belongs to the "rapid validation loop," which requires close cooperation between development, testing, and operations roles to efficiently and high-quality land validation of new features.

 

Process: Difficulties of building testing work in DevOps

Under the trend of DevOps, the test department has changed from a large number of centralized tests to a high-frequency rapid test.

In the past, most enterprises used pure manual testing, which was fundamentally unable to meet the high-frequency and fast-paced demands of DevOps. Breed the demand for automated testing.

Organizational structure: The status quo of the testing department of leading enterprises

A high proportion of human outsourcing:

The ratio of outsourced manpower to formal manpower in large enterprises such as finance/communication/aviation is often more than 5:1, with high personnel mobility and uneven quality. Put forward requirements on the stability and usage threshold of tools and systems.

From centralized to decentralized and back to centralized:

The initial business of the enterprise is relatively simple, and the test requirements are grouped into a unified test department.

With the expansion of enterprise business, in order to quickly meet the testing requirements of each business, testers are directly assigned to each business group to realize rapid development, test and release within the business.

After the business volume becomes larger, avoid decentralized R&D and make each business group repeatedly fall into low-level R&D activities, resulting in technology silos. In order to realize the inheritance and accumulation of technical assets, the R&D process tends to be carried out from decentralized development to centralized infrastructure, that is, The concept of "China Taiwan" is often said now. As a result, testing has changed. In addition to testing the specific functions of each business, it is also necessary to conduct a unified test on the quality of the infrastructure itself and the special capabilities of each module to ensure that the overall health is maintained at a controllable standard. Therefore, there is a need for centralized testing.

The multiple forces of culture, process, organizational structure, and new technology (container technology, etc.) promote the agility of testing.

Based on the new requirements of DevOps for testing, there are more and more automated testing tools on the market. Faced with a large number of tool systems, developers often need experience and time costs to screen:

Question 1: In which links to join the test? What types of tests are applicable to each link to ensure the quality of this link?

Question 2: Under the constraints of personnel structure and organizational structure, what testing methods and tools should be selected for each link?

Let's analyze these issues one by one.

Question one: Where can the test penetrate?

In the DevOps culture, the emphasis is on breaking down the barriers between different functions. For the testing department, it means "shifting to the left" and "shifting to the right" of testing activities. From requirement analysis to product launch, quality is controlled in every link. In some links that are partial to R&D and partial to operation and maintenance, testers can help establish a complete set of quality evaluation systems and tool sets to ensure the overall quality of upstream and downstream.

For example, in the development and coding process, it is mainly unit testing and code review. For most companies, this work is mainly done by R&D personnel. The things that need to be done in the test, promote the quality awareness of this link, and help the development students to get the tools and result records of unit testing and code review, so that there are traces to follow.

The test time is advanced:

The test no longer waits for the development to be completed, but intersperses the test time in the development stage to reduce the length of the test period.

Unit testing in advance:

After the development of each module is coded, unit testing is performed on this module first. The business logic is relatively clear, and there is no need to review it again, which is more efficient.

Unit testing is well-founded:

The test provides the use case design of each module before the development of unit testing for reference in development, making unit testing more comprehensive.

Unit test review:

After the unit test of each module is completed, the test will conduct a unit test review to make the unit test more comprehensive and the code quality higher; at the same time, if the problem of the code or unit test is found, the development will be revised in time, without packaging, and the test time will be shortened.

Close cooperation between development and testing:

Each module requires close cooperation and completion of development and testing. The closer cooperation between testing and development, the faster the development and testing level, and the faster the testing ability to read code.

Reduced testing workload and test cycle:

Since the unit test is sufficient, the test of each job can be omitted, and only the integration|joint debugging test is done, which greatly reduces the test workload and test time, thereby shortening the entire project cycle.

Question two: Under the constraints of personnel structure and organizational structure, what kind of testing methods and tools should be selected for each link?

Test Management

The goal of management tools is to increase efficiency and collaboration. There are already many mature management tools on the market. If it is a giant enterprise with complex business, the test department is more concerned about the adaptability to different test management systems.

To give two examples, the following two pictures are the processes of the testing departments of two companies with very good technical accumulation, and the differences are obvious.

In this actual environment, the requirements for the flexibility of the test management platform are very high.

Flexible and Configurable Workflow

Can be integrated with existing management systems

Visualization of process/results

 

At present, our platform supports organizing test plans by workflow, assigning tasks, and visually tracking project progress and manpower allocation.

The test itself is also a pipeline, which supports users to wirelessly expand the test link to the right; each test link supports downward wireless expansion of the "test task item.

At the same time, the platform supports the integration of management systems such as jira/tapd, and opens up requirements and defects. In terms of scripts, it supports connecting git and svn, and directly synchronizing scripts to the WeTest testing platform.

UI Testing

UI testing is the lowest threshold and the most common type of testing. Generally, it is more commonly used in the functional acceptance and special testing stages.

There are web and mobile terminals for UI testing. The test on the web side is mainly based on the selenium framework. There are also more general recording and playback tools on the market. Due to the large number and variety of device models, UI automation testing on the mobile terminal brings greater challenges to the testing department. In recent years, more and more mobile terminal testing frameworks have emerged.

As mentioned earlier, the proportion of outsourcing in the organizational structure is relatively high. Therefore, the threshold for using UI automation testing tools must be low. There are many script recording and playback tools in the industry. In practice, these tools are often found to have the following weaknesses:

1. Difficulty identifying elements

The out-of-sequence password disk and anti-screenshot security controls in the financial industry have blocked many use cases from the door. In fact, most problems with out-of-order cipher disks are solvable.

2. The success rate of script playback is low

Low robustness: For scripts with sequential flow, as long as there is one step that does not follow the preset process during playback, it will get stuck, and subsequent scripts will be useless.

Low adaptability: The recorded scripts cannot be played back due to different resolutions and sizes on different devices. This puts forward higher requirements for the identification method of the tool, which cannot be a simple coordinate identification.

3. Poor usability affects efficiency

In order to improve the recognition efficiency, many script recording tools use image + control dual recognition. The image recognition process often requires the user to select a region to be recognized. Greatly reduces the efficiency of recording scripts. It also raises the threshold for tool use. We expected a sensorless recording tool. The user recorded the script by the way during manual testing.

These points are all solved by our self-developed gadget UITrace. At present, this tool is also the main tool used by WeTest to deliver compatibility testing tasks.

In addition to tool issues, since mobile UI testing involves the operation and maintenance management of a large number of devices, on the basis of stable operation, effectively reducing operation and maintenance costs and improving operation and maintenance efficiency are the prerequisites for daily UI testing.

For the operation and maintenance of hardware, WeTest summed up 43 operation and maintenance rules in the process of managing tens of thousands of devices, which can automatically identify and deal with "developer mode closed by mistake", "memory is full", "screen off and lock screen", etc. question. Ensure the stable operation of the computer room 7x24 hours, and achieve the effect that one operation and maintenance personnel can manage thousands of devices.

Interface Testing

Interface testing is a cost-effective testing activity. Compared with UI, the interface has little change and is relatively stable. Interface testing mainly focuses on the following points. If you can do all these points, you can basically cover more than 90% of the needs.

1. Adaptability:

The range of supported protocols/message formats is more comprehensive (such as dubbo/grpc/trpc that has emerged in recent years, and the classic http/https, which can be supported by the WeTest platform)

2. Ease of use and management capabilities are taken into account:

Real-time debugging like postman, but also support management functions of use case management, test tasks, and report management, which is convenient for reuse and historical traceability. At present, the WeTest platform can meet the demand

3. Mock ability:

Mock rules can be customized according to requirements

4. The ability to automatically generate test cases to organize tests:

Provides fuzz security testing: supports automatic generation and execution of attack simulation scripts such as random padding, SQL injection, XSS attack, and OS command injection.

5. Preprocessing script coding ability: flexible logic control

6. Upstream and downstream link integration:

Support link interface testing, not just a single interface. The output of the previous interface is used as the parameter input of the subsequent interface

Stress Testing

If there is a server downtime, the business will be paralyzed; if the delay is high, the user experience will be significantly affected, resulting in a decline in word-of-mouth. The server stress test mainly focuses on the following points:

1. Concurrency:

Large and stable concurrency is the basic element and the prerequisite for stress testing. At present, the concurrent volume of WeTest can reach one million levels.

2. Simulate real scenarios:

WeTest supports building context link scenarios through interface parameters, simulating the concurrency of each interface in the real environment.

3. Compatible with various scripts:

Mainstream script frameworks such as jmeter/fiddle

4. Support coding mode, multiple protocols and languages, which is convenient for building test scenarios flexibly.

5. The monitoring dimension is scientific and comprehensive:

Covering basic performance indicators such as TPS, response time, sending and receiving packets, and 14 items of data such as process-level servers, see the screenshot of the WeTest stress test report below.

Other

In addition to solving the problems of the above tools and systems, we must have the awareness of setting quality red lines on the assembly line

The quality red line is not only used to ensure the quality of retesting/release, but also the code quality of Git workflow. Avoid low-quality code from entering the next link, wasting downstream testing resources. Examples are as follows:

About WeTest

WeTest Quality Open Platform is the official one-stop testing service platform for game developers. We are a dedicated team of experts with more than ten years of experience in quality management. We are committed to the highest quality standards of game development and product quality and tested over 1,000 games.

WeTest integrates cutting-edge tools such as automated testing, compatibility testing, functionality testing, remote device, performance testing, and security testing, covering all testing stages of games throughout their entire life cycle.