A Brief Discussion on Precise Testing: Concepts, Industry Practices & ICBC Development

Source: TesterHome Community

I. Background and Concepts

With the rapid development of the software testing industry, testing concepts and technologies are evolving continuously. A complete set of automated test cases is indispensable for every software company. However, despite massive investment in automated test suites and human resources, it is impossible to accurately identify affected functions when versions are updated, making it difficult to select test cases precisely.

Fintech is advancing at an unprecedented pace, leading to a growing number of banking software applications, faster demand iterations, increasingly complex system architectures, and shorter testing cycles. Banking systems feature long business chains, complex structures, and incomplete decoupling. Relying solely on experience to select test cases cannot ensure accurate testing scope.

To address these challenges, precise testing was proposed in 2012. It is a traceable software testing technology that uses technical means and algorithm assistance to visualize, analyze, and optimize the traditional software testing process. Its core idea is to monitor testing activities, analyze collected monitoring data, and generate accurate quantitative data for quality evaluation.

Minor program changes require additional testing. Quality assessment based on experience is prone to issues, while full regression testing is too resource-intensive. Precise data-driven testing enables testers to clearly understand program changes, testing scope, and test coverage integrity, ultimately delivering more effective test analysis results.

Core Objectives of Precise Testing

• Identify changed code to locate modified programs and precisely define the scope of regression test cases.

• Provide test metrics based on changed programs or call chains to verify full coverage of modified code, help assess testing risks accurately, and establish test exit criteria.

• Precisely locate error causes by visualizing call relationships for failed test cases, accelerating troubleshooting and improving collaboration between development and testing teams.

In summary, precise testing provides effective analysis tools for testing scope definition, test metrics, and defect localization, significantly improving testing efficiency.

II. Industry Practices

(1) iQiyi Precise Testing System

iQiyi’s precise testing supports multi-terminal use (client and server) and both manual and automated scenarios. It integrates multiple internal platforms to streamline workflows and accurately capture code coverage for individual test cases. The server supports system-level series connection, while the client enables multi-component integration, achieving case-level association with code branches or blocks.

The precise testing service acts as the core, establishing a knowledge base of case-code mapping relationships and calculating testing and impact scopes based on Git commit records. Different terminals use different code coverage tools. The server integrates with the environment platform for tool deployment, while the client connects with the build platform.

(2) ByteDance Precise Testing Practices

ByteDance’s R&D process consists of five phases. Precise testing is mainly applied in self-testing, functional acceptance testing, and regression testing. In these phases, businesses obtain recommended test cases for self-testing, functional testing, and regression testing by requesting the SmartEye service.

• Self-testing and functional test cases verify whether code changes cause abnormalities in existing functions, shifting defect detection left and saving time for fixes.

• Regression test case recommendations reduce manual effort and improve regression testing efficiency.

The overall precise testing solution relies on the Code Graph platform, which provides data support for SmartEye. With dual-end static code call chain analysis and full-link topology data, Code Graph performs MR-based change DIFF analysis and upstream/downstream call chain analysis, and calculates code change rates for modified parts.

III. Current Status and Development at ICBC

Currently, ICBC’s precise testing lacks test case recording and recommendation functions.

1. Coverage Collection Types

ICBC has conducted in-depth exploration in coverage collection, supporting multi-language coverage collection including Java, stored procedures, CTP, and MyBatis.

2. Code Coloring System Assisted Testing

The bank retrieves modified methods from Git-submitted Java changes and supports collecting modified stored procedures, MyBatis methods, and CTP files. The coverage tool uses code coloring to visually display incremental coverage for applications supporting multiple languages, helping reduce online risks.

Applications configured with code coloring can obtain method-level coverage of modified programs for monthly versions, which serves as practical data for precise testing to support in-depth analysis, expand promotion scope, and enhance application effects.

3. Static Call Chain Parsing

ICBC explores relationships between heterogeneous call chains for full-link precision, delivering more comprehensive test analysis.

4. Static Call Chain Parsing Categories

It covers three major categories: CTP, stored procedures, and Java.

5. Integration with Internal Platforms

Integrated into the internal PaaS platform, modified applications only need three PaaS parameters to download toolkits and use services. For non-cloud WAS environments, configuration information is obtained from CMDB for automatic nightly deployment.

Data Flow of ICBC Precise Testing

• The bottom-layer Docker image parsing service obtains bytecode content by parsing Dockerfiles.

• The data collection layer sends client coverage results to message middleware via the coverage cluster, while the static call chain parsing service processes static files.

• The data storage layer stores coverage data in Redis, massive coverage and call chain data in HBase, and environment-related method information in Oracle.

• The service layer provides three core functions: real-time modified method coverage collection and display, production-day dimension coverage reports, and in-degree call chain coverage reports.

IV. Further Construction

1. Hybrid Static-Dynamic Call Chain Analysis

Current change impact analysis mainly relies on static call chains, which have limitations (e.g., reflection). Future efforts will integrate dynamic code analysis to refine and supplement call chains.

2. Intelligent Precise Testing

Focus on test case recommendation to form a closed-loop testing process: precise analysis → automated testing → coverage measurement. ICBC Shanghai R&D Center has proposed an Intelligent Specification Push Tool that intelligently extracts information from existing demand data, case content, and test specification libraries to realize intelligent case recommendation without extra investment, improving testing efficiency.

3. Service Layer Expansion

Expand the three existing services to achieve full-link connectivity between heterogeneous programs, support microservice scheduling, and deliver comprehensive test analysis covering call chains based on underlying data support.