Source: TesterHome

Introduction

Production monitoring is one of the most critical foundational components of stable online business systems. All production-running applications rely on monitoring mechanisms to sustain stable operation. However, most engineering teams implement monitoring without clear, standardized logic.

During annual business and technical planning, production monitoring was confirmed as a core priority for the new fiscal year. This brings up three essential questions that every technical team must answer:

• What counts as valid, effective production monitoring?
• What are the core goals of online monitoring?
• What complete response workflows should follow after monitoring alerts are triggered?

This article systematically summarizes practical experience for building standardized, high-value production monitoring systems. It covers pre-deployment business analysis, monitoring framework design, classification standards, implementation schemes and value evaluation, helping teams build low-cost, high-efficiency online monitoring capabilities.

1. Pre-Monitoring Business System Analysis

Targeted and reasonable monitoring systems are built on in-depth business and system cognition. Before deploying monitoring rules, teams must fully sort out business system characteristics to avoid blind and redundant monitoring construction. Two core analysis dimensions are included:

1.1 System Lifecycle and Business Stability

Monitoring granularity must match the system’s development stage. For start-up and rapid iteration systems with unstable business logic and incomplete functions, overly refined monitoring is unnecessary.

Teams only need to cover core business scenarios and focus on key indicators closely related to core goals such as user activity and transaction volume. For external user-oriented business systems with high user sensitivity, priority should be given to monitoring high-frequency user operation functions to perceive user-experience anomalies in the first place.

1.2 Technical and Business Strategic Direction

Monitoring construction needs to be aligned with the team’s technical iteration and business expansion plans. When introducing new technologies or optimizing business logic, supplementary monitoring points should be preset to cover new business and technical links.

It is worth noting that unified and consistent technical stacks are the premise of efficient monitoring construction. Dispersed and inconsistent technical architectures will greatly increase the difficulty of monitoring deployment and later operation and maintenance costs. Sufficient pre-analysis can effectively avoid rework and reduce overall monitoring construction costs.

2. Standard Framework for Building a Complete Monitoring System

Based on practical engineering experience, a complete online monitoring system can be built around six core modules: monitoring objectives, monitoring scope, monitoring classification, monitoring stakeholders, scheme implementation and value evaluation. This standardized framework ensures comprehensive coverage and clear logic of the monitoring system.

2.1 Core Monitoring Objectives

The fundamental goal of production monitoring is to improve the team’s online anomaly perception capability. Specifically, it supports teams to quickly detect online faults, accurately locate problem sources, and minimize business loss and operation costs caused by system anomalies. All subsequent monitoring design and deployment work serves this core objective.

2.2 Clear Monitoring Scope Sorting

Defining a clear monitoring scope is the first step of monitoring optimization. Teams need to comprehensively sort out system architecture, technical stack and third-party dependencies, and complete monitoring check and optimization through four core steps of "check, sort, eliminate and supplement".

• Inventory existing monitoring points: Sort out all current monitoring rules from the functional business coverage layer and system resource coverage layer, and verify the actual application value of each monitoring item.
• Clean up invalid monitoring: Eliminate obsolete monitoring points generated by business iteration and function scrapping. Useless monitoring data will interfere with problem investigation and cause alert noise.
• Supplement missing monitoring blind spots: Add monitoring rules for core functions and key business data that lack perception mechanisms, avoiding problems that can only be found through user feedback.
• Optimize alert mechanisms: Improve missing alert channels to ensure that abnormal monitoring information can be accurately pushed to responsible personnel in real time.

2.3 Two Core Monitoring Types: System Monitoring and Business Monitoring

Online monitoring is divided into two independent and complementary categories: system monitoring focusing on underlying operating environment health, and business monitoring focusing on upper-layer business operation status. The two types jointly realize full-link coverage of online systems.

2.3.1 System Monitoring

System monitoring focuses on the basic operating conditions required for business system operation, covering four core layers: resource layer, basic architecture layer, data layer and dependency call layer.

• Resource Layer: Monitor basic operating hardware and network resources, including CPU utilization, memory usage, disk occupancy, network traffic, DNS access status and load balancing equilibrium. It also tracks intermediate network services and message middleware indicators such as message accumulation, to avoid business blockage caused by unbalanced resource allocation.
• Basic Architecture Layer: Targets underlying technical operation status. Taking Java systems as an example, it focuses on JVM operating indicators, including thread occupancy, memory recovery efficiency, memory leakage risk, method call status, stack usage rationality, code exception throwing and log completeness.
• Data Layer: Monitors database operation status, including database proxy connection number, slow query quantity, data table volume, data reading and writing delay, and tracks abnormal time-consuming data operation links.
• Dependency Layer: Tracks the calling status of cross-system and third-party dependencies, including call success rate, response delay, abnormal call and unauthorized call behaviors, to ensure the stability of the whole call link.

2.3.2 Business Monitoring

Different from universal system monitoring, business monitoring is highly customized based on actual business scenarios, mainly covering functional operation monitoring and business data monitoring.

Functional Monitoring realizes real-time perception of user operation and system function anomalies through business point burying:

• Monitor abnormal user operations such as frequent repeated operations and function access restrictions;
• Track the execution normality of benefit-related functions such as coupon and prize claiming;
• Detect link anomalies in associated business interfaces to locate single-point failures causing overall business blockage;
• Collect front-end page anomalies including white screen, loading timeout and compatibility errors;
• Supervise the execution status of scheduled tasks such as automatic prize distribution and data synchronization to ensure normal business cycle operation.

Data Monitoring focuses on the rationality and stability of core business data:

• Set threshold alarms based on year-on-year and month-on-month data changes to perceive abnormal fluctuations of daily business report data;
• Monitor key business data such as prize issuance volume, user claiming volume and inventory surplus, and trigger alarms for over-limit data and insufficient inventory;
• Verify the rationality of data calculation logic, automatically alarm for abnormal data that does not conform to business rules, and avoid low-efficiency manual data checking.

2.4 Clarify Monitoring Stakeholders and Responsibility Division

Different roles in the team have different attention priorities for monitoring data. Reasonable responsibility division can avoid invalid information transmission and improve problem response efficiency.

• Business and Product Teams: Focus on user behavior data, business conversion indicators and core data fluctuations, which provide basis for product demand optimization and business strategy adjustment.
• Technical and Operation Teams: Focus on system resource anomalies, program error logs and underlying operation failures, responsible for system stability maintenance and fault repair.
• Dependent Party Teams: Pay attention to the call stability and data normality of cross-team dependent interfaces.

Unreasonable alarm distribution (such as pushing underlying system error alarms to product personnel) will lead to redundant information transmission and delay fault handling. Clarifying the corresponding relationship between monitoring indicators and responsible personnel is a key link of standardized monitoring.

3. Standardized Monitoring Implementation Scheme

After completing the sorting and planning of monitoring points, teams can implement landing construction through standardized monitoring schemes. A complete monitoring implementation plan includes seven core dimensions:

1. Monitoring Layer: The underlying layer corresponding to the monitoring object (system resource layer, business function layer, data layer, etc.)
2. Monitoring Item: The secondary classification of monitoring indicators (CPU, memory, user access, etc.)
3. Monitoring Point: The finest granularity monitoring indicators (network IO, packet loss rate, task execution success rate, etc.)
4. Monitoring Tool: Technical tools and schemes for data collection and statistics
5. Early Warning Strategy: Custom threshold rules for triggering abnormal alarms
6. Alarm Carrier: Information push channels such as email and DingTalk
7. Responsible Stakeholder: The team/person in charge of abnormal processing

For system monitoring with complex indicators and wide coverage, single-point manual deployment is inefficient and difficult to maintain. The industry mainstream Prometheus + Grafana monitoring stack can realize full-dimensional system data collection, visual display and automatic alarm, covering most system monitoring scenarios of online business systems.

4. Monitoring System Value Evaluation Mechanism

Monitoring construction is not a one-time deployment, but needs continuous value evaluation and iterative optimization. The core evaluation standard is whether it improves online problem perception efficiency and reduces troubleshooting costs.

4.1 Effective Monitoring Value Performance

High-quality monitoring systems can accurately locate abnormal data and faulty links, reduce manual investigation time, and push alarm information to accurate responsible personnel, realizing fast perception and rapid disposal of online problems.

4.2 Common Monitoring Problems and Optimization Suggestions

Many monitoring systems have construction deviations, resulting in reduced practical value:

• Alarm fatigue caused by invalid alarms: Excessive redundant and meaningless alarm information makes the team ignore real key faults, losing the monitoring warning effect.
• Excessive monitoring intrusion: Complex buried point logic increases system code intrusion, bringing potential online bugs and affecting system stability.

Teams need to regularly evaluate the actual effect of monitoring points, eliminate invalid and low-value monitoring rules in time, and retain only high-value monitoring mechanisms that can effectively reduce operation and maintenance costs.

5. Conclusion

Effective online monitoring relies on standardized construction and accurate alarms rather than quantity. All monitoring work must be based on actual business needs, with systematic planning and refined landing.

Whether building monitoring systems or other operation and maintenance architectures, taking business value as the core starting point can effectively avoid redundant construction and resource waste. This paper sorts out the complete process of online monitoring construction, hoping to provide systematic reference for technical teams to build standardized and efficient production monitoring systems.