Introduction

Mobile UI automation acts as the foundational quality control measure for App iteration and test efficiency improvement. Three mainstream technical solutions dominate current testing workflows: widget element positioning, screenshot comparison, and multimodal LLM image recognition. Yet all three fail to balance stability, cloud computing cost and runtime observability, bringing persistent pain points including heavy script maintenance, limited compatible scenarios, excessive resource overhead and opaque execution logic that blocks fast failure troubleshooting.

This post elaborates an innovative decoupled architecture: local structured page parsing paired with LLM intelligent decision-making. Instead of sending raw screenshots to multimodal large models, we extract standardized page metadata locally and feed lightweight text data into LLMs. This framework delivers low cost, high positioning precision, complete traceability and convenient iterative optimization. All technical insights shared below are independent original viewpoints published on the TesterHome community.

1. Three Core Pain Points of Traditional Mobile UI Automation

After years of technical iteration, mobile UI automation relies on three mature mainstream stacks, each with inherent drawbacks that cannot simultaneously satisfy demands for testing efficiency, cost control and running stability.

1.1 Widget Locator Automation: Fragile Scripts & Heavy Maintenance Burden

Tools including Appium and UIAutomator2 adopt widget IDs and XPath to locate interactive UI elements. This technical route boasts stable runtime performance but lacks flexibility. Minor page structure adjustments will directly invalidate all associated test scripts.

Common breaking changes include:

• Renaming element ID attributes (e.g., changing btn_submit to submit_btn)
• Adding extra view container layers in the page hierarchy
• Adjusting component nesting logic

For projects with hundreds or thousands of regression test cases, every App version update requires massive manpower to repair broken scripts. Beyond maintenance pressure, this method has obvious scenario limits: it cannot extract complete DOM tree information from H5 pages, WebView components and Flutter interfaces, resulting in failed element matching and interrupted test execution.

1.2 Screenshot Comparison Testing: Poor Cross-Device Compatibility & High False Failure Rate

Many testing teams adopt OpenCV-based screenshot matching to bypass widget positioning limitations, with NetEase open-source AirTest as a typical representative tool. This solution is extremely sensitive to external test environments, generating a large number of false assertion failures.

Tiny environmental differences will trigger abnormal test results:

• Different mobile device resolutions and screen pixel densities
• Page rendering delays causing inconsistent loading states
• App theme color and skin version updates

Low fault tolerance and weak cross-device adaptability make screenshot comparison unsuitable as the core enterprise-grade automated testing solution.

1.3 Multimodal LLM Automation: High Token Cost, Slow Inference & Black-Box Limitations

Multimodal large models have become a popular UI automation solution over the past two years. The standard workflow captures full-page screenshots and inputs pixel images into models such as GPT-5.4 Pro, Gemini 3.1 Pro and Qwen3.7-Plus. The model identifies UI elements and generates operation instructions through pixel analysis.

Although this stack eliminates complex script coding and supports natural language test case generation, large-scale enterprise production deployment exposes four fatal flaws:

1. Excessive Token Consumption
2. A single screenshot occupies 800–2000+ tokens. Long regression test suites with dozens of interactive steps bring continuous, rapidly growing LLM API calling costs.Slow End-to-End Response Speed
3. Multimodal models spend 2–5 seconds parsing image data before outputting the first token. Latency accumulates exponentially with each test step, drastically reducing overall testing throughput.Unstable Element Recognition Accuracy
4. General multimodal models only perform passive pixel matching without UI vertical scenario optimization. Recognition precision fluctuates severely when facing custom App business interfaces.Untraceable Black-Box Execution Logic

When element misrecognition or wrong operation decisions occur, engineers lack clear channels to locate root causes, creating huge barriers for framework iteration and tuning.The above unresolved defects of the three mainstream technical stacks drive the development of this new mobile UI automation framework, which unifies low cost, stable execution and full runtime observability.

2. Core Technical Innovation: Separate Local Visual Parsing and LLM Decision Logic

The core breakthrough of this solution lies in abandoning the industry’s universal practice of delivering raw screenshots to LLMs. It decouples page data extraction and intelligent logic judgment to maximize the efficiency of each independent module:

1. Local visual engine completes full-page parsing, extracts standardized structured metadata including element types, pixel coordinates, interactive status and text content, and outputs unified JSON format data.
2. Lightweight text-based structured data is transmitted to LLMs, which are only responsible for analyzing test business logic and outputting operation decisions.

This restructured architecture brings three irreplaceable core strengths:

1. Dramatic Cost & Latency Reduction
2. LLMs process hundreds of text tokens instead of thousands of image tokens, cutting cloud API expenses and shortening model inference delay significantly.Tunable, Scenario-Oriented Recognition Precision
3. The local visual engine supports small-scale custom dataset training tailored to target business Apps, delivering far higher element detection accuracy than universal multimodal models.Complete Observability & Closed-Loop Iteration
4. All page parsing metadata and LLM reasoning records are fully stored and traceable, supporting precise root cause analysis and continuous framework optimization.

3. Dual-Channel Page Parsing Pipeline: The Framework’s Core Architecture

The framework builds a dual automatic parsing channel: DOM Fast Track and Visual Track. The system automatically selects the optimal parsing route based on page type, merges data from both channels, and generates unified structured metadata to support subsequent LLM decision workflows.

3.1 DOM Fast Parsing Channel (UIAutomator2 Driven)

The system prioritizes the DOM Fast Track by default. It calls UIAutomator2 via ADB to obtain the original screen XML view hierarchy for element parsing, completing a full parsing cycle within only 0.3–0.8 seconds with outstanding throughput performance.

Beyond basic element positioning, this channel extracts native widget interactive attributes including clickable, enabled, checked and selected. These attributes enable 100% accurate judgment of core UI states: button click availability, checkbox selection status, input box focus state and more.

3.2 Visual Recognition Channel (YOLO + OCR + Otsu Threshold Algorithm)

For WebViews, Flutter pages, mini-programs and other interfaces with incomplete DOM tree data that cannot be fully resolved by native parsing, the system automatically switches to the Visual Track. Three collaborative modules complete full-page visual analysis:

1. YOLOv8 target detection: Locate pixel coordinates and category labels of all interactive UI components
2. OCR text recognition: Batch extract text content displayed on page elements
3. Otsu color threshold analysis: Capture element RGB values and brightness data to distinguish interactive states (selected/unselected/disabled/highlighted warning text)

Production verification shows the Visual Track achieves high detection confidence for mainstream business components:

• Mini-program popups: 0.95 confidence
• Page close icons: 0.93 confidence
• Customer service entrance buttons: 0.91 confidence

Color metadata accurately restores UI business semantics: green tabs represent active selection, light gray tabs are inactive, gray buttons stand for disabled controls, and red text marks critical warning prompts. This granular state information lays an accurate data foundation for LLM logical reasoning.

3.3 Standard Structured JSON Sample for LLM Input

Merged data from dual parsing channels is standardized into lightweight JSON schemas, completely replacing raw screenshots to cut LLM computing overhead. Below is a real payload extracted from an e-commerce mini-program promotional popup page:

{
  "page_width": 720,
  "page_height": 1560,
  "elements": [
    {
      "id": "elem_2",
      "type": "mini-program close button",
      "bbox": [643, 61, 685, 102],
      "bbox_center": {
        "center_x": 664,
        "center_y": 81
      },
      "confidence": 0.93
    },
    {
      "id": "elem_4",
      "type": "search icon",
      "bbox": [41, 63, 84, 107],
      "bbox_center": {
        "center_x": 62,
        "center_y": 85
      },
      "confidence": 0.78
    },
    {
      "id": "text_4",
      "type": "text_block",
      "bbox": [230, 69, 490, 103],
      "bbox_center": {
        "center_x": 360,
        "center_y": 86
      },
      "text": "XX Cotton Official Store",
      "color": "dark_gray",
      "color_brightness": 77.0,
      "confidence": 0.83
    },
    {
      "id": "elem_0",
      "type": "promotional popup",
      "bbox": [124, 339, 598, 1163],
      "bbox_center": {
        "center_x": 361,
        "center_y": 751
      },
      "confidence": 0.96,
      "children": [
        {
          "id": "text_10",
          "type": "text_block",
          "bbox": [205, 439, 519, 495],
          "bbox_center": {
            "center_x": 362,
            "center_y": 467
          },
          "text": "Tuesday Member Day",
          "color": "dark_green",
          "color_brightness": 60.0,
          "confidence": 0.77
        },
        {
          "id": "text_18",
          "type": "text_block",
          "bbox": [291, 960, 425, 989],
          "bbox_center": {
            "center_x": 358,
            "center_y": 974
          },
          "text": "Flash Sale >",
          "color": "white",
          "color_brightness": 255.0,
          "confidence": 0.71
        },
        {
          "id": "elem_5",
          "type": "popup close icon",
          "bbox": [334, 1102, 386, 1155],
          "bbox_center": {
            "center_x": 360,
            "center_y": 1128
          },
          "confidence": 0.92
        }
      ]
    },
    {
      "id": "elem_6",
      "type": "button",
      "bbox": [35, 1354, 106, 1435],
      "bbox_center": {
        "center_x": 71,
        "center_y": 1395
      },
      "confidence": 0.54,
      "children": [
        {
          "id": "text_29",
          "type": "text_block",
          "bbox": [53, 1408, 92, 1430],
          "bbox_center": {
            "center_x": 72,
            "center_y": 1419
          },
          "text": "Home",
          "color": "green",
          "color_brightness": 127.5,
          "confidence": 0.67
        }
      ]
    },
    {
      "id": "elem_7",
      "type": "button",
      "bbox": [181, 1353, 248, 1436],
      "bbox_center": {
        "center_x": 215,
        "center_y": 1394
      },
      "confidence": 0.81,
      "children": [
        {
          "id": "text_30",
          "type": "text_block",
          "bbox": [196, 1408, 236, 1431],
          "bbox_center": {
            "center_x": 216,
            "center_y": 1419
          },
          "text": "Categories",
          "color": "light_gray",
          "color_brightness": 193.0,
          "confidence": 0.66
        }
      ]
    },
    {
      "id": "elem_8",
      "type": "button",
      "bbox": [464, 1351, 544, 1433],
      "bbox_center": {
        "center_x": 504,
        "center_y": 1392
      },
      "confidence": 0.82,
      "children": [
        {
          "id": "text_31",
          "type": "text_block",
          "bbox": [475, 1407, 533, 1431],
          "bbox_center": {
            "center_x": 504,
            "center_y": 1419
          },
          "text": "Cart",
          "color": "light_gray",
          "color_brightness": 196.0,
          "confidence": 0.65
        }
      ]
    },
    {
      "id": "elem_9",
      "type": "button",
      "bbox": [610, 1354, 681, 1437],
      "bbox_center": {
        "center_x": 646,
        "center_y": 1396
      },
      "confidence": 0.71,
      "children": [
        {
          "id": "text_32",
          "type": "text_block",
          "bbox": [627, 1408, 668, 1430],
          "bbox_center": {
            "center_x": 647,
            "center_y": 1419
          },
          "text": "My Account",
          "color": "light_gray",
          "color_brightness": 192.0,
          "confidence": 0.63
        }
      ]
    }
  ]
}

With this structured text data as input, LLMs can accurately parse complete page status: a promotional popup overlays the main page, the popup carries marketing copy, the bottom navigation “Home” tab is active, while other navigation entries remain inactive. All logical judgments rely on definite structured metadata rather than ambiguous pixel guessing, fundamentally eliminating recognition errors.

4. Side-by-Side Benchmark: New Framework vs Traditional Automation Solutions

This section compares the structured parsing framework (MobileVision) with widget automation and raw-screenshot multimodal LLM solutions across all core business dimensions, highlighting differentiated competitive advantages.

4.1 Full Dimension Comparison Table

4.2 In-Depth Analysis of Core Competitive Advantages

Advantage 1: Long-Term LLM API Cost Reduction

Single multimodal LLM calls seem low-cost on the surface, yet repeated regression executions for every App release generate cumulative high cloud expenses. MobileVision transfers heavy image parsing work to local servers, only calling low-cost text LLMs for decision logic. Teams can also deploy the entire framework on private on-prem infrastructure to eliminate third-party LLM service fees entirely.

Advantage 2: Full Traceability Enables Closed-Loop Iteration

Pure multimodal image automation is a complete black box. When misrecognition or wrong operation instructions appear, engineers cannot distinguish whether failures come from pixel recognition defects or flawed LLM prompts, with no clear tuning entry points.

MobileVision permanently stores all parsing metadata, LLM reasoning chains, generated operation commands and runtime screenshots to form a replayable complete audit log. When test failures occur, engineers can rapidly classify root causes and match targeted optimization plans:

1. Missing element detection → Supplement annotated training data and retrain YOLO visual model
2. Wrong LLM decision logic → Optimize prompt templates or switch alternative text LLMs
3. Ambiguous test case description → Rewrite natural language test specifications

Generic multimodal models pursue universal generalization but lack adaptation to customized App UI styles. MobileVision integrates a one-click YOLO training pipeline: engineers label around 300 screenshots via online annotation tools to build exclusive detection models for business-specific components, including product cards, specification popups and add-to-cart buttons.

5. Production Environment Real-World Test Results & Verification Data

The framework has completed full production validation on an e-commerce mini-program testing pipeline. The team trained the visual detection model using 800 annotated business screenshots covering 35 types of custom UI components, with measurable benchmark results as below:

6. Applicable Scenarios & Framework Summary

MobileVision is still under continuous iteration, with core business testing pipelines fully online and verified on live e-commerce projects. Its core design logic can be summarized in one sentence: Replace raw pixel image input to LLMs with locally parsed structured page metadata, and leave business test logic judgment to lightweight text large models.

No single automation stack fits all business scenarios:

• Multimodal image-based LLM automation delivers stronger out-of-box generalization for rapidly updated products with highly divergent UI styles
• For long-term regression testing of Apps and mini-programs with stable core UI workflows, MobileVision significantly cuts recurring labor and LLM cloud costs, improves test stability and framework tunability, delivering obvious ROI for enterprise QA teams