In October 2025, one of the world's largest cloud service providers, Amazon Web Services （AWS）, experienced a rare large-scale outage.

This incident caused applications including Snapchat, Duolingo, Slack, Roblox, Signal and other applications to be paralyzed globally for several hours, affecting tens of millions of users.

As a direct result, Snapchat can't send messages, Venmo payments are paralyzed, Roblox games are out of service, and even the Her Majesty's Revenue and Customs （HMRC） system has come to a standstill.

What’s even more astonishing is that Catchpoint estimates that the direct economic losses caused by this incident exceed 10 billion US dollars. If hidden costs such as business shutdowns and reduced productivity are included, the total may exceed 100 billion US dollars.

As Mike Chapple, a professor at the University of Notre Dame, said:
“When the cloud giant sneezes, the entire Internet catches a cold.”

AWS issued a statement afterwards, confirming that the cause was a latent defect in an internal DNS automation system, which caused the endpoint resolution of its core database service DynamoDB to fail, triggering a chain reaction that ultimately affected the control plane of the cloud computing infrastructure.

This incident became one of the most typical cases of "automation error → system-level crash → global impact" in the history of cloud computing. But from a testing perspective, it was much more than an “operations incident”—it revealed a long-ignored blind spot in modern software testing.

 What problems did this AWS outage expose?

1. "Black box risk" of automated systems

AWS's automated DNS management system contains two modules （Planner and Enactor）. An extremely rare race condition caused the old plan to be cleared early, thus clearing the DNS records of the database service.
➩ Automation has no error detection and rollback mechanism and becomes a single point of failure.

2. The control plane's invisible dependence on DynamoDB

It is not only the customer database, but also widely used by AWS's internal control system: instance startup, load balancing, health checks, etc. all rely on it.
➩ The failure of an underlying service triggers a chain collapse of the entire link.

3. Regional concentration risk

Most of the affected services are deployed in the us-east-1 region.
➩ Multi-AZ ≠ Multi-Region, regional dependencies may still become "a single point in the cloud world".

 Behind the downtime: 5 major test blind spots that were torn apart.

AWS post-mortem review pointed to the "DNS automation system competition condition defect" - the planner and executor operated the same record at the same time, resulting in the generation of null values. However, the essence of this disaster was the systemic failure of the modern cloud testing system:

1. Disaster recovery testing "avoids the important and takes the easy"

Most companies only verify the "single machine down" scenario, but ignore the risk of control plane failure. AWS's DNS resolution failure falls into this type of "extreme blind spot", resulting in no plan available in the early stages of the failure and requiring manual emergency intervention.

2. Concurrency scenario tests are "in name only"

（RaceCondition） are not uncommon in automated systems, but operational conflicts under high concurrency are rarely simulated in tests. The loophole in the AWS dual-module mutual override operation is a typical case of "not covered by testing".

3. Dependency chain testing "broken link and oversight"

DynamoDB not only serves customers, but also supports core capabilities such as AWS internal instance startup and load balancing. However, this "invisible dependency" is not included in the test topology.

As Cisco expert Angelique Medina said: "Empty DNS records are like losing the phone book. The service obviously exists but cannot be found."

4. Self-healing mechanism test "on paper"

AWS automation system lacks fault rollback capability, and cannot automatically repair empty DNS records after they are generated ； What's more serious is that there is no traffic throttling mechanism when EC2 recovers, and the instant overload aggravates the paralysis - this exposes the lack of "monitoring-self-healing" closed-loop testing.

5. Regional risk testing "deception"

Enterprises are superstitious about the security of "Multi-AZ" but fail to conduct "regional-level disaster recovery drills". The centralized deployment in the US-East-1 region allows a single failure to escalate into a global crisis.

How to prevent “cloud-level accidents” from a testing perspective?

1. Establish "Cloud Dependency Risk Test"
• Introduce CloudDependencyCheck into the CI/CD pipeline.
• Automatically scan configuration items that depend on specific AWS regions and services.

2. Design the "control plane failure" scenario
• Simulate the situation of "database DNS cannot be resolved" or "internal API timeout".
• Verify whether the system can remain basically available when the control plane is abnormal.

3. Introduce “race condition injection” into testing
• Use tools or scripts to force multiple automated processes to execute simultaneously and observe state conflicts.
• For example, implement concurrent test cases in Playwright, Postman or PyTest.

4. Verify "automated rollback mechanism"
• Design a verifiable rollback process for key configuration or deployment steps.
• An exception is forced to be triggered during testing to verify whether it can be rolled back safely.

5. The testing team should jointly conduct drills with the architecture team and conduct "regional disaster recovery drills （RegionalDRDrill）" regularly.
• Simulate a downtime in the us-east-1 region and observe whether the system can automatically cut off traffic.

6. Daily risk prevention training for team members.
Initiate cloud dependency risk theme sharing meeting: discuss testing strategies learned from AWS outages. Establish a "cloud disaster recovery test template library" to help the test team quickly build DR test scenarios. Design "AI simulated cloud failure" training: let members use LLM to construct an abnormality propagation chain. Promote enterprises to establish multi-region verification mechanisms: testing not only verifies functions, but also verifies "cloud stability assumptions." Using AWS downtime as a teaching case: Cultivate testers' systems thinking and resilience thinking （Resilience Mindset）.

 Ways to break the situation: From “passive fire-fighting” to “AI active defense”

The AWS incident has forced the industry to reflect: testing in the cloud era needs to shift from “functional verification” to “resilience verification”, and AI is becoming the core breakthrough.

(1) 3-step method for upgrading the conventional testing system

• Step 1: Draw a "dependency topology map" based on Ookla's suggestions, use automated tools to scan cloud resource dependencies, mark "dual-role nodes" such as DynamoDB, and force them to be included in the core test scope.

• Step 2: Inject "extreme fault scenarios" and add use cases such as "DNS resolution failure" and "control plane stuck" in the CI/CD pipeline. For example, use PyTest to simulate multi-module concurrent operations to verify the effectiveness of the lock mechanism.

• Step 3: Practice "regional-level disaster recovery" and conduct cross-regional traffic cut tests every quarter. Refer to AWS's experience of "current limiting and starting EC2" afterwards to set the traffic threshold in the recovery phase in advance.

(2) Three major practical values ​​of AI testing.

AI is moving from "concept" to "implementation". Referring to the practice of Testin cloud testing, concrete answers are given:

1. "Automatic generation" of complex scenarios.
   Through LLM technology, its XAgent system can automatically generate chain failure use cases of "DNS failure → EC2 overload → load balancing abnormality" based on cloud resource topology, covering extreme scenarios that are difficult to imagine manually. After a joint-stock bank applied it, the test case generation efficiency increased by 60%.

2. "Advance warning" of abnormal risks
   Trains the AI ​​model to monitor "hidden signs" in the logs, such as the response delay deviation between the DNS planner and the executor, which can trigger an alarm 15 minutes before a failure occurs. Tencent Cloud Security Center has implemented similar capabilities to identify abnormal exposure risks in large model deployments through AI.

3. Root cause analysis "completed in minutes"
   After a fault occurs, AI can automatically correlate DNS logs, EC2 startup records, and load balancing indicators to generate a causal chain report. This allows the AWS-style "four-day review" to be shortened to less than 10 minutes.

Conclusion: The more automation develops, the more "anti-fragile" testing must be.

AWS's outage reminds us: in a highly automated world, the biggest risk often comes from "automation itself."

Cloud computing makes everything faster, but it also makes the same flaw more impactful. For test engineers, this is an "industry-level practical teaching material" - testing is not just about verifying functionality, but also verifying the behavior of the system on the verge of collapse.

What AI brings is not only an increase in efficiency, but also the ability to transform testing from "post-facto verification" to "pre-prevention defense." It may be difficult to completely avoid the next outage, but at least we can pass AI testing to prevent the "global Internet cold" from happening again.

AWS large-scale outage event review

1. Event background

On October 20, 2025 （Monday）, Amazon Web Services （AWS） experienced a large-scale outage event, affecting thousands of companies and millions of users around the world. The outage was mainly concentrated in AWS's US-East-1 region （located in Northern Virginia, USA）, which is one of AWS's largest and oldest data center regions. The outage affected many services including social media, banking, gaming, smart home devices, etc., highlighting the importance of AWS as the backbone of Internet infrastructure.

• Timeline:

• Early Monday morning （around 00:00 Pacific Time）: The AWS service status page reported increased error rates and delays for multiple services in the US-East-1 region.

• Around 2:00 a.m.: AWS identifies the potential root cause and begins applying mitigation measures.

• 3:35 a.m.: AWS announced that the DNS issue has been basically resolved and most services are back to normal.

• After 8:00 a.m.: As the workday begins on the U.S. West Coast, outage reports surge again, indicating that the problem has not been fully resolved.

• 3:53 p.m.: AWS announces the issue has been resolved.

• October 29 （Wednesday）: Some users reported similar outages, but AWS denied the existence of new outages and said the service was operating normally.

• Affected scope:

• According to data from Downdetector, more than 2,000 companies and 9.8 million users reported being affected, including 2.7 million in the United States and 1.1 million in the United Kingdom. Other reports came from Australia, Japan, the Netherlands, Germany, and France.

• Affected services include:

  • Social media: Snapchat, Reddit, Signal

  • Games: Roblox, Fortnite, PlayStation Network

  • Finance: Lloyds, Halifax, Venmo

  • Smart home: Ring doorbell, EightSleep smart bed

  • Others: Netflix, Starbucks, United Airlines, Canva, HMRC, Duolingo, Amazon's own services, etc.

2. Implications

The AWS outage is one of the most widespread cloud service failures in recent years, exposing the following key issues:

• Technical vulnerability: Hidden flaws in automated systems may cause cascading failures, and interruptions in basic services such as DNS will have widespread impacts.

• Centralization risk: The global Internet’s reliance on a few cloud providers such as AWS increases the risk of single points of failure.

• Social dependence: From banks to smart beds, modern life is highly dependent on cloud services, and downtime has a significant impact on daily life and the economy.

• Necessity for improvement: AWS needs to rebuild customer trust through technical optimization and transparent communication, and the industry needs to explore decentralization solutions to improve overall resilience.

AWS's rapid response and detailed analysis demonstrate its ability to respond to the crisis, but also remind enterprises and users that relying entirely on a single cloud service may bring unpredictable risks. In the future, the industry may require more diverse cloud infrastructure and stronger regional solutions to ensure the stability and security of the Internet.

Source: TesterHome Community