Amazon’s massive failure is a reminder that we trust too much in big tech companies

On Monday, October 20, millions of Internet users got a heartbreaking answer to a question they barely knew existed. The question is, what do Snapchat, Roblox, Fortnite, Signal, United Airlines, Delta Air Lines, and countless other web-based sites and services have in common?

The answer is that they were all brought down by a cascading failure at a data center in northern Virginia owned and operated by Amazon Web Services, a division of the giant e-commerce company.

AWS is one of the top three cloud platforms. This means that AWS maintains client data on its own servers and manages the transfer and transmission of data within the client company and between the client and end users.

When AWS’ Northern Virginia data hub went down a few minutes before midnight PDT on Sunday, 141 AWS services and customers that relied on the hub lost power, causing a cascading outage that affected users around the world. Users of Amazon’s own Ring home security devices, including video-enabled doorbells, were affected.

Amazon didn’t declare the issue fixed until Monday at 3:53 p.m. PDT, but some customers were still reporting the issue as of Tuesday.

The damage done to AWS clients and their millions of users is immeasurable. As my colleague Queenie Wong reported, web users were unable to access services or accounts.

Some banks and customers of web brokerage Robinhood were unable to complete their trades. Delta and United Airlines passengers were unable to track their reservations, check in online, or obtain seat assignments. Airline employees have had to rely on manual alternatives, much like in prehistoric (i.e., pre-internet) times.

Owners of Eight Sleep mattress covers, which cost thousands of dollars and require a $300 or $400 annual fee, use a web app to adjust the temperature and slope, and report getting stuck in uncomfortable positions and sweltering in uncontrollable heat. The company’s chief executive apologized online and said Eight Sleep would roll out a feature that would allow owners to connect to their beds via Bluetooth if their internet connection fails.

The failure is sure to raise questions about whether Amazon and its Big Tech peers are monitoring their systems with the rigor appropriate for a critical service that operates on a global scale. As lawyers say, “res ipsa loquitur,” or “things speak for themselves.” The answer is no.

Once upon a time, when “plain old telephone service” (POTS) was under the complete control of one company, AT&T, the company promised “99.999%” reliability, meaning it would be up 99.999% of the time, or about 5.26 minutes or less of downtime per year. That standard was effectively trashed this week as AWS systems were down for at least 15 hours, or 900 minutes.

The Five-Nine standard reflects the belief that telephone service is so important that it does not need to be connected virtually all the time. Today’s high-tech service providers often seem to have the attitude that there should be enough for everyone.

As I pointed out last year, some of today’s wealthiest companies have billions of dollars in profits, but they don’t spend enough money to protect their customers’ personal data from hackers. For example, AT&T, which posted $16.7 billion in pre-tax profits last year, was so lax about protecting its customers’ personal information that the data of nearly all of its customers, or 110 million users, fell into the hands of “financially motivated” hackers.

Amazon has said convincingly so far that its outages were not caused by hackers or other hostile actors. It came entirely from inside the house, so to speak.

To keep the technical gibberish to a minimum, let’s just say that something went wrong with the Domain Name System. The Domain Name System allows the system to translate the web addresses you type into your browser to communicate with the website itself. The technical disruption rippled throughout the AWS structure, causing pain for websites and users. Amazon says it will eventually provide a “post-mortem summary” that identifies the cause of the failure.

Amazon clearly bears most of the blame for this debacle. Some Amazon watchers have speculated that the glitch is related to mass layoffs the company implemented in its cloud computing division over the summer, with jobs said to be replaced by artificial intelligence. The company confirmed the layoffs, but did not say how many people had been cut. According to Reuters, the number is in the hundreds.

Amazon has denied speculation that the outages are related to layoffs. A spokesperson shared an interview with AWS CEO Matt Garman in which he disparaged the idea of ​​replacing entry-level staff with AI bots, calling it “one of the stupidest things I’ve ever heard.” However, it is unclear who in the cloud division was fired.

Some technology experts have been warning for years about website operators not having a Plan B for exactly the kind of failure that occurred this week. AWS isn’t the only cloud platform in existence. Microsoft and Google are also in the top three.

AWS users are also not obligated to rely on the company’s Northern Virginia data hub. AWS has data hubs across the country and was advising users to switch to other hubs, but customers who hadn’t implemented a workaround before this issue were out of luck, as the hub in Virginia went out of service.

Lydia Leung of technology consulting firm Gartner advised this week that IT departments should “design for failure, because it will happen.” “Modern cloud-native apps need to be able to distribute workloads across multiple availability zones and quickly failover to another region if needed,” Leong wrote. This means it should be configured to automatically move data away from problematic locations. “It’s not about eliminating risk, it’s about reducing blast radius and recovery time.”

As Jorg Dekker of Internet backbone company Arelion pointed out, this problem may be a product of Internet history. The Internet was designed to be a neutral system where we trust that all data flowing through connected networks is authentic.

“This assumes all updates are valid and means networks can announce whatever they like and cannot check available resources,” he pointed out.

The net’s first designers addressed its imperfections by ensuring that the network kept data away from failures and other problems. “The Internet avoids damage” is a mantra, but that doesn’t always work, especially when the damage is in core functionality. There are also times when you shouldn’t trust trusted updates.

Last year’s CrowdStrike failure was similar. A poorly designed update to a program rolled out by a cybersecurity company and automatically installed on users’ machines caused millions of computers running Microsoft programs to instantly crash and become inoperable until they could be manually fixed.

By design, the errant CrowdStrike application was embedded deep within the Microsoft operating system, encountering the same glitch every time the machine was restarted, causing it to die again in an endless loop of doom. As I wrote at the time, “Thousands of flights were canceled, doctors were unable to perform surgeries, banking transactions were frozen, and emergency services were silenced.”

There are certainly advantages to having the critical backbone of the Internet under the control of three of the world’s wealthiest technology companies. After all, they have the financial resources to maintain quality and reliability. The disadvantage is that the system works perfectly until the moment it stops working. That is when global dependence on a few large operators turns into a global meltdown.

An unavoidable feature of modern life is that those living in the modern world increasingly have nowhere to escape from web service failures. Not only do voice calls, data calls, email, and video entertainment happen over the web, but some appliances require an Internet connection to operate.

The noise canceling mode on Bose headphones cannot be adjusted without using the phone app. Same goes for my super fancy automatic drip coffee maker and self-heating coffee mug. When I tried to add a line to a family member’s T-Mobile account the other day, T-Mobile required me to load the T-mobile app on my (non-T-Mobile) iPhone to complete the deal. At the time I was sitting with a T-mobile representative at the T-mobile store.

However, reflecting the Internet of Things ideals advocated by Web promoters and appliance manufacturers, more and more appliances are being sold with unnecessary Internet functionality. As a rule of thumb, if your refrigerator or countertop doesn’t need an internet connection to work, don’t connect it. That way, your brain won’t be turned into a broken brick by some human error somewhere in Northern Virginia.

Web connectivity has brought us benefits that were unimaginable even at the turn of the century. But as with all things, with blessings comes burdens. With a few lines of rebel code, we can transport our 21st century lives back to the 1950s and 60s.

Back then, when household appliances were mechanical or electrical rather than electronic, faults were easy to diagnose and fix by replacing vacuum tubes or tightening screws. Today, if your TV goes dark and you can’t watch HBO Max, you have no idea where the problem is inside your TV, your cable box, or HBO Max.

You just wait for someone to fix it, hoping that the problem isn’t just in your home or neighborhood, but spread out far enough that the service provider notices and drives off. We all live in balance. Today’s technology is great when it works well. When we don’t, we act on our own. There’s a lesson somewhere.

The Los Angeles Times in 2025. Distributed by Tribune Content Agency, LLC.