“This was entirely our problem, this was a mistake which we made.” It was not an external attack, and it was not related to the BGP problems of the previous week.
On July 2, 2019, network services provider Cloudflare suffered a significant outage. It brought down all of its services, and with them a large part of the Internet, including Discord, Marketo, Down Detector and more.
“We built Cloudflare with a mission of helping build a better Internet and, this morning, we didn’t live up to that,” Cloudflare CEO Matthew Prince told DCD. “I take personal responsibility for that. And so I think that it’s disappointing, and it’s painful.”
In an interview with DCD as he rushed to the airport, Prince explained what went wrong, dispelled rumors about nation-state attacks, and discussed the unrelated BGP outage of the week before.
In this article
Here’s What Happened
“There has been speculation online that this was caused by some sort of external attack,” Prince said. “We don’t see any evidence that this was related to an external attack, although that was also our own team’s initial speculation.”
Some blogs and social media chatter blamed the outage on a DDoS attack by the Chinese government, trying to bring Hong Kong protesters offline. Prince denied the claims: “We needed to make sure that nobody believes that was the case, because it wasn’t the case. And while it would be incredibly convenient if that had been the case, because that would be an understandable issue, this was not that.”
Instead, the problem, which affected users worldwide for up to 30 minutes, was caused internally by Cloudflare itself. In fact, Cloudflare’s own protection systems were to blame.
“When we see an attack, our systems are designed to be able to scale up across all of those services to be able to mitigate it,” Prince said. “Unfortunately, this morning it appears there was a bug in our firewall service that caused it to grow and scale over time, even though there was no attack that was targeting the service in any way.”
Prince added: “This was a unique problem, it was something that we had never seen before. I understand the system decision that we made that allowed it to become as broad of a problem as it did, because we again wanted to be able to design our systems in such a way that when they saw large attacks, they could scale up and use all the resources necessary in order to mitigate them.
“Unfortunately… it appears that we did not have the appropriate controls in place to make sure that a bug like this didn’t cause a wider spread issue. But we will going forward.”
Transparency and a “Blameless Culture”
As the company learns more about what exactly happened, Prince promises full transparency. He said this was essential for keeping customers after such incidents.
“We’re fortunate that we haven’t had a number of significant issues,” he said. “But I remember the first really significant issue we had was a hack back in 2012 that was incredibly painful. It affected one of our customers, and it impacted me personally, because the hacker had actually hacked into my personal email in order to get in.
“And I was embarrassed. Frankly, I didn’t want to share the details of everything that happened. Our team said, ‘No, that’s not our culture, and that’s not what we stand for, and we really believe in this idea of being radically transparent with whatever happened.’ I was afraid that we would lose customers. It turned out instead… that transparency actually helped people build trust.”
Equally important for keeping customers is honoring service level agreements (SLAs), something which Prince said “our team is already working to resolve. We’ve always taken a very broad view of whatever the SLAs are and honored them whenever there’s something that is impacting the ability for customers to reach our network.”
Prince also said that the company has a “blameless culture,” adding: “Unless there was active malfeasance or something that was going wrong, I can’t imagine this is something that someone is going to lose their job over.”
What makes this incident particularly painful for Prince, the company and its customers is that Cloudflare was also unavailable just one week earlier.
Bad Timing
“Today’s issue was entirely our problem, this was a mistake which we made,” he said. The outage of the previous week, however, was caused by an external failure.
“22,000 networks had their network routes hijacked, [something] which impacted some fraction of Cloudflare’s network,” Prince said. “That is much more of an Internet-wide issue, and something that… [the] entire Internet community needs to work in order to resolve.
“The two things were completely and totally unrelated,” he said, admitting that even within Cloudflare there was speculation about a connection between the two outages: “You see something happening and you think, ‘Oh, it must be the same thing as what happened before.’ These were completely unrelated processes, completely unrelated teams.”
The previous week’s issue was a problem with the Internet’s Border Gateway Protocol (BGP), which manages how packets are routed across the Internet. Telecoms company Verizon accepted and spread a huge set of accidentally leaked BGP routes, which sent Internet traffic to the wrong places.
Individual service providers can set up specific routes to optimize their traffic. In this case, one of these “more specific” routes leaked and was used far more widely than it should have been, which created an unwanted bottleneck. The route went from another ISP through a metals manufacturer’s corporate network. Verizon accepted and broadcast it by mistake, which effectively invited an overload.
Alex Henthorne-Iwane, of network monitoring company ThousandEyes, explained in a blog post for DCD: “The end result was that a huge set of user traffic headed toward Cloudflare and other providers got routed through a relatively small corporate network, with predictable results. Massive congestion from the traffic redirection led to high levels of packet loss and service disruption. Users simply weren’t able to reach the Cloudflare Edge servers and the apps and services that depended on them.”
This is a problem much wider than Cloudflare: it is an issue with the fundamental design of the Internet itself.
“The good news about today’s issue is that it is entirely under our control, and therefore something that I know that we can fix, that we can put safeguards in place, and will not happen again,” Prince said. “We make mistakes all the time, but we make different mistakes all the time, which I think is a sign of a healthy organization.
“The frustrating thing about last week’s outage is that it is not entirely under our control, and so it requires us to work with other large networks like Verizon in order to clean up their systems. And that’s something that, while we’ll be able to fix [today’s] issue very quickly, the issue about BGP route leaks is going to be something that takes us a lot longer to resolve, and it’s going to take much more than just Cloudflare working on it to get it fixed.”
What Cloudflare Found Later
Ten days later, on July 12, 2019, Cloudflare published a detailed post-mortem. The outage lasted 27 minutes. A new rule in Cloudflare’s Web Application Firewall (WAF) Managed Rules contained a regular expression that backtracked enormously. It pushed the CPUs that handle HTTP and HTTPS traffic to 100% across the whole network, so visitors to Cloudflare sites saw 502 errors until the change was rolled back.
You can read Cloudflare’s own reports here: Details of the Cloudflare outage on July 2, 2019, and for the route leak of June 24, 2019, How Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Today.
Another big outage we covered: YouTube Went Down Today, 14 December 2020.
Originally reported by Sebastian Moss for DatacenterDynamics (DCD) on July 2, 2019.
