For about 30 minutes yesterday, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilisation on their network. This CPU spike was caused by a bad software deploy that had to be rolled back. A large number of websites use Cloudfare.

Cloudfare went on record to state that this was not an attack and that they and were incredibly sorry that this incident occurred.

CLoudfare stated on their blog –

“Update at 2009 UTC:

Starting at 1342 UTC today we experienced a global outage across our network that resulted in visitors to Cloudflare-proxied domains being shown 502 errors (“Bad Gateway”). The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Application Firewall (WAF) during a routine deployment of new Cloudflare WAF Managed rules.

The intent of these new rules was to improve the blocking of inline JavaScript that is used in attacks. These rules were being deployed in a simulated mode where issues are identified and logged by the new rule but no customer traffic is actually blocked so that we can measure false positive rates and ensure that the new rules do not cause problems when they are deployed into full production.

Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. This 100% CPU spike caused the 502 errors that our customers saw. At its worst traffic dropped by 82%.”

Just shows how fragile the internet can be!