by Charles Miller

What happened? When it comes to computering this is a question often asked, and only rarely answered to everyone’s satisfaction. In the case of the worldwide internet outage of July 19 it seems that there is an answer, and one that can be easily understood. Observing cause and effect is easy. Determining culpability is not so easy.

CrowdStrike is a security company that serves large corporate clients by attempting to protect their critical computer networks. CrowdStrike’s software, functioning similarly to the antivirus software on your computer, does what is called Endpoint Detection and Response (EDR) constantly monitoring for any suspicious or malicious activity. This powerful software is designed to help security professionals protect important networks. If the name CrowdStrike sounds vaguely familiar, that might be because the company was at the center of the shadowy political intrigue surrounding the hacking of the Democratic National Committee (DNC) email servers during the U.S. presidential election cycle of 2016. That sordid affair may now be memory-holed, and CrowdStrike henceforth only remembered as being responsible for the biggest and most spectacular outage in the history of the internet.

If the news reports are to be believed, what happened was actually very simple, and therefore should have been avoidable. CrowdStrike pushed out an update to their EDR software that caused every computer running it into an endless loop of crashing and rebooting, crashing and rebooting... CrowdStrike’s faulty update apparently was pushed out to over eight million machines. Since CrowdStrike has quite a few important clients in critical positions, that included airlines, banks, hospitals, and governments. Eight million Windows machines is a small fraction of the total number of Windows machines, but when they all crashed at once it caused an enormous and widespread outage given how many important companies were affected.

Fixing all of those crashed computers was not as simple as pushing out another update to reverse the damage done by the faulty update. An unknown number of Information Technology techs gave up their weekend to get the internet back up and running. Fixing each crashed Windows system required manually rebooting then deleting one file. This meant that each one of the millions of crashed machines had to be hands-on manually serviced by a technician.

For a software update of any kind to be the cause of such a widespread internet outage as was caused on July 19 is unprecedented. This has led many to suggest that CrowdStrike might be guilty of skipping common-sense software designing procedures, such as actually testing the update to see what it did before releasing it. Clearly there should have been a phased rollout of the update to a small number of users before pushing it out to millions of users. This would have prevented or mitigated the catastrophe.

And a catastrophe this was for the airlines that were forced to cancel thousands of flights, hospitals that could not access their records, and businesses that could not do business. It is reported that the costs from the global outage could top $1 billion USD, which brings up the question of who should pay. While CrowdStrike has apologized, it has not mentioned if it will provide compensation to affected customers.

It is going to be interesting to watch what happens over the next few months or years with regard to damages and legal liability. CrowdStrike might be counting on being protected by the software industry’s standard, and unconscionable, contract of adhesion that says no matter how much a software product damages the customer; the software maker has zero liability. Whether or not such licenses are legally enforceable remains a matter of legal debate, and also a topic better left to be addressed in another column.

**************

Charles Miller is a freelance computer consultant with decades of IT experience and a Texan with a lifetime love for Mexico. The opinions expressed are his own. He may be contacted at 415-101-8528 or email FAQ8 (at) SMAguru.com.

Discover Lokkal:
Watch the two-minute video below.
Then, just below that, scroll down SMA's Community Wall.
Mission

Visit SMA's Social Network

Contact / Contactar