When Making Things Better Only Makes Them Worse

[Ninshub]

When Making Things Better Only Makes Them Worse
Our very attempts to stave off disaster make unpredictable outcomes more likely.

Apr 26, 2019
Erik Larson
Author of the forthcoming book The Myth of Artificial Intelligence
The Atlantic

Accidents are part of life. So are catastrophes. Two of Boeing’s new 737 Max 8 jetliners, arguably the most modern of modern aircraft, crashed in the space of less than five months. A cathedral whose construction started in the 12th century burned before our eyes, despite explicit fire-safety procedures and the presence of an on-site firefighter and a security agent. If Notre-Dame stood for so many centuries, why did safeguards unavailable to prior generations fail? How did modernizing the venerable Boeing 737 result in two horrific crashes, even as, on average, air travel is safer than ever before?

These are questions for investigators and committees. They are also fodder for accident theorists. Take Charles Perrow, a sociologist who published an account of accidents occurring in human-machine systems in 1984. Now something of a cult classic, Normal Accidents made a case for the obvious: Accidents happen. What he meant is that they must happen. Worse, according to Perrow, a humbling cautionary tale lurks in complicated systems: Our very attempts to stave off disaster by introducing safety systems ultimately increase the overall complexity of the systems, ensuring that some unpredictable outcome will rear its ugly head no matter what. Complicated human-machine systems might surprise us with outcomes more favorable than we have any reason to expect. They also might shock us with catastrophe.

When Making Things Better Only Makes Them Worse
Our very attempts to stave off disaster make unpredictable outcomes more likely.

Apr 26, 2019
Erik Larson
Author of the forthcoming book The Myth of Artificial Intelligence
The Atlantic

There's an old saying you may have heard. "The road to Hell is paved with good intentions"