Horror #2: Granting unearned ‘God mode’
Imagine you’re facing the cockpit of a commercial aircraft. These systems are built on a design ethos that assists users in their decision making, and have set up parameters to check human choices given environmental factors. Now, consider for a moment, while activating another control, you accidentally activate the thrust reverser, a device that redirects engine power to slow down on the ground. Sensors confirm when they can be deployed on the ground, but if activated during flight, the system does more than display a warning; it prevents you from taking the action. The system design for aeronautics employs situational awareness and anticipates human error to protect everyone on board.
In 2021 a video-on-demand streaming service sent a blank test email, subject line: “Integration Test Email #1” to 6 million subscribers, bypassing the internal test list. The mass mailing was sent by an intern who likely received insufficient system feedback when confirming, with no perceived consequences before clicking “Send.” Ask any Site Reliability Engineer (SRE) about a failsafe, and they’d wax lyrical about the concept of a “Game day.” A simulation environment that enables practicing for disasters and the ability to run various tests in a controlled learning environment before an incident.
The company’s help team took to a social media platform and apologized for the incident, but called out its intern.
What they didn’t expect was thousands of industry professionals sharing their stories of mistakes made with the tools they use to carry out their jobs. Most related to posts confirming they too experienced poorly designed hands-off systems, treating the “Send” button as low, if not no friction. In this case, the software assumed the user understood the volume of the distribution list chosen and confidently didn’t need draft mode or a sandbox, following the send command.
Being able to select a full list of subscribers from a test account is nerve wracking; then to hit send with no fail safe, is a design flaw with rightfully placed anxiety on the user. Here, the system assumed a level of confidence in the intern, who likely hasn’t garnered this level of trust, yet was afforded an unusual amount typically granted a senior or seasoned manager driving a marketing platform.
Without parameters like role-based access control (RBAC), there’s a projected fallacy of perfect knowledge onto humans by software, giving free rein and superpowers over decisions that can cause a ripple effect of issues. Imagine a system design deciding that a newly onboarded user would never confuse a test for six million subscribers.