📖 yourdailystory
Systems & Scale Thinking
Published on Friday, 20 November 2026 · ⏱ 7 min read

Netflix and the Chaos Monkey

Today's skill is thinking about failure before it finds you. The one big idea is this: a system you have never seen fail is a system you do not understand.

Let's walk through it in four small steps.

The Story

Start with the first step: break the skill down to its smallest piece.

Systems thinking sounds huge. It isn't. The smallest piece is one question. "If this one part died right now, what would happen?"

That's it. You pick one part. You ask what happens if it dies. You write down the honest answer.

Now the second step: remove what makes it hard to start.

What makes it hard is that you have to look at something you built and imagine it broken. That feels like doubting your own work. So make it small and make it safe. Don't redesign anything. Don't fix anything. Just ask the question and write the answer down.

Here's the third step, and it's the one that keeps you honest.

Watch for the answer "it should be fine." That phrase is the warning light. "Should" means you are guessing. If you can't say exactly what happens, in plain words, you haven't understood the part yet. You've only hoped about it.

And here's the thing nobody tells you about this skill. The hard part is not intellectual. It's emotional. The first time you ask the question honestly, you will feel a bit stupid. You'll realize you don't know how your own system behaves. That feeling is the price of learning. Everyone who gets good at this pays it.

Now the fourth step. Practice, with real evidence. Let me tell you about a company that paid that price in public.

In August 2008, Netflix had a bad few days. A major database corruption hit its systems. For about three days, the company could not ship DVDs to its customers. Back then, DVDs by mail were the business. Streaming was still the small side project.

Picture the engineers in those days. They were not careless. They were good people running a normal setup. A big database in a data center. A handful of machines that everything leaned on. The kind of design nearly every company had.

And it had a single point of failure. Everyone had known that, in the way you know you should floss. It was on the list. It was not on fire. Then it was on fire.

The lesson Netflix took was not "be more careful." Careful people had built that system. The lesson was bigger. Any design where one machine can take down everything will eventually be tested by reality. You just don't get to pick the day.

So the company made a large decision. It would move to the cloud, to Amazon Web Services. The idea was to stop depending on a few big machines. Run on many small ones, spread out, any of which could vanish.

But that created a new problem. Machines in the cloud do vanish. They get shut down without warning. That is just how the cloud works. So Netflix engineers faced an uncomfortable fact. It's one thing to say your software can survive a lost machine. It's another thing to know it.

Here's where a Netflix engineer named Greg Orzell comes in. He and his colleagues built a small tool around 2010. They called it Chaos Monkey. Its job was simple and a little alarming. During working hours, it would pick a running server in the live production system at random, and shut it down.

Think about that. On purpose. In production. While customers were watching movies.

Why would anyone do that? Because of the question from earlier. "What happens if this part dies?" Most teams answer it with a meeting and a diagram. Netflix answered it with a real machine dying, on a normal Tuesday, with engineers sitting at their desks and ready to help.

That changes how you build. If you know a random server will be killed this week, you don't write code that needs it alive. You build it so the work moves somewhere else. The failure stops being a rare disaster and becomes a routine chore. And chores get automated.

There was real cost and real doubt here. Early on, the monkey exposed weak spots, and those weak spots meant real pain. Engineers had to fix things they'd rather not have known about. Nobody enjoys being shown a flaw by a machine whose whole job is to find flaws. That's the emotional barrier again, and it showed up in a company full of very smart people.

Then came April 2011. A large part of Amazon's main US data center region went down. Many well-known websites went dark or crawled for hours. Netflix, which relied on that same cloud, mostly stayed up. It wasn't magic. It was practice. They had been losing servers on purpose for months, so losing many at once was a bigger version of something they'd already survived.

Netflix later grew the idea into a whole family of tools they called the Simian Army, each one breaking a different thing. A server. A whole zone. A slow connection. The pattern stayed the same. Don't wait for the failure. Invite it in, when you are ready for it.

Notice what they did not do. They did not try to build a system that never fails. That is a fantasy. They built a team and a system that expected failure and recovered quickly.

The Skill

The principle is this: you only understand a system's behavior under failure by testing it, so ask "what happens if this dies?" about one named part, and get a real answer, not a hopeful one.

Let's check that against the four steps.

The smallest piece is one part and one question. Not "is my system reliable?" That's too big to answer. "What happens if the payment service is unreachable for ten minutes?" That you can answer.

The friction remover is making it safe. You don't have to break anything in production like Netflix did. Start on paper, or in a test setup. Write down what you think will happen first. Then check.

The self-correction signal is the word "should." Every time your answer includes "it should just retry" or "it should fall back," you've found something you haven't verified. Mark it. That mark is your to-do list.

The practice is repetition on small things. Netflix didn't begin with a giant experiment. They began with one random server.

And the checkable result is simple. At the end, you have a written line: "If X dies, then Y happens, and I confirmed it by Z." Either that line exists or it doesn't.

One more point, about the AI era. You can use an AI model as a sparring partner here, and it's a good one. But the test is who does the thinking. Don't ask it to design your system's failure plan for you. That's hiring someone to run your race. Instead, write your own answer first. Then ask the AI to quiz you, to attack your answer, and to find the part you called "fine" without checking. You stay the expert. It plays the monkey.

Do This Today

This evening, before you close your laptop, pick one service or part that you own or work on at your job. Write one sentence on paper or in a note: "If [that part] stopped working for ten minutes, here is what happens to the user." Don't look anything up first. Then ask an AI model to quiz you with three hard questions about that sentence. Mark every place your answer used the word "should." You're done when you have one written sentence and a short list of "should" marks, even if the list has only one item.

Sources


This is a dramatized editorial narrative created for personal inspiration, drawn from publicly available sources listed above. It is not affiliated with or endorsed by the person, company, or their estate.

Read on yourdailystory.com →

One true story a day to get a little better. Start today's →

Share this story: WhatsApp · X · LinkedIn