Chaos Engineering Basics: Principles, Examples, First Steps

Chaos engineering is the discipline of experimenting on a distributed system to build confidence in its capacity to withstand turbulent conditions in production. Instead of waiting for an outage to expose a weakness, a team forms a hypothesis about how the system should behave, injects a controlled fault, and compares the observed behavior with the prediction. The sections below set out the definition, the governing principles, the safeguards that keep experiments contained, representative examples, the tool categories involved, and a small-scale starting procedure.

TL;DR

• Chaos engineering is empirical and hypothesis-driven: it generates new knowledge about unknown failure modes, whereas conventional testing verifies behavior that is already expected.
• A controlled experiment requires a measurable steady state, a specific hypothesis, a limited scope (blast radius), stop conditions, and a rollback path.
• Typical first experiments involve resource exhaustion, slow or failing network dependencies, and API errors, each revealing whether timeouts, retries, autoscaling, and monitoring behave as designed.
• A sensible start is one component, one hypothesis, a small scope, and a non-production environment, with scope widened only as confidence accumulates.
• Organizational readiness (ownership, communication, and rehearsed response) matters as much as the fault injection tooling.

Table of Contents

• What chaos engineering is
• How it differs from conventional testing
• Why organizations adopt it
• Core principles
• Safeguards that keep an experiment controlled
• Common experiments and what they reveal
• Tooling categories
• A small-scale first experiment
• Connecting experiments to recovery planning
• Working with PROJECT-JTH
• FAQ
• Sources

What chaos engineering is

The canonical definition appears in the Principles of Chaos Engineering, which describes the practice as experimenting on a system in order to build confidence in its ability to withstand turbulent conditions. The project’s rationale is that the inherent unpredictability of distributed systems should be managed proactively, so that a team can preserve development velocity and confidence as the system scales.

The word “chaos” is therefore somewhat misleading. The practice is not random destruction. It is a structured form of experimentation in which each fault is chosen deliberately, bounded in scope, and observed against a prediction. The objective is to surface weaknesses, such as retry storms, before they produce outages or other disruptions in production, as AWS Prescriptive Guidance on chaos engineering explains.

A retry storm illustrates the kind of defect at issue. When a downstream dependency slows, upstream clients retry, the additional load deepens the slowdown, and a partial degradation becomes a full outage. Such behavior rarely appears in a unit test, because it emerges only from the interaction of many components under load.

How it differs from conventional testing

Conventional testing is largely deterministic. A test asserts that, given a defined input, a component produces a defined output, and it therefore verifies behavior that the engineers already expect. According to the AWS guidance, chaos engineering is instead empirical and hypothesis-based, and aims to generate new knowledge about unknown failure modes. O’Reilly’s treatment of the subject makes the same distinction: testing usually confirms a property that is already known, whereas chaos engineering produces new and sometimes unexpected knowledge about how a system survives in the real world.

The two approaches are complementary. Functional and integration tests establish that the system does what it should under normal conditions; chaos experiments probe what it does when conditions are abnormal.

Why organizations adopt it

The motivations reported in the literature fall into three groups.

Availability and reputation. IBM’s overview links customer satisfaction directly to availability and notes that chaos engineering can mitigate the reputational damage and revenue loss that accompany downtime. The same source states that organizations use it to reduce downtime, which it says can decrease by as much as 20 percent, to improve data security through patching, and to identify resource bottlenecks that affect scalability.

Confidence at scale. As services multiply, no individual engineer holds a complete model of the dependency graph. Experiments substitute observation for assumption.

Human and procedural readiness. The Google Cloud guidance on getting started observes that chaos engineering considers a system’s technology, people, and processes, and that it helps teams rehearse their responses to failures. An experiment therefore tests not only whether a circuit breaker opens but also whether the on-call engineer notices, escalates, and recovers the service in the expected time.

Core principles

Several sources converge on a compact set of principles. The Google Cloud guidance lists the establishment of a steady-state hypothesis, the simulation of real-world conditions, the minimization of blast radius, the automation of experiments, and their performance in production.

Define a steady state

A steady state is a measurable indication that the system is behaving normally, typically expressed through a few customer-facing metrics such as successful request rate, latency percentiles, or order completions. Without a baseline, an experiment cannot distinguish an effect from ordinary variation. The Gremlin adoption guide identifies the definition of a steady state and a hypothesis, together with baseline monitoring and prepared stop conditions, as key planning components that precede any fault injection.

Vary real-world conditions

Faults should resemble those that occur in operation: a server failing, a dependency slowing, a network partition, an availability zone becoming impaired. Exotic faults with no realistic analogue yield little actionable knowledge.

Run in production, progressively

The Principles project argues that production traffic is frequently the most realistic test environment, because systems behave differently across environments and real traffic exercises the actual request path. It balances this against an obligation to minimize and contain customer impact. The O’Reilly practitioners describe the resolution as progressive risk: experiments escalate from smaller to broader impact rather than beginning with all production traffic.

Automate

The Google Cloud guidance emphasizes that automation, including integration into CI/CD pipelines, converts chaos engineering from an occasional exercise into a scalable, repeatable, and continuous operational practice.

Safeguards that keep an experiment controlled

The most common objection to the discipline is the risk of causing the very outage it seeks to prevent. The literature answers this with explicit engineering controls rather than reassurance. The AWS Architecture Blog states that a controlled experiment should have a clearly defined impact scope, rollback mechanisms, and tight real-time monitoring integration, and that it should be fail-safe through stop conditions, continuous monitoring, rollback or post-actions, stakeholder communication, and restoration of the system to a known-good state.

In practice these controls translate into a checklist:

1. Scope. Limit the experiment to a small fraction of instances, users, or requests.
2. Tiering. The Google Cloud guidance recommends categorizing services into tiers according to potential impact, so that the initial blast radius is safe and critical customers are not adversely affected.
3. Stop conditions. Define thresholds, for example an alarm on error rate, at which the experiment halts automatically.
4. Abort mechanism. The O’Reilly practitioners recommend a human-accessible “big red button”, ideally supplemented by automatic abort logic.
5. Rollback. Prepare the path back to a known-good state before injecting anything.
6. Communication. Notify stakeholders so that an induced fault is not mistaken for a genuine incident.

The AWS guidance adds that safe experiment design starts in lower-level environments and advances as confidence builds. A properly isolated staging environment for safe testing and deployment is one way to validate controls, such as stop conditions and rollbacks, before any change reaches live customers.

Common experiments and what they reveal

Examples of chaos experiments tend to cluster around a few failure classes. The AWS guidance lists resource exhaustion (CPU, memory, disk), API errors, network latency or connectivity problems, and availability zone or region-wide impairments. The Prime Video account of its resilience work adds that such experiments can reveal whether monitoring, autoscaling, restarts, timeouts, retries, and circuit breakers operate as intended.

Each row follows the same structure: a specific prediction, a bounded fault, and a set of observations that either support or contradict the prediction. A contradicted hypothesis is the valuable outcome, since it exposes a defect while its cost is still small.

Tooling categories

The tooling landscape can be described by function rather than by product. The AWS Architecture Blog identifies fault-injection and experiment-orchestration tools, monitoring and alerting systems, synthetic monitors, and rollback mechanisms. In the AWS ecosystem specifically, the same article describes AWS Fault Injection Service, CloudWatch alarms serving as stop conditions, and Systems Manager documents for custom actions.

Monitoring deserves particular emphasis. The AWS Cloud Operations Blog argues that it is not optional: without measurements at the host, network, application, and human-process levels, a team cannot reliably detect drift, evaluate the hypothesis, or quantify detection, escalation, degradation, self-healing, and recovery times. An organization with immature observability should therefore strengthen it before expanding experimentation. Related hygiene, such as detecting configuration drift and maintaining a system hardening baseline, makes experimental results easier to interpret, because the environment under test is known rather than assumed.

A small-scale first experiment

The O’Reilly chapter on designing experiments outlines a sequence: choose a hypothesis, select the scope, identify metrics, notify the organization, run the experiment, analyze the results, increase the scope, and automate. For a team beginning from nothing, the same source suggests a modest entry point: a simple hypothesis about a component believed to be resilient, a defined steady state with a few customer-impact metrics, and a small scope, preferably outside production at first.

Step by step

1. Select one component. Choose a service with a known owner and a well-understood dependency, for instance a service that calls a single downstream API.
2. State the hypothesis. An example: when the downstream API responds 500 milliseconds slower, the service keeps its error rate within the agreed threshold.
3. Establish the baseline. Record the steady-state metrics for a representative period before the fault.
4. Prepare controls. Define the stop condition, rollback, and a named owner; confirm the monitoring that will be used.
5. Notify. Inform the people who might otherwise treat the fault as an incident.
6. Inject and observe. Apply the fault to the small scope and record observations and any divergence from the prediction.
7. Analyze. The Gremlin guidance states that when results reject the hypothesis, the team should use that knowledge to address the uncovered failure point and to refine later experiment designs.
8. Expand gradually. Widen scope, move toward production, and eventually automate recurring experiments.

The process is deliberately slow at the beginning. Early experiments are as much a test of the team’s monitoring, communication, and ownership as of the software.

Connecting experiments to recovery planning

Chaos experiments examine resilience while the system is running. Recovery planning examines what happens when resilience is exhausted. The two complement each other, and a team that has rehearsed a fault injection is better placed to rehearse a wider failure. A discussion-based disaster recovery tabletop is a low-risk way to test decision-making and escalation before technical experiments reach production scale.

Our perspective on chaos engineering

In our advisory work, we observe that the technical half of chaos engineering is rarely the obstacle. Fault injection is comparatively straightforward; the difficulty lies in deciding what a normal system looks like, who owns the answer, and who is permitted to stop an experiment.

We therefore treat resilience as a combined technical and human property. An experiment succeeds when the hypothesis is explicit, the blast radius is small, the stop condition is agreed in advance, and the people involved remain clear in their communication when the system departs from expectation. That is the same discipline we teach in leadership work on clarity under pressure, and it is the reason we pair systems review with practical rehearsal.

For organizations without a dedicated reliability team, we recommend beginning with a systems and risk review that maps dependencies, identifies the single component most worth testing, and establishes the monitoring on which any experiment depends. The first experiment can then be small, documented, and repeatable.

Jesse Hart

Working with PROJECT-JTH

Organizations that want a structured starting point can rely on PROJECT-JTH’s technical advisory work, which includes systems and risk review and infrastructure automation and review. The engagement is designed to establish observable steady states, identify the first component worth testing, and document controls before any fault is injected. Details are available on the Technical Consulting and Systems Risk Review page.

Teams that prefer to build internal capability can also pair the review with mentoring on incident communication and decision-making under pressure. Further material on adjacent topics, including vulnerability management and patch management, is collected on the PROJECT-JTH Insights weblog.

FAQ

Is chaos engineering safe to run in production?

It can be, provided that the experiment is bounded and reversible. The AWS Architecture Blog specifies a defined impact scope, rollback mechanisms, real-time monitoring, and stop conditions, and the O’Reilly practitioners recommend escalating from smaller to broader impact. Teams that are new to the practice typically begin outside production and move forward as confidence grows.

What are some basic chaos engineering examples?

The most common are resource exhaustion on CPU, memory, or disk, added network latency, injected API errors, and the loss of an availability zone. Each is paired with a hypothesis, such as a circuit breaker opening or autoscaling adding capacity, and the results show whether timeouts, retries, and monitoring behave as designed.

Does a small team need special tools to begin?

Not initially. The essential prerequisites are monitoring that measures the steady state, a defined stop condition, a rollback path, and a named owner. Dedicated fault-injection and orchestration tooling becomes more valuable once experiments are repeated and automated.

How is chaos engineering different from load testing?

Load testing examines behavior under high demand, usually against expected capacity. Chaos engineering injects faults, such as failing or slow dependencies, to discover unknown failure modes, and it also assesses the human and procedural response. The two can be combined, since a fault injected during elevated load often exposes the most instructive weaknesses.

Sources

• Principles of Chaos Engineering
• AWS Prescriptive Guidance: Increasing resilience and chaos engineering on AWS
• O’Reilly: Chaos engineering
• IBM: What is Chaos Engineering?
• Google Cloud Blog: Getting started with chaos engineering
• AWS Architecture Blog: Verify the resilience of your workloads using chaos engineering
• Gremlin: Chaos Engineering Adoption Guide
• AWS Open Source Blog: Building resilient services at Prime Video with chaos engineering
• AWS Cloud Operations Blog: Chaos engineering leveraging AWS Fault Injection Simulator
• O’Reilly: Designing Experiments, Chaos Engineering
• Shopify Staging Environment: Safe Testing and Deployment