Circuit Breaker Testing: A Practical Guide for Engineers
A downstream service starts returning 500s during a busy release window. Your circuit breaker opens, requests stop piling onto the failing dependency, and the incident initially looks contained. Then the dependency recovers, but the breaker never returns to normal because the test suite never exercised its half-open transition under realistic concurrency. The service remains degraded even though its dependency is healthy again.
That failure is common because teams test the idea of a circuit breaker more often than they test its behavior. A few unit tests confirm that a counter increments and an enum changes. They don’t prove that callers are protected during a real failure, that only an appropriate probe reaches a recovering dependency, or that the breaker closes cleanly after recovery.
Circuit breaker testing must prove three things: the breaker trips when failure conditions justify it, protects the caller while the dependency remains unhealthy, and recovers without trapping healthy traffic behind stale state. Those outcomes apply to software libraries and to electrical protection equipment, although the test methods differ sharply.
Why Circuit Breaker Testing Matters More Than the Pattern Itself
A breaker that never opens creates a false negative. The application continues sending work to a failing dependency, consumes connection pools, increases latency, and may spread the outage to otherwise healthy services. A breaker that opens and never closes creates a false positive. It protects the dependency but turns a temporary fault into a lasting availability problem for callers.
The most dangerous bugs sit between those two obvious cases. A timeout may count as a failure in one code path but not another. A retry may increment the breaker counter twice. A half-open probe may race with normal traffic and allow a flood through. A process restart may reset local state while other instances remain open. Tests that only inspect the final state won’t reveal these transitions.
Practical rule: A passing test should demonstrate a state transition and its operational consequence, not merely prove that a method returned the expected enum.
Test the recovery path, not only the outage
A useful baseline has a deterministic failure sequence, an observable open state, rejected or short-circuited calls while open, a controlled half-open probe, and successful closure after recovery. The assertions should include timing behavior, concurrency behavior, and the downstream request count. If the breaker opens but the dependency still receives normal traffic, the protection mechanism isn’t working.
Production evidence supports the broader lesson that testing and inspection need careful controls. A U.S. nuclear-industry failure review covering 21 years found that improved maintenance, testing, and inspection requirements were associated with a statistically significant reduction in common-cause-failure events, with p = 0.0001 (failure review from the U.S. nuclear industry). In that reviewed event set, testing was associated with 60% of events, compared with 21% attributed to demand and 9% to maintenance, a result that highlights the risk of poor test configuration, restoration, and post-test verification rather than an inherent problem with testing.
Use the right vocabulary
Software teams usually discuss closed, open, and half-open states. Electrical engineers test whether a breaker can make and interrupt fault current, withstand dielectric stress, and operate mechanically. These are different systems, but the engineering discipline is similar: define the failure mode, create a controlled stimulus, observe the transition, and verify safe recovery.
For medium- and high-voltage AC breakers, IEC 62271-100 test guidance defines making and breaking tests at 100%, 90%, 60%, and 30% of rated short-circuit breaking current, designated T100, T90, T60, and T30. It also specifies transient recovery voltage behavior, because interrupting current isn’t enough if the voltage across the contacts rises faster than the insulation can withstand.
For software, the equivalent question is not “does the breaker class compile?” It is “does the breaker change state under the failure that matters, and does the surrounding system behave safely at every transition?”
The Five Test Types That Actually Catch Breaker Bugs
Each test layer answers a different question. Treating them as interchangeable leaves blind spots, especially around concurrency, recovery, and unusual request patterns.
Unit tests validate the state machine
Unit tests should answer whether the implementation applies thresholds and windows consistently. Inject a deterministic clock, send the configured number of failures, advance time into the half-open interval, and verify that exactly the permitted probe behavior occurs.
A useful assertion is: after the failure threshold is reached within the configured window, the state becomes open and subsequent calls don’t invoke the dependency. Add tests for counter reset after successful calls, timeout classification, concurrent updates, and transitions caused by clock movement.
Integration tests validate real interaction
Use a controllable downstream service that can return scripted 5xx responses, delays, malformed responses, and connection closures. The test should observe both the caller and the dependency, because a breaker can report “open” while an adapter still sends requests through a separate path.
For example, return failures until the breaker opens, restore successful responses, then assert that a half-open probe reaches the dependency and that normal traffic resumes only after the configured recovery rule is satisfied.
Contract tests catch response interpretation errors
A breaker often classifies failures based on status codes, headers, exceptions, or response bodies. Contract tests verify that the client and dependency agree on those signals. They catch bugs where a newly introduced error schema is treated as success, or where a valid response is incorrectly counted as a failure.
This layer doesn’t replace failure injection. It proves that the client recognizes the failures the integration test intends to create.
Chaos tests exercise environmental failure
Kill a downstream process, partition the network, delay connections, or exhaust a dependency’s ability to accept work. The assertion should focus on caller protection: bounded waiting, controlled fallback, stable resource usage, and no cascading retry storm.
Chaos testing is valuable when the failure doesn’t arrive as a neat application response. It exposes assumptions about DNS, connection pools, process restarts, and transport-level errors.
Replay tests expose distribution mistakes
Synthetic requests usually represent the traffic engineers remembered to model. Recorded traffic includes uncommon endpoints, odd parameter combinations, header variations, retries, and clients that behave differently from the happy path. Replay tests ask whether the breaker policy remains appropriate against that distribution.
The five layers work together. Unit tests are fast and precise, chaos tests are realistic about environmental failure, and replay tests are realistic about request shape. None proves the others.
Building a Layered Test Plan With Pass and Fail Criteria
Start with the cheapest evidence and move toward tests that require more infrastructure. Every layer needs an assertion that describes a failure a user or operator would notice.
At the unit layer, control time and concurrency. Assert state transitions, counter resets, probe admission, and rejection behavior. Avoid sleeping in these tests. A fake clock makes the half-open path repeatable and prevents timing noise from turning a state-machine test into a scheduler test.
At the integration layer, run a scripted downstream. The test should verify that the breaker opens after the configured failure condition, rejects calls while open, permits the intended probe during half-open, and closes only after the recovery rule is met. Record downstream invocation count, because a response from the breaker alone doesn’t prove that traffic was stopped.
A chaos test should remove the dependency during active traffic. Pass criteria include bounded caller impact, no unbounded queue growth, and a breaker transition that remains visible in telemetry. The test fails if the application keeps waiting for connection timeouts after the breaker should have protected it.
Load testing adds a different concern. Send concurrent requests across the same breaker instance and across instances where shared state is expected. Look for leaked state, race conditions, uneven probe admission, and recovery behavior that changes with scheduler timing.
| Layer | What it proves | Example assertion |
|---|---|---|
| Unit | State-machine correctness | A failure sequence opens the breaker, and an injected clock transition permits the expected half-open probe |
| Integration | Client and dependency interaction | Open state prevents downstream calls, while recovery traffic follows the configured probe rule |
| Contract | Failure classification | The client maps the dependency’s documented error response to the intended breaker outcome |
| Chaos | Protection during environmental failure | A dependency outage doesn’t create unbounded waiting or a retry cascade |
| Load and replay | Behavior under realistic concurrency and traffic shape | State transitions and downstream volume remain consistent with the approved baseline |
Choose an acceptance model deliberately
Absolute pass/fail thresholds work well for deterministic unit tests. They become brittle for replay and load tests, where scheduling, connection reuse, and background activity can alter raw timings. A trend-based baseline is usually more useful for breaker state, latency distributions, and downstream call volume.
Fingerprint comparison is another option. Capture a known-good sequence of state transitions, timing relationships, and event counts, then compare a candidate build against it. Don’t demand byte-for-byte identity when the behavior is concurrent. Flag meaningful changes for review instead.
For electrical equipment, acceptance should similarly start with manufacturer limits and historical trends. Field-test guidance for medium- and high-voltage breakers describes isolation and verification, static contact resistance with at least 100 A DC injection, insulation resistance for one minute, and operational timing and travel tests. It also emphasizes trend interpretation, pole spread, rebound, and auxiliary contacts rather than relying on one pass/fail reading.
Why Replaying Real Traffic Beats Synthetic Chaos Tests
Synthetic load is fast, repeatable, and easy to scale. It also reflects the model its author created. If the model omits a rare endpoint that produces unusually expensive downstream work, the test can declare success while the production breaker remains poorly calibrated.
Fault injection solves a different problem. It tells you what happens when the dependency fails, but it doesn’t necessarily reproduce the traffic distribution that causes the breaker to trip too early or too late. A service may tolerate a broad outage yet fail under a narrow combination of endpoint frequency, retries, slow clients, and response sizes.
Three methods, three blind spots
Synthetic load generators give you controlled concurrency and predictable request rates. They are ideal for testing saturation, threshold behavior, and resource limits, but they only cover scenarios someone encoded.
Fault-injection tools make outages deliberate. They are strong for testing timeout handling, process failure, network partitions, and recovery. They don’t automatically capture the long tail of real request shapes.
Production-traffic replay preserves observed request patterns while sending them to a controlled target. It is harder to make perfectly deterministic, and sensitive data must be handled carefully, but it can expose a breaker that trips because one uncommon route dominates dependency failures.
A breaker policy can be correct for a synthetic workload and wrong for the workload your users actually create.
The practical answer isn’t to replace synthetic chaos with replay. Use synthetic tests to isolate a failure mode, chaos tests to disrupt the environment, and replay to verify that the policy survives real traffic shape. A practical chaos experiment workflow is useful when the replay and fault-injection steps need to be coordinated without sending destructive traffic to production.
Replay also changes the question you ask. Instead of “does the breaker open when the dependency returns errors?” you can ask “which real request patterns cause it to open, what does the caller experience, and does it recover without an abnormal surge?” That evidence is much closer to the operational decision you need to make.
Running a Replay-Based Circuit Breaker Test With GoReplay
GoReplay is an open-source tool for capturing and replaying live HTTP traffic into testing environments. Its official product documentation describes PRO capabilities including documented TCP-session recognition, advanced traffic storage, data masking, and priority support for enterprise use.
The safest workflow separates capture, sanitization, replay, and fault injection. Don’t capture indiscriminately and point the result at a candidate system that can reach production data stores.
Prepare the capture
Install the listener on a non-critical host or another approved capture point. Select a bounded route set or user cohort, and exclude write-heavy paths unless the candidate environment has isolated, disposable dependencies. Preserve enough context to reproduce request ordering and timing, but remove credentials, tokens, personal data, and payload fields that the test doesn’t need.
Store the capture as a versioned test artifact. Record its scope, filtering rules, masking policy, and expected traffic characteristics. A captured file makes later builds comparable, while a fresh live capture changes too many variables at once.
Prepare the candidate
Deploy the candidate service with the circuit breaker configured to emit every transition, rejection, probe, and downstream outcome. Add correlation identifiers that connect the replayed request to the breaker event without exposing original user identifiers.
Place the downstream behind a controllable test double or isolated service. Inject failures there, not by allowing replayed requests to damage a real dependency. Start with one failure mode, such as server errors or connection delays, then test combined behavior once the basic transition is understood.
Replay and compare
Replay at the original relative timing when timing matters. Throttle the stream when the candidate environment can’t safely absorb the captured volume, and document the throttle because it changes the experiment. Run the same capture against a baseline build and a candidate build, then compare state transitions, rejected requests, downstream calls, latency, and recovery.
An example assertion set can be explicit without pretending that one policy fits every service:
- The breaker opens within 5 seconds of the downstream error rate crossing 50%, with those values treated as test-specific acceptance criteria rather than universal defaults.
- The breaker enters half-open within the configured recovery window.
- The breaker closes after 3 successful probes in half-open, again as a policy chosen for this service.
- The downstream doesn’t receive ordinary traffic while the breaker is open.
- The candidate doesn’t produce unexplained state flapping compared with the approved baseline.
Those numbers are useful only when they match the configured policy and the service’s risk. The test proves implementation consistency, not industry compliance.
Wiring Circuit Breaker Tests Into CI Without Flaking Every Build
Fast feedback and deep confidence belong in different pipeline stages. Running every replay and chaos experiment on every commit makes developers wait for tests that are sensitive to infrastructure and timing. Running only unit tests creates confidence that the state machine works in isolation, not that the service behaves correctly under realistic failure.
Put deterministic tests on every commit
Unit tests and lightweight integration tests should run for each change. Give them isolated dependencies, injected clocks, controlled randomness, and explicit cleanup. A test that leaves a process, socket, queue, or background retry loop running will contaminate later tests and create failures that are difficult to reproduce.
Contract tests can also run early when their fixtures are stable. Fail the build on a changed error classification, an unexpected state transition, or a request that reaches the dependency while the breaker is open.
Schedule expensive evidence
Run chaos and replay tests nightly, before a release, or manually for a deployment candidate. Use a fixed captured traffic artifact rather than capturing live during CI. Pin its scope and masking version, store it alongside the test configuration, and run the same artifact against multiple builds for meaningful comparison.
A GitHub Actions testing workflow can provide the orchestration layer, but reliability still depends on test isolation. Give each run its own candidate environment, terminate background processes in cleanup hooks, and avoid parallel replay into a shared dependency.
Clock skew creates especially confusing failures. Keep the candidate, replay controller, and metrics collector synchronized, or make assertions depend on event ordering and relative durations rather than wall-clock timestamps. If a replay test is non-deterministic by design, report a baseline range and require investigation when the candidate falls outside it.
Slow tests should gate releases when their evidence matters, not delay every edit that doesn’t touch breaker behavior.
Metrics, Assertions, and the Monitoring Feedback Loop
The metrics used in tests should match the metrics operators use in production. At minimum, record state, transition count, time in each state, request volume by state, and downstream latency and error rate by state. Export them through the same Prometheus or OpenTelemetry instrumentation used by the deployed service.
A test can then assert that an outage produces an open transition, that open-state request volume behaves as intended, and that recovery produces a controlled half-open sequence. Production alerts can use the same definitions to detect a breaker stuck open, flapping between states, or remaining closed while downstream errors rise.
For electrical breakers, the equivalent feedback loop combines operating timing, contact resistance, travel, coil current, auxiliary contacts, and historical fingerprints. IEEE high-voltage breaker timing guidance calls for capturing main and auxiliary contact behavior, relevant resistor contacts, travel characteristics, and phase synchronization. It specifies closing synchronization better than one-quarter cycle and opening synchronization better than one-sixth cycle, which at 50 Hz corresponds to less than 5 ms for closing and 3.33 ms for opening, while at 60 Hz the corresponding values are 4.17 ms and 2.78 ms.
Keep this runbook beside the test configuration:
- Capture traffic: Select a bounded, representative request set.
- Scrub sensitive data: Remove credentials, personal data, and unsafe write effects.
- Replay against a candidate: Use an isolated downstream and controlled faults.
- Assert transitions: Verify open, half-open, and closed behavior.
- Compare baselines: Review state, timing, volume, and error metrics.
- Gate the release: Treat unexplained behavioral changes as release evidence.
- Monitor production: Use the same definitions after deployment.
GoReplay captures and replays live HTTP traffic into controlled testing environments, giving your team a way to validate circuit breaker behavior against realistic request patterns rather than only synthetic scenarios. Visit GoReplay to evaluate how recorded traffic, masking, and replay can fit into your failure-testing workflow.