Most disaster recovery plans live in a document nobody reopens until an auditor asks for it. That's usually fine until the system in question is critical or holds sensitive data. Then the plan is only worth what you've actually tested.
A few days ago I ran a real AWS FIS experiment against a pre-prod environment: simulate a full power interruption in one Availability Zone, watch what breaks, watch what doesn't. This is the writeup of that exercise.
The setup
The stack under test was pretty much a standard multi-AZ architecture having EKS on EC2 nodes hosting the application, Aurora PostgreSQL (writer/reader pair) behind it, an NLB in front of Nginx+, and Route 53 handling DNS. Nothing exotic. Which is exactly why it was a good candidate, also if a boring, well-architected setup can't survive losing a zone cleanly, no amount of FIS testing on the exotic stuff matters. AWS FIS has a pre-built scenario for this: AZ Availability: Power Interruption. It doesn't just stop instances, but has it chain together the actual symptom set of a real power loss in a zone:
aws:ec2:stop-instances: stops every EC2 instance in the target AZ for the interruption window, then restarts them after.aws:ec2:asg-insufficient-instance-capacity-error: blocks the ASG from spinning up replacement capacity in that same AZ, forcing it to actually prove it can scale elsewhere.aws:rds:failover-db-cluster: forces the Aurora writer to fail over if it happens to be sitting in the affected zone.aws:network:disrupt-connectivity: cuts network connectivity for all subnets in the AZ for 2 minutes, which is long enough to force DNS timeouts and refreshes without also hiding the AZ's continued unavailability from the rest of the test.
That last one is the detail I like most about this scenario. It doesn't disrupt the network for the full outage window but just the first 2 minutes. That's deliberate because it's enough to force clients to hit stale DNS and retry, without preventing AWS's own regional services from updating their records for the rest of the test.
Test parameters
- Target AZ: eu-central-1a
- Induced failure: 1 hour 30 minutes
- Observed recovery: 30 minutes
- Timing: during business hours, on purpose to ensure there were real traffic, with users still on the system
The hypotheses
Before running it, I wrote down what we expected to see:
- Traffic redirects to healthy infrastructure.
- Unhealthy EC2 instances get replaced by the ASG in another zone.
- EKS nodes terminate and pods reschedule elsewhere.
- Aurora fails over if the writer landed in the affected AZ.
- A brief latency bump during the transition.
- The whole thing rebalances once the zone comes back.
Six hypotheses. If any of them didn't hold, that's the actual finding.
What I saw
Traffic shifted fast
Active flow count in eu-central-1a dropped off while eu-central-1b picked it up in the same window, just a classic failover, no ambiguity in the graph
The ASG did its job under a harder constraint than usual
An EC2 instance in 1a went unhealthy from the injected capacity error, and instead of retrying in the same (blocked) zone, the ASG launched the replacement directly in 1b. This is the part I actually wanted to see because it's easy for an ASG to look healthy when it's allowed to just retry in place. FIS blocking capacity in the target zone forces it to prove the cross-AZ path works, not just the retry path.
EKS followed the same pattern
Node terminated in the impacted AZ, replacement node came up, pods rescheduled. No manual intervention, no stuck pods.
The DNS health dip was short and matched the timeline exactly
Zonal health for 1a dropped to 0.8 in about a five-minute window around the power interruption, then returned to 1.0. It lined up precisely with when the outage was injected signifying no lag, no false negatives elsewhere.
Aurora is where it got interesting
Write latency on the node in the affected AZ went flat because it was taken out of the write path entirely, as expected. The other node took a real latency spike as it absorbed the failover traffic, then settled. Connection count dropped hard right as the zone's health bottomed out and then not only recovered but spiked slightly above baseline afterward, which reads like client-side retry behavior kicking in and briefly overshooting.
Every hypothesis held. Nothing needed a manual fix. That's a good outcome, but it's also the least interesting part of the story.
What I took from this
A clean result is not the same as a wasted test. It's tempting to think a DRP exercise where nothing breaks didn't teach you anything. It taught me the ASG will actually use a healthy AZ under pressure instead of just retrying blindly and that's a real claim I can now make with evidence instead of assuming it from the architecture diagram.
Blocking the "easy" recovery path is what makes the test honest. The asg-insufficient-instance-capacity-error action is the whole point of this scenario. Without it, you'd just be testing "does the ASG replace a dead instance," which it will always do. With it, you're testing "does the ASG replace a dead instance somewhere useful," which is the question that matters during a real AZ loss.
The 2-minute network disruption window is a smart constraint, not a limitation. It forces the DNS-refresh behavior you'd see in a real outage without artificially masking the AZ's unavailability for the rest of the hour. If your own FIS experiments run connectivity disruption for the full outage window, you're testing something slightly different than a real AZ power loss.
Aurora's connection-count overshoot after recovery is worth watching, not just noting. A small spike above baseline after a failover usually means retry storms from the client side. It stabilized here, but on a busier production system that's exactly the kind of thing that turns a clean failover into a self-inflicted second incident if your retry/backoff settings aren't tuned for it.
Unlike in my previous write-ups, none of this was dramatic, no 2 AM alerts, no war room and no postmortem. That's exactly why it's worth writing about. The system failures that make good war stories just like the ones I've written before all share similar root cause: nobody tested the failure path until the failure showed up uninvited. This time I invited it. It behaved. Now I have evidence instead of a diagram and a hope, and a baseline to hold every future test against.