If your resilience program consists of flipping traffic out of a zone for five minutes, nodding at a dashboard and flipping it back, you have tested a switch, not a business. A multi-day AZ evacuation drill, where you deliberately run production on two Availability Zones for 48 to 72 hours, is how you find out whether your platform can actually live without a zone. AWS just published a detailed playbook for running these drills with Amazon Application Recovery Controller (ARC) zonal shift, and it is worth a seat at the leadership table.
The problem with five-minute tests
Short failover tests answer one narrow question: does traffic move? Useful, but real zonal impairments rarely resolve in five minutes. When an AZ degrades, you may be running without it for hours or days, and the failures that hurt you in that window are not the ones a quick test can see.
Here is what tends to break only with time:
- Auto Scaling drift. Capacity settles into a new shape over a day of real traffic cycles, and nobody has validated that shape.
- Scheduled jobs. The nightly batch, the weekly report and the cron job pinned to a subnet in the missing zone all get their turn.
- Certificate and credential rotation. Rotation workflows that quietly depend on something in the evacuated AZ fail on their own schedule, not yours.
- DNS caching and long-lived connections. Clients and connection pools that never re-resolve keep reaching for the old zone.
- The restore itself. Rolling back a five-minute shift is trivial. Bringing a zone back after three days of drift, while scaling down safely, is a real operation that deserves rehearsal.
What an AZ evacuation drill gives you that a tabletop exercise cannot
The obvious output is confidence. The less obvious one is evidence. Regulated industries, financial services in particular, are under growing pressure to show operational resilience rather than describe it. A sustained drill produces timestamped shift history from ARC, per-zone dashboards showing traffic draining and recovering, database failover records and a written restore timeline. That is the kind of artifact an auditor or regulator can actually evaluate.
It also changes internal conversations. “We think we can lose a zone” becomes “we ran on two zones for 72 hours in October, here is what broke and here is what we fixed.” That is a far stronger position when the board asks about concentration risk.
What it costs
The zonal shift capability itself carries no additional charge, per the ARC documentation. The real cost is capacity. If you run across three AZs and lose one, the remaining two must absorb the full load, which means each carries 50% more than usual. Running pre-scaled for that N-1 scenario all the time is roughly 50% more compute than a lean three-zone footprint.
You have options, and each is a business decision, not just an engineering one:
- Always-on N-1 headroom. Highest cost, lowest risk, and the only option that is truly statically stable.
- Scheduled scaling for the drill window. Pre-scale ahead of the drill, pay for the extra capacity for three days, then scale back.
- Aggressive scale-out policies. Cheapest at rest, but you are betting that you can launch capacity during the very event that might impair launches.
The drill tells you which of these bets your platform can actually afford. That alone justifies the spend.
There is also a people cost: an on-call rotation watching per-zone dashboards for three days, plus engineering time to fix whatever the drill uncovers. Budget for that explicitly. The findings list is the point of the exercise, and a drill whose fixes never get scheduled is just expensive theater. Compare all of it against the alternative line item nobody likes to write down: the revenue, penalties and reputational damage of discovering these gaps during a real zonal event, with customers watching and no expiry timer to save you. Framed that way, three days of extra capacity is cheap insurance with a receipt.
The risks, stated plainly
An evacuation drill in production is not free of risk, and anyone who says otherwise is selling something. The main exposures:
- Under-provisioned survivors. If N-1 capacity was never validated, the drill becomes an outage.
- Services zonal shift does not cover. ARC handles load balancers and EKS traffic, but database primaries and some compute placement require separate, deliberate failover steps.
- Restore missteps. Bringing traffic back too fast to a zone with cold capacity can cause its own incident.
Each of these is manageable with sequencing, monitoring and a clear owner. The shifts have expiry times built in, so a forgotten drill ends itself rather than lingering forever.
A sensible adoption path
Do not start with your payments platform on a Monday. A path that works:
- Non-production first. Run a full 72-hour drill in staging to shake out tooling, runbooks and dashboards.
- Low-traffic production. Pick a lower-risk production workload and repeat, with the team watching per-zone metrics.
- Critical workloads. Move to tier-one systems once N-1 capacity and database failover are proven.
- Zonal autoshift. Once you trust the mechanics, let AWS shift traffic automatically when it detects a zonal impairment, with scheduled practice runs keeping you honest. The zonal shift best practices guide covers the readiness bar.
Pair the program with AWS Resilience Hub to define recovery time and recovery point objectives per application, so each drill is measured against a target the business has agreed to, not a vibe.
What to ask your team this week
You do not need to read API docs to drive this. You need answers to a short list:
- Have we ever run production on two zones for more than an hour?
- Is zonal shift enabled on our load balancers and EKS clusters? It is off by default on load balancers.
- Do we know our N-1 capacity cost, and which scaling strategy we chose on purpose?
- Where do our database primaries live, and how long does failover take? AWS cites under 30 seconds for Aurora with a ready replica, closer to 10 minutes if one must be built.
- Could we hand an auditor evidence of our last zonal recovery today?
If most answers are “not sure,” that is your roadmap. The AWS Advanced Multi-AZ Resilience Patterns whitepaper is a good primer for the engineering leads who will own it.
At Enkompass (enkompass.net), this is our home turf. We plan and run zonal evacuation programs end to end, from N-1 capacity modeling to producing the drill evidence that compliance teams actually accept. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.