An AZ-independent architecture is one where losing a zone for three days is boring. Not “we survived,” not “the on-call heroically fixed it,” just boring. ARC zonal shift can pull traffic out of an Availability Zone in seconds, but the AWS guidance on multi-day AZ evacuation drills makes one thing clear: the shift is the easy part. Whether your stack stays healthy on day two depends on decisions you made months earlier.
Static stability: the core of an AZ-independent architecture
The principle from the Advanced Multi-AZ Resilience Patterns whitepaper is simple: when a zone fails, the surviving zones should already have what they need. No control-plane calls, no emergency instance launches, no hoping the Auto Scaling group finds capacity during the same event that is breaking things. In three-zone terms, that means each zone runs at roughly two-thirds of its ceiling, so two zones can carry 100% of the load.
AZ independence is the second half: requests that enter a zone should, as much as possible, be served inside that zone. Every cross-zone hop is a dependency that can turn a single-zone impairment into a multi-zone mess.
Hunt down hidden zonal dependencies
Before touching pods or load balancers, inventory the quiet single-zone components that nobody draws on the architecture diagram. The usual suspects:
- NAT gateways. They are zonal. One shared NAT gateway means private subnets in every zone lose outbound access when its zone goes dark. Run one per AZ and route each subnet to its own.
- Caches and brokers deployed in one zone. A single-node cache or a self-managed broker in AZ A turns every other zone into a dependent of AZ A.
- Batch and cron hosts. The lone EC2 instance that runs the nightly export lives somewhere. Find out where before the drill does.
- Interface endpoints and appliances. Make sure endpoints and any inline network appliances have a presence in every zone your workloads use.
Each one you fix removes a way for a single-zone event to spill into the others.
Spread compute so no zone is special
If one zone holds 60% of your pods, evacuating it is not a drill, it is a capacity incident. On EKS, enforce even spread with topology spread constraints and back them with anti-affinity so replicas do not stack on one node:
spec:
replicas: 6
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels: { app: checkout }
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels: { app: checkout }DoNotSchedule is the honest choice for steady state. It also means that during a shift, new replicas may sit pending because the evacuated zone is cordoned. Size replica counts and node groups so two zones already hold enough, instead of relying on the scheduler to catch up.
Decide target group health and cross-zone on purpose
Load balancers have two knobs that matter here, and the defaults were not picked with your drill in mind. First, target group health thresholds: you can tell the load balancer to treat a zone as unhealthy when it drops below a minimum number of healthy targets, failing it out of DNS before it limps along at a fraction of capacity. Second, cross-zone load balancing, which you can set per target group:
aws elbv2 modify-target-group-attributes \
--target-group-arn $TG_ARN \
--attributes \
Key=target_group_health.dns_failover.minimum_healthy_targets.count,Value=2 \
Key=target_group_health.unhealthy_state_routing.minimum_healthy_targets.count,Value=2 \
Key=load_balancing.cross_zone.enabled,Value=trueCross-zone on smooths uneven target counts but couples zones together; cross-zone off gives you cleaner AZ independence but punishes any spread imbalance. Either is defensible. What is not defensible is never having thought about it. With zonal shift, cross-zone routing to the shifted zone is blocked either way, as the ALB zonal shift docs explain.
Make the data tier AZ-aware
Databases are where AZ independence goes to die. The Aurora cluster endpoint follows the writer, so after a failover your writes land wherever the new writer lives, often in a different zone than your app pods. That is fine and expected; accept the cross-zone write path.
Reads are different. The default reader endpoint load-balances across all replicas, including the one in the zone you just evacuated. If you want zone-local reads, create custom endpoints per zone and route to them from locality-aware clients or a service mesh (an Istio ServiceEntry per zone works):
aws rds create-db-cluster-endpoint \
--db-cluster-identifier catalog-aurora \
--db-cluster-endpoint-identifier catalog-reader-az2 \
--endpoint-type READER \
--static-members catalog-aurora-reader-az2Then fix the client side, because stale connections are the classic multi-day failure. Give connection pools a maximum lifetime so they re-resolve DNS and rebalance, and stop runtimes from caching DNS forever:
# HikariCP: recycle connections every 10 minutes
maxLifetime=600000
# JVM: java.security
networkaddress.cache.ttl=30Alarm per zone, not per service
A service-wide 5XX rate averages away a zone that is on fire. Build per-AZ dashboards and alarms for HealthyHostCount and 5XX on each target group, plus AuroraReplicaLag and BufferCacheHitRatio on each instance. A hit ratio dropping below 99% right after failover tells you the new writer is reading from storage instead of cache.
aws cloudwatch put-metric-alarm \
--alarm-name checkout-healthy-hosts-use1-az2 \
--namespace AWS/ApplicationELB --metric-name HealthyHostCount \
--dimensions Name=TargetGroup,Value=targetgroup/checkout/abc \
Name=LoadBalancer,Value=app/web/abc123 \
Name=AvailabilityZone,Value=us-east-1b \
--statistic Minimum --period 60 --evaluation-periods 3 \
--threshold 2 --comparison-operator LessThanThresholdThese same alarms become the outcome alarms for zonal autoshift practice runs, so the work pays twice.
Control scaling drift and plan the restore
During a long shift, Auto Scaling settles into a two-zone shape. When the zone returns, rebalancing pours capacity back in, and a cold zone suddenly gets a third of the traffic. Pin minimum capacity before the shift, restore traffic before removing headroom, and scale down gradually over 15 to 30 minutes. Write that choreography down; it is part of the architecture.
Graduate to zonal autoshift
Once drills are boring, let AWS shift for you. Zonal autoshift moves traffic when AWS detects a zonal impairment, and requires a practice run configuration that rehearses shifts on a schedule, gated by your alarms:
aws arc-zonal-shift create-practice-run-configuration \
--resource-identifier $ALB_ARN \
--outcome-alarms type=CLOUDWATCH,alarmIdentifier=$OUTCOME_ALARM_ARN \
--blocking-alarms type=CLOUDWATCH,alarmIdentifier=$BLOCKING_ALARM_ARN
aws arc-zonal-shift update-zonal-autoshift-configuration \
--resource-identifier $ALB_ARN --zonal-autoshift-status ENABLEDThe zonal shift best practices lay out the readiness bar. If your architecture cannot pass a practice run, it was never going to pass a real impairment.
At Enkompass (enkompass.net), this is our home turf. We have redesigned EKS and Aurora workloads for zone independence, from spread constraints and AZ-local reader endpoints to per-zone alarms that gate zonal autoshift practice runs. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.