Tutorial: Run a 72-Hour Zonal Shift Drill on AWS

This walkthrough runs a full 72-hour zonal shift drill with Amazon Application Recovery Controller (ARC): you will evacuate one Availability Zone across an ALB, an EKS cluster, an ECS service, RDS Multi-AZ and Aurora, keep it out for three days, and bring it back without a self-inflicted outage. It follows the reference architecture from the AWS post on multi-day AZ evacuation drills. Run it in staging first. Seriously.

Prerequisites

• A workload spread across three AZs, with cross-zone load balancing on.
• N-1 capacity validated: two zones can carry full load, or you pre-scale before the drill.
• Per-AZ CloudWatch dashboards for request count, 5XX and healthy hosts.
• ALB target group deregistration delay of 60 seconds and ECS task stopTimeout of 55 seconds, so connections drain before tasks die.
• EKS workloads using topology spread constraints with maxSkew: 1 across zones.
• AWS CLI v2 and kubectl with admin access to the account and cluster.

Run the zonal shift drill step by step

1. Map zone names to zone IDs and set drain timing. ARC uses zone IDs; your dashboards probably use names. The mapping differs per account, so the us-east-1a on your dashboard is not necessarily use1-az1 in ARC. Pick the AZ to evacuate (we use use1-az1), write the mapping into the drill runbook, and confirm the target group drain setting so in-flight requests finish before targets drop out.

   aws ec2 describe-availability-zones \
     --query "AvailabilityZones[].[ZoneName,ZoneId]" --output table
   
   aws elbv2 modify-target-group-attributes \
     --target-group-arn $TG_ARN \
     --attributes Key=deregistration_delay.timeout_seconds,Value=60

2. Enable zonal shift on the load balancer. It is off by default on ALBs and NLBs, per the ELB zonal shift docs. Repeat for every NLB in the path. Note that an ALB will refuse a shift if any target group has healthy targets in only one AZ, so fix lopsided target groups now rather than at kickoff.

   aws elbv2 modify-load-balancer-attributes \
     --load-balancer-arn $ALB_ARN \
     --attributes Key=zonal_shift.config.enabled,Value=true

3. Enable zonal shift on the EKS cluster. See the EKS zonal shift guide for what this changes. Once enabled, a shift cordons nodes in the zone, removes its pod endpoints from EndpointSlices, suspends AZ rebalancing and keeps node group Auto Scaling in the healthy zones. Pods keep running; they just stop receiving traffic. The second command confirms ARC can see both resources.

   aws eks update-cluster-config \
     --name prod-cluster \
     --zonal-shift-config enabled=true
   
   aws arc-zonal-shift list-managed-resources

4. Start the shifts with a 72-hour expiry. One shift per resource. The expiry is your safety net if everyone forgets about the drill. Start the load balancer and cluster shifts within a minute of each other so external and in-cluster traffic move together, and post the shift IDs in the drill channel.

   aws arc-zonal-shift start-zonal-shift \
     --resource-identifier $ALB_ARN \
     --away-from use1-az1 --expires-in 72h \
     --comment "72h AZ evacuation drill"
   
   aws arc-zonal-shift start-zonal-shift \
     --resource-identifier $EKS_CLUSTER_ARN \
     --away-from use1-az1 --expires-in 72h \
     --comment "72h AZ evacuation drill"

5. Verify traffic actually moved. Nodes in the evacuated zone should show SchedulingDisabled, EndpointSlices should drop that zone’s endpoints, and ALB RequestCount for that zone should fall toward zero.

   kubectl get nodes -L topology.kubernetes.io/zone
   kubectl get endpointslices -A -o wide
   
   aws cloudwatch get-metric-statistics \
     --namespace AWS/ApplicationELB --metric-name RequestCount \
     --dimensions Name=LoadBalancer,Value=app/web/abc123 \
                  Name=AvailabilityZone,Value=us-east-1a \
     --start-time 2026-10-14T09:00:00Z --end-time 2026-10-14T10:00:00Z \
     --period 300 --statistics Sum

Give the shift fifteen minutes before you call it good. DNS caches need to expire, and long-lived client connections need to recycle before the per-zone numbers settle. If a zone’s request count refuses to drop, look for clients that pin IPs or connection pools with no maximum lifetime. That finding alone is worth the drill.

6. Pull ECS out of the evacuated subnets. Zonal shift stops ALB traffic to those tasks, but ECS will still place tasks there. Update the service to use only healthy-zone subnets. ECS replaces tasks as part of the deployment, and the 55-second stop timeout gives each task time to finish requests. Repeat for every service behind the shifted load balancer.

   aws ecs update-service \
     --cluster prod --service web \
     --network-configuration "awsvpcConfiguration={subnets=[subnet-bbb,subnet-ccc],securityGroups=[sg-123],assignPublicIp=DISABLED}"

7. Fail over the databases. If the RDS primary lives in the evacuated AZ, force a Multi-AZ failover; RDS then rebuilds the standby. For Aurora, promote a reader in a healthy zone. With a pre-provisioned reader this takes under 30 seconds.

   aws rds reboot-db-instance \
     --db-instance-identifier orders-db --force-failover
   
   aws rds failover-db-cluster \
     --db-cluster-identifier catalog-aurora \
     --target-db-instance-identifier catalog-aurora-reader-az2

8. Extend if needed on day three. Found something worth watching longer? Push the expiry out instead of starting a new shift, which keeps the history in one record. Use the extra time to let the weekly jobs, certificate rotations and scaling cycles that only run occasionally take their turn.

   aws arc-zonal-shift list-zonal-shifts --status ACTIVE
   
   aws arc-zonal-shift update-zonal-shift \
     --zonal-shift-id $SHIFT_ID --expires-in 24h

9. Restore in order. Confirm the zone is healthy, cancel the EKS shift first so pods are reachable, then the load balancer shift. Re-add the ECS subnets, scale down extra capacity gradually over 15 to 30 minutes, and watch for 30 more.

   aws arc-zonal-shift cancel-zonal-shift --zonal-shift-id $EKS_SHIFT_ID
   aws arc-zonal-shift cancel-zonal-shift --zonal-shift-id $ALB_SHIFT_ID
   
   aws ecs update-service --cluster prod --service web \
     --network-configuration "awsvpcConfiguration={subnets=[subnet-aaa,subnet-bbb,subnet-ccc],securityGroups=[sg-123],assignPublicIp=DISABLED}"

10. Export the audit trail. Save the shift history alongside dashboard snapshots and failover events. That bundle is your evidence for auditors, and the baseline for the next drill.

    aws arc-zonal-shift list-zonal-shifts \
      --resource-identifier $ALB_ARN --output json > drill-shifts-alb.json
    
    aws rds describe-events --source-type db-cluster \
      --duration 4320 --output json > drill-rds-events.json

What you built

You enabled zonal shift on a load balancer and an EKS cluster, evacuated an AZ for three days, handled the parts ARC does not cover (ECS placement and database leadership), restored in a safe order and captured evidence. That is a repeatable drill, not a one-off stunt. Next time, rotate which AZ you evacuate, because each zone hides different surprises, and compare the restore timeline against the previous run. If the findings point at static stability gaps, the Advanced Multi-AZ Resilience Patterns whitepaper is the right next read.

Cleanup and next steps:

• Confirm list-zonal-shifts --status ACTIVE returns nothing.
• Remove any scheduled scaling you added for the drill so you stop paying for N-1 headroom you no longer need.
• Decide whether the original Aurora writer placement matters and fail back if so.
• File every finding as a ticket with an owner. Then read the zonal shift best practices and consider zonal autoshift with practice runs.

At Enkompass (enkompass.net), this is our home turf. We have scripted and supervised multi-day AZ evacuations across ALB, EKS, ECS and Aurora stacks, including the restore choreography most teams skip. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.