Self-Healing ECS Architecture: Health Events + EventBridge

Getting the AGENT_CONNECTIVITY signal out of ECS is the easy half of the problem on EC2. The harder half is building a self-healing ECS architecture around it that doesn’t create new incidents of its own — the classic trap with any auto-remediation system. This is about the pattern specifically for self-managed EC2 clusters: how to structure the EventBridge-to-Lambda pipeline, how to avoid replacement storms, and how to keep it from fighting with the ALB health checks you already have in place.

The baseline pattern for a self-healing ECS architecture

The building blocks are the same three AWS has always given you for event-driven remediation: an EventBridge rule matching ECS container instance state changes, a Lambda function that decides what to do, and enough observability to know whether the automation itself is behaving. A minimal rule scoped to draining instances:

{
  "source": ["aws.ecs"],
  "detail-type": ["ECS Container Instance State Change"],
  "detail": {
    "status": ["DRAINING"]
  }
}

Wired to a Lambda target:

aws events put-rule \
  --name ecs-remediation-rule \
  --event-pattern file://agent-connectivity-pattern.json

aws events put-targets \
  --rule ecs-remediation-rule \
  --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123456789012:function:ecs-remediation-handler","RetryPolicy"="{MaximumRetryAttempts=2,MaximumEventAgeInSeconds=3600}"

That retry policy matters more than it looks like it should. Remediation Lambdas fail sometimes — throttling, a transient IAM issue, a cold-start timeout — and the default EventBridge behavior without an explicit retry policy can leave you assuming automation ran when it didn’t. Set it explicitly, and pair it with a dead-letter queue so failed remediation attempts land somewhere you’ll actually see them instead of silently vanishing.

Designing the Lambda so it doesn’t overreact

The naive version of this Lambda drains and terminates on every event. Don’t build that version. Two design decisions keep it from becoming the thing that pages you:

• Check current state before acting. Call DescribeContainerInstances inside the handler before draining anything — if the instance already shows DRAINING or has already been deregistered, exit without doing more work. Idempotency here isn’t optional; EventBridge can deliver the same event more than once, and ECS can emit adjacent state-change events in quick succession.
• Rate-limit remediation per cluster. Use a DynamoDB item (or a Step Functions execution) as a lock — if more than N instances in the same cluster have been remediated in the last M minutes, stop and alert a human instead of continuing automatically. A pipeline that “successfully” drains half your fleet in ten minutes because of a shared root cause (a bad AMI, a botched security group change) is not a win.

Avoiding flapping and false-positive replacement storms

Flapping is the real risk in any health-driven automation: an instance that’s marginal rather than truly dead can bounce between healthy and unhealthy states, and a naive remediation pipeline will happily drain and replace it repeatedly, burning capacity and creating churn that looks like an incident from the outside. A few concrete guards:

• Debounce before acting. Don’t remediate on the first DRAINING event for an instance. Hold state (again, DynamoDB works fine) and require the condition to persist across a short window — 60 to 120 seconds is usually enough to filter out a transient blip without meaningfully slowing recovery from a real failure.
• Track a per-instance remediation count. If the same instance ID has been drained and replaced by your automation more than once in a short window, that’s a signal something upstream is wrong — a bad launch template, a subnet with a connectivity problem — and the fix is not “replace it a third time,” it’s paging someone.
• Emit a custom CloudWatch metric on every remediation action. A metric like RemediationsTriggered with cluster and reason as dimensions gives you a CloudWatch alarm on remediation frequency itself, which is often the earliest warning that something systemic is going on rather than an isolated instance failure.

Where ALB target group health checks fit in

If your ECS service sits behind an Application Load Balancer target group, you now have two health-checking systems watching related but distinct things: ALB target health checks (is this task responding to HTTP requests) and the ECS AGENT_CONNECTIVITY event (can the ECS agent talk to the control plane). They can legitimately disagree — an instance can fail agent connectivity while its running tasks are still serving traffic just fine for a few more seconds, or an ALB can flag a target unhealthy for reasons that have nothing to do with agent connectivity at all, like a slow-starting container.

Two things matter for keeping these systems from working against each other:

• Make sure your target group’s deregistration delay is long enough to let in-flight requests finish before ECS’s drain-and-replace pulls the instance out from under the ALB. A deregistration delay shorter than your slowest request’s typical duration turns a clean drain into dropped connections.
• Let ECS’s drain mechanism do the deregistration, rather than building a separate Lambda path that also tries to modify target group membership. Two automated systems independently touching the same target registration is exactly the kind of race condition that produces the flapping this whole section is about avoiding.

Alerting and observability design

A remediation pipeline that runs silently is a pipeline nobody trusts once something goes wrong with it. Treat the automation itself as a service with its own SLOs, not just a background script. Concretely:

• Log every decision, not just every action. When the Lambda decides not to remediate — because the instance is already draining, or the rate limit tripped — log that too, with the reason. Silence in the logs should mean “no event fired,” never “the function ran and chose to do nothing for a reason you can’t see later.”
• Route remediation events to the same on-call channel as everything else. A Slack or PagerDuty notification on every successful auto-remediation, kept low-priority/informational, means your on-call engineer sees the pattern building before the rate-limit alarm has to fire. Silent automation that only escalates once it’s already given up is escalating too late.
• Dashboard the full loop, not just the trigger. A single CloudWatch dashboard showing AGENT_CONNECTIVITY event volume, remediation actions taken, remediation failures, and current ALB target health side by side turns “is our self-healing pipeline doing its job” into a five-second glance instead of a cross-referencing exercise across three consoles during an incident.

Putting it together

The shape that works in practice: EventBridge rule scoped to DRAINING state changes, a Lambda that checks current state and a per-cluster rate limit before acting, a debounce window before the first remediation action, custom CloudWatch metrics on every remediation attempt with an alarm on unusual frequency, and ALB deregistration delays tuned to your actual request duration so the load balancer and ECS agree on when an instance is really gone. None of this is exotic — it’s the same discipline you’d apply to any auto-remediation system, applied specifically to the failure mode ECS just started surfacing for you on EC2. See the ECS container instance health guide and the deregistration reference for the API details this pipeline leans on.

At Enkompass (enkompass.net), this is our home turf. We design AWS architectures that heal themselves at 3 AM so nobody has to. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.