The AWS reference build for ambient agents on AgentCore gets you a working S3-to-agent pipeline in an afternoon. Hardening ambient agents for production takes longer, because the failure modes are new: an agent that is retried runs the model twice, an agent that writes to the bucket it watches runs forever, and an agent that nobody caps spends like it found your card. Here is the checklist we use, built on top of the aws-samples/sample-ambient-agent stack.
Hardening ambient agents starts with idempotency
Standard SQS is at-least-once. A visibility timeout that expires mid-turn, a Lambda that crashes after the model call but before the DynamoDB write, and you get a second invocation of a non-deterministic, billable agent. The sample uses Powertools for AWS Lambda idempotency backed by a DynamoDB table. The important part is choosing the key. Key on the job, not on the message, because the same job can legitimately arrive via the API path and the SQS path.
from aws_lambda_powertools.utilities.idempotency import (
DynamoDBPersistenceLayer, IdempotencyConfig, idempotent_function,
)
persistence = DynamoDBPersistenceLayer(table_name=IDEMPOTENCY_TABLE)
config = IdempotencyConfig(
event_key_jmespath='[jobId, turn]', # one record per job turn
expires_after_seconds=3600,
)
@idempotent_function(data_keyword_argument="job",
persistence_store=persistence, config=config)
def run_turn(job: dict) -> dict:
return invoke_agent(job) # model call happens at most once per key
def handler(event, context):
config.register_lambda_context(context)
...
Include a turn counter in the key. A resumed job after a human answer is a new turn and should run; a redelivered message for the same turn should not. Also set the queue’s visibility timeout to at least six times the function timeout, as the Lambda SQS guidance recommends, so a long agent turn is not handed to a second worker while the first is still thinking.
Loop detection, three layers deep
Ambient agents have two kinds of loops, and you need to stop both.
Reasoning loops happen inside a turn: the model calls the same tool with the same arguments over and over. The sample’s max_iterations: 10 in config.yaml maps to LangGraph’s recursion limit (doubled internally, since each iteration is more than one graph step). Keep it low and treat hitting it as an error status, not a silent success.
Event loops happen across turns: the agent writes a summary to invoices/, which fires the signal, which runs the agent. The fixes, in order of reliability:
- Write outputs to a different bucket, or at minimum a prefix no signal matches.
- Tag agent-written objects and have the Signal Processor skip anything carrying that tag or metadata.
- Track a
depthorparentJobIdon job rows and refuse to create a job more than N generations deep.
Lambda has its own recursive loop detection, which helps for some service combinations. Treat it as a smoke detector, not a fire code.
DLQ, alarms and redrive
The sample sets maxReceiveCount: 3 and puts a dead-letter queue behind the job queue. A DLQ without an alarm is a landfill. Alarm on any visible message, and redrive with the native move task once you fix the cause:
aws cloudwatch put-metric-alarm \
--alarm-name ambient-jobs-dlq-not-empty \
--namespace AWS/SQS --metric-name ApproximateNumberOfMessagesVisible \
--dimensions Name=QueueName,Value=ambient-job-execution-dlq \
--statistic Maximum --period 60 --evaluation-periods 1 \
--threshold 0 --comparison-operator GreaterThanThreshold \
--alarm-actions "$ONCALL_SNS_ARN"
# after the fix ships, replay slowly so you don't stampede the model
aws sqs start-message-move-task \
--source-arn "$DLQ_ARN" \
--destination-arn "$JOB_QUEUE_ARN" \
--max-number-of-messages-per-second 2
Before redriving, mark the affected job rows so the UI shows them as retrying, and make sure idempotency records have expired or the replay will be politely ignored.
Sentinel design and envelope versioning
Human-in-the-loop in the sample is elegant: one ask_human tool returns a string prefixed with __HUMAN_INPUT_REQUIRED__::, the orchestrator spots it in the ToolMessage stream, and the runtime returns status: interrupted. It is also a string protocol, which means it will break the day somebody changes it in one place only. Harden it:
- Only trust the sentinel from the tool. Check it on
ToolMessagecontent fromask_human, never on model text. Otherwise a document that contains the sentinel can pause your pipeline on demand. - Version the envelope. Add a
schemaVersionfield so the worker and the container can be deployed independently. - Bound the question. Cap its length and strip markup before it lands in the UI.
ENVELOPE_VERSION = 2
SUPPORTED = {1, 2}
def parse_envelope(raw: dict) -> dict:
v = raw.get("schemaVersion", 1)
if v not in SUPPORTED:
raise ValueError(f"unsupported envelope v{v}")
if raw["status"] not in {"completed", "interrupted", "error"}:
raise ValueError("bad status")
if raw["status"] == "interrupted":
raw["question"] = raw.get("question", "")[:2000]
return raw
An unknown version should go to the DLQ, not get interpreted as “completed.”
Reserved concurrency is your cost circuit breaker
Every concurrent Job Execution worker is a concurrent agent session burning tokens. Reserved concurrency on that one function is the cheapest spend cap you will ever configure, and SQS turns the excess into a backlog instead of a bill.
aws lambda put-function-concurrency \
--function-name "$JOB_EXECUTION_FN" \
--reserved-concurrent-executions 5
# kill switch during an incident: stop all agent runs, keep messages queued
aws lambda put-function-concurrency \
--function-name "$JOB_EXECUTION_FN" \
--reserved-concurrent-executions 0
Pair it with a per-agent token or cost metric. The sample already publishes to a CloudWatch namespace called AmbientAgents (signal matches, jobs enqueued, completed, interrupted, errored). Add tokens per job from the model response and alarm on a per-hour sum. The interrupted rate is a useful health signal too: a sudden spike usually means a prompt regression, not a sudden wave of ambiguous invoices.
TTLs, Guardrails and least-privilege IAM
Conversations carry a 30-day DynamoDB TTL. Decide if that matches your retention policy; if audit needs longer, stream expired items to S3 via DynamoDB Streams before they vanish. Give idempotency records a short TTL and leave interrupted jobs alone, or add a reaper that expires questions nobody answered in N days.
Agents read untrusted documents, so prompt injection is a when, not an if. Attach Amazon Bedrock Guardrails to model calls inside the container, and keep consequential actions behind the Review pattern (APPROVE/REJECT/MODIFY) regardless of what the guardrail says.
Finally, IAM. The worker needs exactly one AgentCore action, on exactly one runtime. The runtime’s execution role needs model invocation on the specific model or inference profile, plus whatever narrow S3 prefixes its tools touch.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "bedrock-agentcore:InvokeAgentRuntime",
"Resource": "arn:aws:bedrock-agentcore:us-east-1:123456789012:runtime/ambient_agent-*"
}]
}
No wildcard on bedrock:*, no s3:* on the documents bucket, and no write access to any prefix a signal watches. That last rule kills the event loop at the permission layer, which is the one place an agent cannot argue its way around.
At Enkompass (enkompass.net), this is our home turf. We build event-driven serverless systems that scale to zero and bill like it, with idempotency, DLQ alarms and concurrency caps baked in from day one rather than bolted on after the first surprise invoice. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.