This morning we covered the service-observability half of the launch. This afternoon is the half that keeps AI teams up at night: CloudWatch Omni agent observability. Classic APM tells you an endpoint took four seconds. It does not tell you the agent picked the wrong tool, hallucinated a refund policy, and then politely apologized for it. Omni’s agent side is built for that second problem, and the good news is that getting data into it is mostly standard OpenTelemetry work.
The feature set, from the AWS launch post for agentic workloads: 17 built-in evaluators, third-party evaluators such as AutoEval and DeepEval, a Playground, Experiments, Prompt Management, Session Explorer, Agent Topology, Compare mode and Ask Assistant. Supported frameworks include LangChain, LangGraph, CrewAI, the OpenAI SDK, Strands, the Vercel AI SDK and AgentCore, across Python and TypeScript. Instrumentation rides on OpenInference and ADOT.
Why CloudWatch Omni agent observability needs two libraries
People trip on this, so let’s be precise. OpenInference is a set of OpenTelemetry instrumentations and semantic conventions for LLM applications. It knows what an LLM call, a tool call, a retriever and an agent step look like, and it records prompts, completions, token counts and tool arguments as span attributes. ADOT, the AWS Distro for OpenTelemetry, is the SDK and exporter plumbing that gets those spans to AWS with SigV4 auth and sensible AWS resource detection.
OpenInference decides what gets captured. ADOT decides where it goes. You want both.
Instrumenting a LangGraph agent
LangGraph sits on LangChain’s callback system, so the LangChain instrumentor covers it. Install the pieces:
pip install aws-opentelemetry-distro \
openinference-instrumentation-langchain \
langgraph langchain-awsThen register the instrumentor before you build the graph, and tag each conversation with a session ID so Session Explorer can stitch multi-turn runs together. OpenTelemetry baggage is the clean way to carry it across spans:
from openinference.instrumentation.langchain import LangChainInstrumentor
from opentelemetry import baggage, context
LangChainInstrumentor().instrument()
from my_agent.graph import build_graph
graph = build_graph()
def handle(user_message: str, session_id: str):
token = context.attach(baggage.set_baggage("session.id", session_id))
try:
return graph.invoke({"messages": [("user", user_message)]})
finally:
context.detach(token)Strands is simpler still: it emits OpenTelemetry spans for agent loops, model calls and tool invocations natively, so running it under the ADOT auto-instrumentation wrapper is usually enough. Check the Strands Agents docs for the current telemetry options. If you host on AgentCore, its runtime observability wiring is covered in the AgentCore documentation.
Shipping traces: containers, ECS and Lambda
For a container or ECS task, the least-drama path is ADOT auto-instrumentation exporting OTLP directly to CloudWatch’s OTLP endpoints. Set these on the task definition:
OTEL_PYTHON_DISTRO=aws_distro
OTEL_PYTHON_CONFIGURATOR=aws_configurator
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces
OTEL_RESOURCE_ATTRIBUTES=service.name=support-agent,deployment.environment=prod,team=cxAnd start the process through the wrapper:
CMD ["opentelemetry-instrument", "python", "-m", "support_agent.server"]The task role needs permission to write traces, and your account needs CloudWatch Transaction Search enabled for spans sent to the OTLP endpoint to be indexed. Verify both against the CloudWatch docs for your Region. If you would rather keep a collector in the loop for sampling or redaction, run the ADOT collector as a sidecar and point the SDK at localhost:4318 instead.
On Lambda, attach the ADOT Python layer for your Region and architecture (ARNs are listed in the ADOT docs) and set the exec wrapper:
aws lambda update-function-configuration \
--function-name support-agent \
--layers "<ADOT Python layer ARN for your Region>" \
--environment "Variables={AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument,OTEL_SERVICE_NAME=support-agent}"One Lambda-specific gotcha: agents often stream, fan out tool calls and run long. Make sure spans flush before the invocation ends, or you will see truncated traces that look like the agent quit halfway through a thought.
Name things so Agent Topology makes sense
Agent Topology draws your supervisor, sub-agents, tools and model calls as a graph. It can only be as readable as the names you give it. Give every deployable agent its own service.name, name tools after what they do (lookup_order, not tool_3), and keep the same names across dev, staging and production, using deployment.environment to tell them apart. TypeScript teams on the Vercel AI SDK or the OpenAI SDK follow the same rules with the Node flavors of OpenInference and ADOT. Mixed-language estates end up on one coherent map, which is more than most hand-rolled LLM logging ever managed.
Privacy before volume
OpenInference captures prompts and completions by default. That is exactly what makes evaluation possible, and exactly what makes your security team twitch. Decide before production what gets recorded. OpenInference supports configuration to hide inputs and outputs, and a collector sidecar gives you a place to redact attributes. Customer PII in a trace store is still customer PII.
Evaluators: turning traces into quality signals
Once traces flow, Omni’s 17 built-in evaluators score them on dimensions such as coherence, helpfulness, faithfulness and routing correctness. Routing correctness is the one multi-agent teams will love: it flags when the supervisor handed a billing question to the shipping agent. Faithfulness is your hallucination canary for RAG flows. If you have already standardized on AutoEval or DeepEval, Omni supports those as third-party evaluators, so you do not have to rewrite your test suites.
A practical rollout:
- Start with two or three evaluators on a sampled slice of production sessions, not everything. Evaluators cost money and attention.
- Use Session Explorer to read the worst-scoring sessions end to end. Agent Topology shows which sub-agent or tool is involved.
- Promote recurring failures into an Experiment dataset so they become regression tests.
Compare mode, Playground and Prompt Management
This is where Omni stops being a dashboard and starts being a workflow. Store prompt versions in Prompt Management, iterate in the Playground, then run an Experiment with the old and new prompt (or model, or tool config) against the same dataset. Compare mode lays the runs side by side with evaluator scores, latency and token usage. “The new prompt feels better” becomes “faithfulness went up, cost went up a bit, routing stayed flat.” Ship on that, not on vibes. Ask Assistant lets you query all of it in plain language when you do not feel like building a filter.
Local mode, Cloud Login and the IDE extensions
The free VS Code and Kiro extensions bring agent tracing into the editor. By default they run in a local, on-disk mode: traces from your dev runs stay on your machine, which is perfect for iterating on a prompt without shipping half-baked experiments to a shared account. Opt into Cloud Login when you want local runs to land alongside production data, for instance to compare a branch against last week’s production sessions in Compare mode.
Our rule of thumb: local mode for inner-loop hacking, Cloud Login for anything you plan to show someone else. Same instrumentation either way, which is the quiet payoff of standardizing on OpenTelemetry.
At Enkompass (enkompass.net), this is our home turf. We pressure-test brand-new AWS services inside real client workloads the week they drop, so you get the upside without volunteering as the cautionary tale. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.