Amazon Bedrock AgentCore Evaluations Framework Guide
On this page
Quick answer
Amazon Bedrock AgentCore Evaluations currently documents Python-only supported instrumentation for Strands Agents, LangGraph, OpenAI Agents, LlamaIndex, Google ADK, and Claude Agent SDK. It also offers a generic path for agents that emit the documented OpenTelemetry GenAI or OpenInference conventions.
Framework name is not the compatibility test. Verify the installed instrumentation library, emitted scope.name, span-kind attributes, prompt and response fields, tool data, session correlation, CloudWatch destination, and evaluator-visible result for your exact agent.
Current framework map
| Framework | Documented instrumentation | scope.name | AWS recommended version at source check |
|---|---|---|---|
| Strands Agents | Built into Strands Agents SDK | strands.telemetry.tracer | Latest |
| LangGraph | OpenTelemetry | opentelemetry.instrumentation.langchain | >= 0.55.0 |
| LangGraph | OpenInference | openinference.instrumentation.langchain | >= 0.1.62 |
| OpenAI Agents | OpenTelemetry | opentelemetry.instrumentation.openai_agents | >= 0.61.0 |
| OpenAI Agents | OpenInference | openinference.instrumentation.openai_agents | >= 1.5.0 |
| LlamaIndex | OpenTelemetry | opentelemetry.instrumentation.llamaindex | >= 0.61.0 |
| LlamaIndex | OpenInference | openinference.instrumentation.llama_index | >= 4.4.1 |
| Google ADK | OpenInference | openinference.instrumentation.google_adk | >= 0.1.13 |
| Claude Agent SDK | OpenInference | openinference.instrumentation.claude_agent_sdk | >= 0.1.3 |
This is a current AWS documentation snapshot, not a package recommendation for every environment. Confirm the live table, SDK compatibility, release notes, and your lockfile before changing dependencies.
What scope.name controls
Every emitted span has a scope that identifies its instrumentation producer. AWS says AgentCore Evaluations uses scope.name to decide whether it can process the span, which framework or convention parser applies, and where to read attributes.
That creates three separate checks:
- Identification: the observed scope exactly matches the named integration or an accepted generic prefix.
- Classification: each agent, inference, and tool span contains the expected identifying attribute.
- Extraction: user prompts, agent responses, tool names, arguments, and results appear in the documented fields and delivery mode.
Do not infer successful evaluation from “traces reached CloudWatch.” Inspect a real session and reconcile the reconstructed evaluation input with the original interaction.
Named framework or generic support?
Use the named framework page when AWS documents one because it defines exact span classification and extraction rules. Generic framework support applies to other scopes beginning with:
opentelemetry.instrumentation.*openinference.instrumentation.*
Generic support is conditional. AWS requires recognized scope names, recognized span identifiers such as gen_ai.operation.name or openinference.span.kind, and content in documented attributes. If you can only emit top-level turn content, AWS documents agentcore.invocation.user_prompt and agentcore.invocation.agent_response as a fallback, but those attributes do not cover inference or tool spans.
AWS excludes transport and infrastructure instrumentation from generic agent parsing, including HTTP client scopes, web frameworks, MCP instrumentation, botocore, and Bedrock Runtime or AgentCore transport scopes. Those spans can be operationally useful without being valid agent-evaluation input.
Unified versus split telemetry
AgentCore Evaluations reads both delivery modes:
| Delivery mode | Where content lives | Operational check |
|---|---|---|
| Unified | Spans and model or tool content remain together in one agent log group | Confirm required attributes remain on each span and IAM/KMS access covers that log group |
| Split | Metadata spans go to aws/spans; content moves to correlated event records in an agent log group | Confirm traceId and spanId correlation, access to both destinations, and complete event records |
AWS recommends unified telemetry and says agents created on or after July 20, 2026 use it by default; older agents use split telemetry unless changed. Unified telemetry requires ADOT 0.18.0 or later according to the current documentation. Changing mode does not move older telemetry, so one review window may contain both storage patterns.
CloudWatch and permissions gate
Instrumentation is only one prerequisite. AWS requires CloudWatch Transaction Search and AgentCore Observability configured for agents hosted on AgentCore Runtime or elsewhere. The evaluation service role needs access to read traces and write results. Custom evaluators can also require Bedrock model invocation, Lambda, or KMS permissions depending on the design.
Use least privilege in the live account. Confirm the exact region, account, log groups, encryption keys, evaluator resources, models, retention, data classification, and cross-region inference policy. AWS notes that built-in evaluation inference may be processed outside the originating region within the applicable routing geography even while stored data remains in the origin region; teams with stricter processing-location requirements should assess the documented custom-evaluator option.
Prove coverage with one trace fixture
Create one fixed session containing:
- a user message with a known identifier;
- one inference result;
- one tool call with safe test arguments and output;
- a controlled tool error or denied action;
- a final response with a known assertion;
- stable session and trace identifiers.
For each framework and instrumentation option, compare the source interaction, exported telemetry, CloudWatch records, reconstructed evaluation input, and evaluation result. Fail the gate if content is missing, mapped to the wrong role, duplicated, truncated, placed on an excluded scope, or joined to the wrong session.
Then use the on-demand versus online evaluation guide to select a rollout mode and the AgentCore evaluation readiness check to capture the evidence boundary.
Frequently asked questions
Which agent frameworks does AgentCore Evaluations support?
AWS currently documents Python support for Strands Agents, LangGraph, OpenAI Agents, LlamaIndex, Google ADK, and Claude Agent SDK, plus a generic path for conforming OpenTelemetry or OpenInference telemetry.
Why does scope.name matter to AgentCore Evaluations?
AWS says the service uses each span’s scope.name to identify the instrumentation library, choose the parsing path, and find framework-specific attributes. A supported SDK alone is not enough if the emitted scope or content is wrong.
Does generic OpenTelemetry make every framework compatible?
No. Generic support requires a recognized scope prefix, identifying attributes on spans, and content in documented fields. AWS also excludes infrastructure scopes such as HTTP clients, web frameworks, MCP instrumentation, botocore, and Bedrock Runtime scopes from generic agent parsing.
Do I need CloudWatch for AgentCore Evaluations?
Yes for the documented workflow. AWS requires observability that exports telemetry to CloudWatch and CloudWatch Transaction Search. The service reads either unified telemetry or split spans and correlated event records.
Official sources
- AWS: Supported agent frameworks
- AWS: Telemetry setup and delivery
- AWS: Generic framework support
- AWS: AgentCore Evaluations prerequisites
- AWS: Cross-region inference for Evaluations
Source check: August 28, 2026. Recheck supported languages, libraries, versions, scope names, telemetry defaults, ADOT requirements, regions, IAM examples, and inference routing before rollout.