AgentCore On-Demand vs Online Evaluation Guide

Quick answer

Choose on-demand evaluation when you need to score exact spans or traces: a failed interaction, a candidate fix, a curated build-time case, or a controlled ground-truth test. Choose online evaluation when you need continuing production monitoring across sampled or filtered live sessions. Choose batch evaluation when a baseline, regression set, pre/post comparison, or time-bounded audit contains many sessions.

Do not treat these modes as competing score engines. They answer different evidence questions. On-demand asks, “What happened in these cases?” Online asks, “What trend is appearing in live traffic?” Batch asks, “How did this defined population perform?”

Decision table

Decision	On-demand	Online	Batch
Input	Selected span or trace IDs	Sampled or filtered live sessions	Sessions found in a specified CloudWatch Logs location
Best fit	Investigation, evaluator trial, fix validation, build-time case	Ongoing production quality monitoring	Baseline, regression, pre/post comparison, periodic audit
Timing	Requested for chosen interactions	Continuous configuration	Asynchronous job
Population control	You choose exact interactions	Percentage sampling and conditional filters	Defined log source and session population
Ground truth	Reference inputs can include expected responses, assertions, and tool trajectories	Do not assume a live sample has a reference answer	Session metadata can carry expected responses, assertions, and tool trajectories
Main risk	Cherry-picked cases	Biased sampling, drift, missing telemetry, alert fatigue	Mixed populations, stale references, aggregation hiding failures

The current AWS documentation describes built-in and custom evaluators across on-demand and online paths. It separately documents batch evaluation as a service-side job that handles session discovery, span collection, scoring, and result aggregation.

When on-demand is the right first step

Start with on-demand evaluation before production monitoring when the team has not proved that telemetry is complete and interpretable. Select cases that expose the workflow rather than cases that merely make a dashboard look healthy:

1. a normal task with a known answer and expected tool path;
2. an ambiguous request that should trigger clarification;
3. a denied or out-of-policy action;
4. a tool timeout, partial response, retry, and duplicate-call case;
5. conflicting or stale source data;
6. a prompt-injection attempt inside retrieved or tool-returned content;
7. an expected escalation or human handoff.

Use expectedResponse when one answer is meaningfully correct, assertions for session-level observable behavior, and expectedTrajectory when tool presence or order matters. AWS documents exact-order, in-order, and any-order trajectory evaluators as programmatic checks. That makes them useful companions to LLM-based correctness and goal scoring.

When online evaluation becomes useful

Move to online evaluation after you can explain which sessions are eligible, what the sample represents, and what happens after a low score. AWS supports percentage sampling and conditional filters. Define those rules around risk and decision value, not a convenient round number.

Record at least:

• eligible and excluded traffic;
• sample and filter configuration;
• framework, instrumentation, agent, prompt, model, tool, and evaluator versions;
• score distribution by task and risk segment;
• missing or malformed telemetry rate;
• investigation owner, alert threshold, and response deadline;
• false-positive, false-negative, override, and escalation outcomes.

A live score without case review is a weak operating control. Review a fixed number of passing and failing sessions, and compare the monitored population with real incidents, support cases, task outcomes, and human decisions.

When batch is the better comparison

Use batch evaluation for a stable pre/post comparison or regression population. Freeze the session selection rule, reference inputs, evaluator configuration, code and prompt versions, tool fixtures, and failure policy before comparing candidates.

Aggregate averages can hide severe failures. Preserve per-session results and report minimums, percentiles, failure counts, missing data, and risk-segment outcomes. A candidate should not pass solely because many easy sessions offset a small number of unsafe ones.

Telemetry is part of the evaluation

AgentCore Evaluations reconstructs sessions from telemetry. AWS requires supported instrumentation, AgentCore Observability, CloudWatch Transaction Search, and telemetry present in CloudWatch Logs. For on-demand evaluation, AWS advises waiting 2–5 minutes for CloudWatch ingestion before starting.

Check that the service can read the user prompt, agent response, inference activity, and tool inputs and outputs that the chosen evaluator needs. Missing content is not a low-quality answer; it is an evidence failure. Treat incomplete, redacted, truncated, mis-scoped, or mis-correlated spans separately from evaluated agent behavior.

Use the AgentCore framework and telemetry guide to map your Python instrumentation and scope.name. Run the AgentCore evaluation readiness check before creating a production configuration.

Score boundary

AWS describes built-in evaluators as LLM-as-a-judge configurations and also supports custom LLM and code-based evaluators. Ground truth improves the comparison boundary, but it does not make every judgment deterministic. Preserve the evaluator ID, model and inference path when applicable, rubric, inputs, references, raw result, ignored reference fields, code version, and reviewer disposition.

An evaluation score does not establish factual correctness, safety, legal or regulatory compliance, business success, or production readiness. Use deterministic assertions where possible and require domain experts to review high-impact claims and actions.

Frequently asked questions

What is the difference between AgentCore on-demand and online evaluation?

On-demand evaluation scores spans or traces you select. Online evaluation continuously samples and filters live deployed-agent sessions for monitoring. Use the first for controlled investigation and the second for production trends.

Should I use AgentCore batch evaluation instead?

Use batch evaluation when you need an asynchronous regression or audit across multiple sessions. AWS says the service discovers sessions in the specified CloudWatch Logs location, collects spans, scores them, and returns aggregate and per-session results.

Can AgentCore Evaluations use ground truth?

Yes. AWS documents expected responses, assertions, and expected tool trajectories for supported evaluators. Some trajectory matches are programmatic, while correctness and natural-language assertion checks use LLM-as-a-judge scoring.

Does a high evaluation score prove an agent is production-ready?

No. A score depends on the collected telemetry, evaluator, model, rubric, reference data, and sampled cases. Keep deterministic checks, domain review, security tests, failure tests, and accountable release approval.

Official sources

• AWS: AgentCore evaluation types
• AWS: Ground truth evaluations
• AWS: AgentCore Evaluations prerequisites
• AWS: On-demand evaluation prerequisites
• AWS: Evaluators

Source check: August 28, 2026. Recheck evaluation types, evaluator behavior, supported reference fields, ingestion timing, permissions, regions, and account behavior before rollout.