Google Agent Evaluation: Offline vs. Online Monitoring

Quick Answer

Use offline evaluation to compare agent versions against controlled cases or selected historical traces before release. Use online monitoring to sample and asynchronously score live production traces after release.

They are complementary. Offline tests make changes reproducible; online monitors reveal shifts in real traffic, external data, and user behavior that a fixed suite did not anticipate.

The Core Difference

Decision	Offline evaluation	Online monitoring
Data	Fixed cases, historical traces, or sessions	Sampled live traces selected by filters
Timing	Development, release gates, incident analysis	Scheduled production loop
Best use	Regression, version comparison, known failures	Quality drift, segment changes, emerging failures
Main risk	Test suite is too clean or unrepresentative	Noisy scores, privacy exposure, or silent sampling bias
Response	Block release or revise candidate	Alert, inspect traces, reproduce offline, then fix

Google defines a trace as an immutable record of one execution path, including model inputs, responses, and tool calls. A session covers the multi-turn interaction. Use sessions when context retention or conversation flow is part of the acceptance criterion.

When to Use Offline Evaluation

Offline evaluation is the better default when you need to:

• test new system instructions, tools, models, or routing;
• rerun a known set after a code change;
• compare two versions on identical inputs;
• inspect a historical incident without affecting live traffic;
• require a release threshold for high-risk tasks.

The current Google workflow can evaluate stored traces or full sessions with prebuilt or custom metrics. It also supports ad-hoc evaluation while inspecting an individual trace or session.

Keep the dataset versioned. Record who approved reference answers, how cases were sampled, and which failures must block release.

When to Use Online Monitoring

Online monitoring is appropriate after the agent is deployed and exporting the required OpenTelemetry signals. Google documents a scheduled loop that typically:

1. queries matching data from Cloud Trace and Cloud Logging;
2. samples traffic according to the monitor configuration;
3. runs configured metrics through the evaluation service;
4. writes detailed results to Cloud Logging;
5. exports numeric scores to Cloud Monitoring.

Filters and sampling percentage matter. A monitor that observes only short, successful, or low-token traces can report a healthy score while missing the hardest traffic.

Use the Same Quality Contract

The strongest connection between offline and online evaluation is not the dashboard. It is a shared quality contract.

For each important task, define:

• the outcome the user must observe;
• forbidden actions or data access;
• required tool behavior;
• acceptable latency and cost boundaries;
• what evidence supports the final answer;
• the threshold that triggers investigation.

Reuse comparable metrics across development and production where practical, but do not assume the datasets are interchangeable. A fixed regression suite measures repeatability; live traffic measures the distribution you actually receive.

A Production Handoff

1. Run the candidate against the fixed regression suite.
2. Review every critical failure, not only the aggregate score.
3. Deploy to a bounded traffic segment.
4. Confirm telemetry completeness before trusting online scores.
5. Compare the monitored segment with the expected production mix.
6. When an alert fires, save representative traces as a new offline regression cohort.
7. Fix and rerun the expanded suite before broader rollout.

This creates a useful flywheel: production discovers cases, while offline evaluation makes the fix reproducible.

Drift Is a Signal, Not a Diagnosis

Google defines quality drift as an observable performance decrease over time, which can arise from user behavior or external data changes even if the model is unchanged. An alert therefore tells you where to investigate, not what caused the decline.

Check telemetry changes, traffic mix, tool health, data freshness, prompt versions, and evaluator changes before attributing the movement to the agent model.

The simulation and drift monitoring guide provides an alert-to-regression workflow. For the full product lifecycle, see the Google Agent Platform Evaluations GA guide.

Data and Governance Checks

Both offline and online workflows depend on trace content. Before enabling either:

• inventory which prompts, responses, tool arguments, and results are exported;
• redact or exclude secrets and sensitive identifiers;
• configure Cloud Storage, Logging, Trace, and Monitoring access deliberately;
• document retention and deletion;
• limit who can inspect evaluator rationales and production sessions;
• validate region and compliance requirements in the current account.

The product documentation describes mechanisms, not an automatic compliance outcome.

Official Sources

• Run offline evaluations
• Continuous evaluation with online monitors
• Configure quality alerts
• Evaluate your agents

Source check: August 1, 2026. Confirm current telemetry requirements, evaluation cadence, quotas, filters, and metric behavior in the live documentation.