Quick Answer
Use Google Agent Simulation before release to generate scenarios, role-play multi-turn users, and inject controlled tool behavior or failures. Use online evaluation after release to detect quality drift in sampled production traces.
The durable workflow is simulate, deploy narrowly, monitor, investigate, reproduce, and rerun. Synthetic success does not prove production reliability, while a drift alert does not diagnose its own cause.
What Agent Simulation Does
Google documents simulation as a way to build an evaluation suite without existing production data. The process uses an immutable agent version and proceeds in two stages:
- generate evaluation cases from agent instructions and tool definitions;
- use a simulated user to run multi-turn interactions and capture traces.
Each case can include a starting prompt and a hidden conversation plan describing the user’s goal and how the simulated user should react. A maximum-turn setting prevents runaway conversations.
The generated trace remains the evidence to inspect: model inputs, responses, tool calls, and the conversation path.
User and Environment Simulation
User simulation stresses conversation behavior: clarification, changes of mind, incomplete information, conflicting constraints, and multi-step goals.
Environment simulation tests the world around the agent. Google documents intercepting tool calls to inject mocked data or behaviors such as HTTP 503 errors and latency spikes without touching production backends.
Use environment simulation for:
- unavailable or slow tools;
- malformed tool responses;
- expired credentials or denied permissions;
- inventory, schedule, or price changes;
- partial writes and retry behavior;
- stale data and conflicting sources.
Do not let synthetic tools mutate real systems. Confirm sandbox boundaries and use test credentials and non-production data.
Build a Useful Scenario Matrix
Cross user intent with environment state instead of generating a large undifferentiated set.
| User dimension | Environment dimension | Expected behavior |
|---|---|---|
| Clear request | Healthy tools | Complete the task and cite the result |
| Ambiguous request | Healthy tools | Ask the minimum necessary clarification |
| Unauthorized request | Tool appears available | Refuse or escalate before acting |
| Changed goal | Partial prior state | Reconcile state without duplicate side effects |
| Urgent request | Timeout or 503 | Report uncertainty, retry safely, or hand off |
| Adversarial instruction | Untrusted tool content | Preserve system and data boundaries |
Keep high-risk cases hand-authored. Generated scenarios are valuable for breadth, but they can reproduce the assumptions and blind spots of the generator.
What Quality Drift Means
Google defines quality drift as an observable decrease in agent performance over time. It may be caused by changing user behavior, external data, tool behavior, or complex prompt interactions even when the underlying model remains unchanged.
Online monitors sample matching production traces, apply configured metrics, and export scores to Cloud Monitoring. Quality alerts can trigger when an aggregate or individual metric crosses a threshold.
An alert is a starting point. Investigate:
- which traffic segment changed;
- whether telemetry or filters changed;
- agent, model, prompt, or tool versions;
- tool error and latency rates;
- source-data freshness;
- evaluator or rubric changes;
- whether the score movement is confirmed by human review.
Turn Drift Into a Regression Test
When a quality alert is credible:
- preserve the affected trace and relevant configuration under the approved data policy;
- remove or mask sensitive data;
- identify the user goal, expected outcome, and failure category;
- create a minimal offline case that reproduces the behavior;
- add adjacent simulated cases for the same failure mode;
- fix the instructions, tools, routing, or environment handling;
- rerun the fixed suite before redeployment;
- monitor the affected segment separately after release.
This prevents a one-off production patch from disappearing without a durable test.
The offline versus online guide explains the measurement handoff. Use the adaptive rubrics guide when a generated evaluator is part of the alert.
Alert Design
Start with a small number of actionable alerts. Pair each threshold with an owner, investigation window, trace query, severity, and rollback or containment path.
Aggregate scores can hide rare critical failures. Monitor safety, task success, and tool-use quality separately where they carry different consequences. Calibrate thresholds on a stable baseline and review false alarms before paging an on-call team.
Boundaries
Simulation does not reproduce every real user, language, tool, data source, or adversarial condition. Online monitoring observes only the traces selected by telemetry, filters, and sampling. Neither mechanism proves complete safety or compliance.
Verify current regional availability, quotas, evaluator cost, telemetry requirements, storage, access, retention, and incident controls in the live Google Cloud account.
Official Sources
- Simulate agent behavior
- Agent evaluation overview
- Continuous evaluation with online monitors
- Configure quality alerts
Source check: August 1, 2026. Validate the current simulator, monitor, alert, telemetry, and data-handling behavior before production use.