Build a Robot Agent with Gemini Robotics ER 2

Quick answer

Build a Gemini Robotics ER 2 agent as a layered system:

Sensors and operator input → ER 2 reasoning → bounded tool proposal → authorization and safety adapter → VLA, navigation stack, or robot SDK → tool result → ER 2 response

ER 2 is the high-level reasoning layer. It can interpret text, images, video, and audio; plan multi-step work; call declared tools; and track progress. It should not bypass your robot controller, safety interlocks, permissions, or physical limits.

Use the standard preview endpoint for deliberate analysis and the streaming preview endpoint for a persistent low-latency agent loop. Both are previews, so keep endpoint identity and capabilities configurable.

This architecture guide was checked against Google’s live documentation on July 31, 2026.

Pick the interaction model first

Google publishes two ER 2 endpoint codes with different capability surfaces.

Requirement	Endpoint	Architecture fit
Uploaded media, spatial analysis, structured output, code execution, file or URL context, grounding, caching, or batch work	gemini-robotics-er-2-preview	Request-response planning and review
Continuous camera or audio context, low-latency responses, and function calls in a persistent session	gemini-robotics-er-2-streaming-preview	Reactive Live API agent loop

The standard endpoint uses the Gemini Interactions API and does not support Live API sessions. The streaming endpoint uses the Live API and does not currently support structured output, code execution, file search, URL context, caching, or batch processing.

See the API access guide for the current input formats, context limits, credential restriction, migration, price, and region checks.

Separate the reasoning and control planes

A practical stack has at least five layers:

1. Observation layer: cameras, microphones, telemetry, work orders, operator instructions, and world state.
2. Reasoning layer: ER 2 interprets the current context, proposes a plan, selects tools, and interprets results.
3. Policy adapter: a deterministic service validates identity, authorization, schema, current state, limits, sequence, and freshness.
4. Execution layer: a VLA model, navigation system, PLC interface, or robot SDK performs the approved physical behavior.
5. Safety layer: independent interlocks, collision controls, workspace limits, watchdogs, emergency stop, and human supervision.

The model should only see capabilities the current task and robot are allowed to use. Do not give it a generic shell, network client, or unconstrained motor command.

The Gemini Robotics ER 2 model guide explains the distinction between the standard reasoning endpoint, the streaming endpoint, and the motor-execution layer.

Design bounded robot tools

Each tool should represent one meaningful, auditable capability. Prefer:

• inspect_workspace(camera_id)
• navigate_to(robot_id, waypoint_id, max_speed)
• pick_object(robot_id, object_id, grasp_id)
• place_object(robot_id, target_id, placement_id)
• request_human_review(reason_code, evidence_ref)
• safe_hold(robot_id, reason_code)

Avoid raw joint arrays, arbitrary coordinates, free-form command strings, or a single control_robot tool.

For every tool, define:

• an exact JSON schema and enumerated identifiers;
• which robot, task, zone, and operator role may invoke it;
• allowed states and prerequisite observations;
• force, velocity, duration, distance, and workspace limits;
• a request or idempotency key;
• timeout, cancellation, retry, and safe-stop behavior;
• the evidence returned to the model;
• what requires human confirmation.

Treat descriptions as guidance for the model, not enforcement. The adapter must reject invalid calls even if the schema is well formed.

Build a standard interaction

Use gemini-robotics-er-2-preview when the agent can operate as a planned request-response workflow.

1. Capture or upload the authorized image, video, audio, and task context.
2. Create an interaction with a bounded instruction and relevant tools.
3. Ask for a structured observation, plan, coordinate, progress label, or tool proposal.
4. Validate the response against the current task and robot state.
5. Execute only an allowed action through the policy adapter.
6. Return the tool result in the continued interaction.
7. stop, recover, escalate, or continue based on fresh evidence.

Structured output can help constrain a plan or observation on the standard endpoint. It does not establish physical validity. Convert image coordinates into robot coordinates through a calibrated perception and control path, then validate reachability and collision constraints.

Build a streaming interaction

Use gemini-robotics-er-2-streaming-preview when the application needs a stateful Live API session.

Google’s streaming documentation specifies a bidirectional WebSocket session, raw 16-bit PCM audio at 16 kHz little-endian, JPEG camera frames at no more than one frame per second, and text output.

The event loop should:

1. open a session with the streaming model and the smallest necessary tool allowlist;
2. stream authorized text, audio, and camera frames;
3. send a text or audio prompt that asks the model to reason over the current context;
4. receive a text response or function call;
5. validate the call through the policy adapter;
6. execute the approved action through the robot SDK;
7. return the tool result to the Live API session;
8. repeat only while the session, task, and safety state remain valid.

Google currently supports only blocking function calls for the robotics Live API, and physical tools must declare "behavior": "BLOCKING". Blocking serializes the model’s call loop; it does not provide emergency stopping, collision safety, or transaction integrity.

Make physical actions stateful and idempotent

A network retry must not turn one requested movement into two physical movements. Before execution:

• attach a unique action ID;
• record the expected robot and task state;
• compare the live state with that expectation;
• reserve the affected robot and workspace;
• reject a completed or in-flight duplicate;
• execute with a bounded deadline;
• record the final controller result;
• require a new observation before an uncertain retry.

If the connection drops after dispatch but before a tool response returns, query the controller’s action ledger. Never assume the action failed and repeat it blindly.

Orchestrate more than one robot carefully

Google’s launch demonstrates ER 2 coordinating heterogeneous robots. That is evidence that the model can reason across tools, not permission to expose a fleet through one broad command.

Give every robot:

• a stable identity and capability list;
• a separately scoped credential and tool namespace;
• current location, payload, health, and reservation state;
• allowed zones and maximum limits;
• collision-aware scheduling and shared-workspace locks;
• a deterministic safe-hold action;
• an operator responsible for takeover.

The orchestrator can choose which bounded capability fits the plan. A fleet manager and local controller still arbitrate physical access.

Put failure and recovery in the architecture

Handle these conditions explicitly:

• model timeout or unavailable preview endpoint;
• malformed or unauthorized tool call;
• stale camera frame or conflicting telemetry;
• robot controller error or partial completion;
• unexpected contact, obstruction, or person entering the zone;
• tool response lost after execution;
• repeated progress estimate without state change;
• operator takeover or emergency stop;
• model alias, capability, price, quota, or terms change.

The default recovery should be a safe hold plus evidence for review. Retrying is a policy decision with a limit, not a reflex.

For evaluation metrics and stop conditions, use the safety and progress guide.

Secure and observe the agent

Google’s robotics API documentation says unrestricted API keys receive a 403 Forbidden response. Restrict the key, keep it server-side, and isolate environments.

Log enough to reconstruct a decision:

• model code, endpoint, and version or alias available at request time;
• interaction or Live API session ID;
• task, robot, and policy version;
• references to authorized inputs rather than unnecessary raw media;
• proposed, rejected, approved, started, and completed tool calls;
• validation result and robot state;
• latency, timeout, retry, stop, and recovery;
• human authorization, intervention, and outcome.

Minimize personal data. Google’s robotics terms address identifiable voices, imagery, and likenesses and require sufficient notice and consent around Robotics Models. Limit camera and microphone scope, retention, and review access.

Pre-pilot checklist

• The task is not safety-critical or prohibited by the model card.
• Standard versus streaming endpoint choice is documented.
• API keys are restricted and stored outside clients and source control.
• Every physical tool has a narrow schema and explicit authorization.
• Physical streaming tools declare blocking behavior.
• Policy checks and safety controls work without the model.
• Duplicate, timeout, cancellation, and connection-loss cases are tested.
• Safe hold, emergency stop, and human takeover are exercised.
• Evaluation covers occlusion, clutter, people, sensor loss, and unexpected state.
• Notice, consent, minimization, retention, and deletion are implemented.
• Price, quota, region, preview lifecycle, and rollback are monitored.

Frequently asked questions

What role should Gemini Robotics ER 2 play in a robot agent?

Use ER 2 as the high-level reasoning and orchestration layer. It interprets multimodal context, plans work, selects bounded tools, and tracks outcomes while a robot API, navigation stack, or lower-level VLA performs motor execution.

Which Gemini Robotics ER 2 endpoint should a robot agent use?

Use gemini-robotics-er-2-preview for deeper non-streaming analysis, structured output, code execution, grounding, or batch work. Use gemini-robotics-er-2-streaming-preview for a persistent low-latency Live API session with blocking function calls.

Should Gemini Robotics ER 2 send commands directly to actuators?

No. Put an authorization and safety adapter between the model and the robot SDK. Validate the tool name, schema, state, limits, permissions, idempotency, and timeout before sending an approved command to a lower-level controller.

Why must physical tools use blocking behavior?

Google’s robotics streaming guide supports only blocking function calls and requires physical-action tools to declare BLOCKING behavior. The client completes the call and returns a tool response before the model continues its action loop.

Can one Gemini Robotics ER 2 session control multiple robots?

Google demonstrates multi-robot orchestration, but an application still needs explicit robot identities, separate permissions and state, collision-aware scheduling, and independent safety controls. Do not expose an undifferentiated move tool across a fleet.

What should I log for a Gemini Robotics ER 2 agent?

Log the model and endpoint, interaction or session ID, bounded input references, proposed and approved tool calls, state checks, latency, tool results, stop reasons, recovery, and human intervention while minimizing personal and sensitive data.

Bottom line

Use ER 2 to reason and orchestrate, not to replace a robot controller. A narrow tool surface, deterministic policy adapter, independent safety system, idempotent execution, explicit recovery, and human takeover turn a model demo into an architecture you can evaluate responsibly.

Official sources

• Google: Introducing Gemini Robotics ER 2
• Gemini Robotics ER API overview
• Gemini Robotics ER 2 orchestration guide
• Gemini Robotics ER 2 streaming guide
• Gemini Robotics ER 2 model card
• Gemini API Additional Terms of Service