Quick answer
Gemini Robotics ER 2 can estimate task progress and locate meaningful moments in physical-world video. Those capabilities can help a robot agent decide whether to continue, wait, retry, ask for help, or report completion. They do not make the model a safety system.
Use progress tracking as one fallible signal inside a larger control structure:
- define an observable completion condition;
- compare the model’s estimate with robot state and deterministic sensors;
- block unsafe or unauthorized transitions independently;
- stop safely when observations conflict or confidence is inadequate;
- require a trained human for ambiguous or high-impact cases.
This guide was checked against Google’s current launch materials, API documentation, streaming guide, terms, and model card on July 31, 2026.
What progress tracking means
Google describes two related ER 2 capabilities:
- Progress classification: estimate how far a physical task has advanced using five bands: 0–20%, 20–40%, 40–60%, 60–80%, and 80–100%.
- Moment finding: identify where a requested event occurs in a video, such as the instant a component is seated or a container reaches a target.
These are perception and reasoning outputs. They can support status reporting, recovery, quality review, or tool selection, but they cannot independently establish that an assembly is correctly torqued, a workspace is clear, a medication is appropriate, or a machine is safe to restart.
For model IDs, access paths, and endpoint capabilities, use the Gemini Robotics ER 2 model guide and API access guide.
How to read Google’s published results
Google reports a 57.4% progress-classification accuracy for ER 2 in its launch article. It also reports 91.3% moment-finding accuracy, a 0.96-second mean absolute distance, and a 4× speed improvement for its evaluation setup.
These are provider-reported results, not a general service-level objective. Before relying on them, ask:
- Which tasks, environments, camera angles, and event definitions were evaluated?
- Does the test distribution match your robot, lighting, clutter, tools, clothing, and people?
- Is the costly error an early transition, a late transition, a missed event, or a false completion?
- Does a percentage band map cleanly to a permitted physical action?
- What happens when the video is occluded, delayed, stale, or inconsistent with robot telemetry?
A useful internal evaluation reports the confusion matrix and operational consequences, not only aggregate accuracy.
Progress is not success
An 80–100% prediction may mean the visible motion looks nearly complete. It does not prove:
- the correct object or part was used;
- hidden fasteners or connections are complete;
- force, torque, temperature, pressure, or electrical state is within limits;
- the result meets a quality specification;
- a person is outside the hazard zone;
- the next movement is authorized;
- the task completed without damage.
Define success as a conjunction of evidence. For example, an assembly step might require the correct work order, matching part identity, robot pose within tolerance, torque telemetry within range, an unbroken safety interlock, and a progress observation. If any required signal is absent or conflicting, the state machine should not advance.
A practical evaluation matrix
Build a representative test set before a supervised pilot. Include routine examples and deliberately difficult cases.
| Evaluation area | Example measure | Why it matters |
|---|---|---|
| Progress bands | Per-band precision, recall, and confusion | Shows whether the model skips or stalls workflow states |
| Moment finding | Early and late error distribution | Reveals whether an event trigger arrives before a safe condition |
| Completion | False-completion rate | Measures the highest-risk progress error directly |
| Intervention | Human takeovers per task | Exposes operational burden hidden by average accuracy |
| Recovery | Safe recovery rate and time | Tests what happens after disagreement or uncertainty |
| Stop behavior | End-to-end stop latency | Confirms the full control path, not just model response |
| Distribution shift | Performance by lighting, clutter, tool, and camera | Finds brittle environment dependencies |
| Sensor failure | Behavior with stale, missing, or conflicting signals | Verifies fail-safe handling |
| Human entry | Detection-to-safe-state outcome | Tests an important real-world interruption |
| Privacy | Identifiable-media exposure and retention | Checks whether collection controls work in practice |
Set acceptance thresholds per task. A low-impact reporting workflow and a robot movement near people should not share the same thresholds or escalation rules.
Keep safety controls independent
Google’s model card says the Robotics Models should not be used for safety-critical applications or in settings such as healthcare and transportation where malfunction could foreseeably cause death, injury, or property damage.
For other controlled experiments, keep these functions outside ER 2:
- emergency stop and protective stop;
- collision avoidance and safety-rated monitored speed;
- force, velocity, position, duration, and workspace limits;
- interlocks, access control, geofencing, and human-presence sensing;
- actuator watchdogs and communication-loss behavior;
- deterministic checks for tool, load, pose, and machine state;
- an allowlist of robot actions and parameters;
- human authorization for irreversible or high-impact actions.
The model may request an action. A constrained adapter decides whether the request is syntactically valid, currently authorized, compatible with machine state, and within safety limits. A separate controller executes it.
Design the state transition
Treat every progress estimate as an observation attached to time, task, camera, and robot state. A robust transition can follow this sequence:
- receive a new observation and reject stale input;
- ask for a bounded progress label plus evidence;
- compare the label with telemetry and the prior state;
- require repeated or independent confirmation where appropriate;
- evaluate tool authorization and safety gates;
- transition, hold, recover, or escalate;
- record the evidence, decision, tool call, result, and operator intervention.
In a streaming session, camera frames alone do not continuously force the model to reason. Google’s guide describes sending a text or audio prompt to trigger a response and suggests a periodic prompt for progress checks. Treat that cadence as an application heartbeat, not a guaranteed control loop. The robot’s own watchdog must handle delayed or missing model responses.
The robot-agent architecture guide explains how to place that decision path between ER 2 and the robot SDK.
Privacy in physical spaces
Robot video and audio can capture faces, voices, bystanders, screens, badges, documents, and private conversations. Google’s robotics terms require sufficient notice and consent before identifiable people interact with or enter the area around a Robotics Model, along with reasonable data-minimization efforts.
Translate that into operational controls:
- point cameras away from unrelated people and sensitive work;
- define a marked capture zone with notice and consent;
- blur faces where practical;
- disable or narrow microphones when audio is unnecessary;
- minimize uploads, logs, samples, and retention;
- separate evaluation media from normal application logs;
- restrict review access and document deletion procedures;
- test incident response for unintended capture.
Privacy review belongs before live trials, not after the first useful dataset has been collected.
Stop conditions to define in advance
Move to a safe hold or stop when:
- a person enters a restricted area;
- sensor data is absent, stale, or contradictory;
- progress regresses unexpectedly;
- an object, tool, or target cannot be identified;
- the proposed tool or arguments are outside the allowlist;
- model, network, tool, or controller latency exceeds the limit;
- the robot reports an error, unexpected contact, or state mismatch;
- repeated recovery fails;
- an operator requests takeover;
- a privacy or authorization boundary is crossed.
The stop path should work without a successful model call.
Frequently asked questions
Can Gemini Robotics ER 2 tell whether a task is complete?
It can classify task progress from physical-world observations and find moments in video, but its output is a prediction. Validate it against your own tasks, camera views, tools, people, and failure modes before using it in an operational workflow.
What progress levels does Gemini Robotics ER 2 use?
Google describes five 20-point progress bands: 0 to 20, 20 to 40, 40 to 60, 60 to 80, and 80 to 100 percent. A band is not proof that the task is safe, correct, or ready for the next physical action.
Is Gemini Robotics ER 2 safe for safety-critical applications?
No. Google’s model card says not to use the Robotics Models for safety-critical applications, including healthcare, transportation, or other settings where a malfunction could foreseeably cause death, injury, or property damage.
Do Google’s reported progress scores certify my robot?
No. Google’s reported evaluations describe model performance on selected tests. They do not certify your robot, sensors, actuators, workspace, process, personnel, or safety controls.
What should happen when progress confidence is low?
Stop or hold the workflow in a safe state, preserve the relevant evidence, and route the decision to a trained human or a deterministic recovery procedure. Do not let the language model silently guess and continue.
How should I handle people captured by robot cameras?
Provide sufficient notice and consent before identifiable people interact with or enter the area, minimize collection, avoid identifiable people where practical, consider face blurring, restrict access and retention, and verify the current robotics terms.
Bottom line
ER 2 can add a useful progress and moment-finding signal to a robot workflow. Validate that signal at your operating point, combine it with deterministic evidence, keep machine safety independent, and make safe hold plus human takeover the default response to ambiguity.