Quick answer
WorkSpaces agent access separates computer input from computer vision. Input lets an agent click, type, and scroll. Vision lets it take screenshots and interpret the desktop. AWS requires at least one capability to be enabled, and currently documents a 1280×720 desktop with PNG or JPEG images.
Screenshots are sensitive operational data. They can contain customer records, credentials, notifications, hidden windows, browser history, or regulated fields. Treat capture, model context, local logs, S3 storage, telemetry, and human observation as separate data flows.
Data-flow map
| Surface | Possible content | Control question |
|---|---|---|
| Live desktop | Full application state and other visible windows | Is the image and user profile dedicated to this workflow? |
| MCP screenshot response | Current desktop image sent to the agent | Which model and agent components receive it? |
| Model context | Current and retained prior screenshots | How are images minimized, pruned, logged, and retained? |
| Local agent directory | The AWS sample stores screenshots and logs | Is local disk encrypted, access-limited, and cleaned? |
| S3 screenshot storage | Optional managed audit capture | Who can write, read, retain, export, and delete objects? |
| Human observer stream | Real-time view of the session | Is the observer authorized for every displayed record? |
| CloudTrail and CloudWatch | Tool and operational metadata | What identifiers or payload details are recorded? |
Data classification should happen before the first capture.
Configure the smallest interaction surface
If a forwarded MCP tool can retrieve a known record through a defined schema, it may avoid some screen ambiguity. If the application has a safe API, prefer it for deterministic writes. When computer use is necessary, disable capabilities the task does not require, use a dedicated image and application profile, remove unrelated applications, and restrict clipboard, storage, browser, and network paths.
Coordinate-based actions depend on the latest screenshot dimensions and current UI layout. Recheck the screen before every consequential click. Verify the target label and surrounding context, not only an (x, y) coordinate. Require confirmation when the UI changed, a dialog appeared, or the visual state is uncertain.
S3 screenshot controls
Current AWS documentation says the connecting agent’s credentials upload screenshots and need s3:PutObject; the bucket must also meet the documented WorkSpaces service access requirements. Build a bucket policy for the exact account, Region, stack or workflow where condition keys permit, encryption key, prefix, and retention period.
Enable object-level audit as required, block public access, restrict readers, test deletion, and prevent screenshots from becoming a shadow record system. Masking after capture is too late for data that already entered a model or log.
Human observation and stop
User control mode currently supports:
VIEW_ONLY: observe the desktop;VIEW_STOP: observe and stop the agent;DISABLED: no observer access.
In VIEW_STOP, stopping requires the agent to begin a new session before it can resume. Test what happens to the current application transaction, streaming instance, MCP client, pending tool, and audit evidence. A stop control is not an automatic rollback.
Define who watches, maximum response time, which actions need preapproval, how the operator verifies identity and context, and how work is reconciled after a stop. For high-consequence workflows, approval should occur before the irreversible control is activated.
Visual acceptance suite
Test representative cases:
- display scaling, focus loss, moved windows, scroll position, and delayed rendering;
- pop-ups, consent banners, password prompts, session timeouts, and error dialogs;
- low contrast, small targets, dense tables, disabled controls, and similar labels;
- stale screenshot followed by a click;
- sensitive notification or unrelated window entering the frame;
- model refusal, hallucinated coordinates, repeated input, and partial text entry;
- human stop before, during, and after an application commit;
- screenshot access, lifecycle expiration, deletion, and incident export.
Track accepted task rate, incorrect-action rate, human interventions, ambiguous outcomes, screenshot count, model usage, and remediation time.
Read the MCP and audit guide for identity and telemetry. Use the governance guide and readiness checklist before a live workflow.
Frequently asked questions
What is computer vision here?
It is screenshot capture of the streaming desktop for the agent to interpret.
Are screenshots only stored in S3?
No. They can pass through MCP and model context, and the sample also stores local files. Audit every path.
Does human stop undo the action?
No. It stops the agent session path; the application transaction still needs reconciliation or compensation.
Official sources
- AWS: agent input, vision, screenshots, and user control
- AWS: WorkSpaces Applications MCP tools
- AWS Samples: screenshot behavior
Source check: August 12, 2026. Recheck interaction features, screen formats, observer modes, storage permissions, tool schemas, and sample behavior before use.