Quick answer
Use RPS or RPM for how often requests arrive, TPM for how many model tokens inference traffic consumes, and CPS for how much simultaneous connection capacity requests hold.
They are complementary. A single long stream counts as one request but holds a connection for its duration and may consume many tokens. A short retry storm may consume many requests with modest tokens. Configure only the metrics supported by the target and justified by the actual bottleneck.
Metric comparison
| Metric | What it counts | Supported targets | Best signal for |
|---|---|---|---|
| RPS | Requests in a second | MCP, inference, HTTP | short bursts and retry storms |
| RPM | Requests in a minute | MCP, inference, HTTP | sustained request throughput |
| TPM | Input plus output tokens per minute | inference only | model capacity and token cost rate |
| CPS | Connection capacity held by requests | MCP, inference, HTTP | streaming and long-lived calls |
A fast request and a long request each count as one request unit. Connection accounting differs because the long request continues to occupy capacity.
How TPM is accounted
AWS says AgentCore Gateway uses a general-purpose tokenizer to estimate the incoming request and deducts that estimate from the token bucket before sending the inference call. When the response returns provider-reported input and output usage, Gateway reconciles against actual total token consumption.
Plan for both estimation and reconciliation. Test large inputs, long outputs, cancellations, provider usage reporting, and throttling near the boundary. TPM cannot be applied to a non-inference target merely because an MCP tool eventually calls a model elsewhere.
Estimate a starting envelope
Use observed percentiles, not averages alone:
- peak active users per bucket;
- peak requests per user per minute;
- input and output token percentiles;
- connection-duration percentiles;
- retry amplification during failures;
- downstream target capacity;
- current service-managed quota;
- a deliberate headroom and degradation policy.
For planning, aggregate RPM is roughly active users multiplied by peak per-user RPM. Aggregate TPM is that request estimate multiplied by a high-percentile token cost. Concurrent demand is roughly requests per second multiplied by high-percentile duration in seconds. These estimates are not AWS quota guarantees; validate them with load tests and production traces.
Avoid four common mistakes
One global request bucket
A shared target-only bucket can let one busy cohort starve everyone else. Combine target and identity dimensions when different groups compete for the same resource.
TPM on every target
AWS documents TPM for inference targets only. Use request or connection limits for MCP and HTTP traffic, plus downstream-native controls.
Ignoring long streams
An RPM limit can look healthy while open connections accumulate. Add a connection dimension when streaming or slow tools can hold capacity.
Setting customer limits above service capacity
Service-managed quotas remain the ceiling. Effective capacity cannot exceed the applicable service quota, even if a customer-defined entry uses a higher number.
Validation matrix
Test the intended bucket under steady load, bursts, large-token prompts, long streams, retries, cancellations, unmatched wildcard cases, and service-quota pressure. Confirm the client handles throttling with bounded backoff rather than synchronized retry storms.
Enable Gateway application logs and rate-limit OpenTelemetry attributes. A limit that never denies may be too loose, mismatched, or bypassed; a limit that constantly denies may be undersized or scoped to the wrong identity.
Frequently asked questions
Which AgentCore targets support RPS and RPM?
AWS says request rate limits apply to MCP, inference, and HTTP passthrough targets.
Which AgentCore targets support TPM?
Token-per-minute limits apply to inference targets. Gateway estimates incoming tokens before dispatch and later reconciles actual input and output usage reported by the provider.
What does CPS protect?
Connection limits protect capacity held by concurrent or long-lived requests. A streaming call occupies a connection slot for the time it stays open.
Should RPS, TPM, and CPS use the same threshold?
No. They measure different resources. Derive each threshold from observed burst rate, token distribution, connection duration, backend capacity, and current service quotas.
Related guides
- AgentCore Gateway rate-limits implementation guide
- OAuth and IAM rate-limit dimensions
- OBO versus M2M agent traffic
- Rate-limit planner
Official sources
Source check: August 8, 2026. Verify current supported targets, metrics, quota values, APIs, accounting, and monitoring before setting production thresholds.