Tokens per Minute vs Requests per Minute: Which LLM Rate Limit Should You Enforce?

What each limit measures

An RPM limit counts calls within a time window, regardless of what each call contains. The count is known the moment a request arrives, so enforcement is deterministic and simple. The weakness is that a request is a poor proxy for load. A short classification prompt and a prompt that fills a long context window count the same.

A TPM limit counts tokens, typically prompt plus completion. The prompt side can be estimated before the call by running the same tokenizer the model uses, or an approximation of it. The completion side is unknown until the model finishes, and it is bounded only by the maximum output setting on the request. That means TPM enforcement works on an estimate first and a measured value later, and the gap between the two has to be reconciled.

Providers express quotas in their own terms, and how they treat input tokens, output tokens and reserved maximum output can differ. Internal limits should be designed against each provider's counting rules rather than assuming one model of accounting applies everywhere.

RPM and TPM side by side

The two limits protect against different failure modes. The table compares them on the dimensions that usually decide the design.

• Measurement accuracy: RPM: exact at request time. TPM: estimated before the call, corrected after it from actual usage.
• Quota exhaustion: RPM: misses a few long-context calls that consume a large share of a token quota. TPM: tracks the resource that token-based provider quotas actually meter.
• Abuse and runaway loops: RPM: stops high-frequency loops and retry storms quickly. TPM: stops oversized prompts and large completions, but only as fast as usage is counted.
• Implementation complexity: RPM: a counter per consumer and window. TPM: tokenization, output reservation, reconciliation and attribution.
• Fairness across consumers: RPM: equal call counts can mean very unequal consumption. TPM: allocates by volume, which better reflects what each consumer uses.
• Streaming, retries, variable context: RPM: retries inflate counts, streaming has little effect. TPM: needs mid-stream accounting, retry policy and handling for long contexts.

Where each limit is sufficient and where it fails

RPM alone is reasonable when payloads are small and uniform, such as short classification or extraction calls, and when the main risk is request frequency. It is also a sensible first control for a new workload where token accounting is not yet in place.

RPM alone fails when context length varies widely. One application sending long documents can consume most of a shared token quota while staying well under its request allowance, and other consumers then see errors they did not cause. It also does nothing about a single call configured with a very large maximum output.

TPM alone fails against rapid small calls. A loop sending thousands of tiny requests may stay within a token budget while still exhausting a provider's request-rate allowance or overwhelming a downstream tool. TPM also depends on estimation quality, so a tokenizer mismatch or an unreserved completion can let real usage drift from counted usage.

Streaming complicates TPM because completion tokens accrue while the response is in flight. Teams usually reserve an estimated amount at the start, count tokens as they stream, and settle the difference when the stream ends or is cut off. Retries need a deliberate rule: a failed call may or may not have consumed provider quota, and a retry should be attributed to the same consumer so it is not counted as fresh, unrelated demand.

A decision framework for choosing RPM, TPM or both

Start from the workload. If payload size is uniform and bounded, RPM may be enough. If prompt or completion size varies by an order of magnitude, or if any consumer can supply long inputs, TPM is required. If you share provider quotas across teams, or tools downstream are sensitive to call rate, use both.

A layered design typically has three tiers. Per-consumer TPM limits allocate token volume fairly across users, applications and teams. Per-consumer RPM limits cap call frequency and contain loops. Aggregate limits per model or provider sit below the provider's own quota, so internal enforcement triggers before the provider rejects traffic.

Choose attribution dimensions based on who needs to be held accountable: user for abuse, application for budgeting of capacity, team for shared pools, model for provider quota alignment, and tool for downstream protection. More dimensions give finer control but multiply the state you have to maintain. Token bucket and sliding window counters are common ways to implement the windows. Token buckets allow short bursts within a sustained rate, while sliding windows give a stricter view of the last interval.

Define scope and limit-breach behavior

Every limit needs an explicit scope and an explicit outcome at the limit. Decide both before deployment, and make the outcome configurable per policy rather than fixed in code. The common outcomes are:

• Reject: Return a clear error immediately. Simple and predictable, but pushes retry logic onto callers.
• Queue: Hold the request until capacity frees. Suits batch and background work, not interactive use.
• Throttle: Slow the rate of admitted calls or reduce allowed output size so traffic continues at a lower rate.
• Escalate: Route to a person for approval when the call is unusual or consequential enough to warrant review.

Where to enforce, and how Trussed enforces rate limits

Enforcement in client code is easy to ship but easy to bypass, and each application implements its own accounting. A proxy gives one place to count, but only for traffic routed through it. Enforcement in the execution path applies policy at runtime to every interaction and tool call, regardless of which application or model is involved. For a regulated enterprise, the evaluation criteria are whether the limit applies across models and providers, whether it can be attributed per user, application, team, model and tool, whether breach behavior is configurable, and whether the system records what limit applied, what was measured, and what decision was taken.

Trussed enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants. Rate limit is one of the policy actions, alongside allow, redact, block and escalate for human approval, and it applies to individual interactions and tool calls. Policy is enforced at runtime, not reviewed after the fact, and 100+ models run under one policy plane.

Enforcement produces audit-ready evidence, mapped to HIPAA, NAIC, NYDFS, EU AI Act and NIST, so a regulated enterprise can show an auditor which limits applied and how breaches were handled. The deployment is self-managed in the customer cloud, with configurable retention and data residency.