Kafka Quotas: How Brokers Contain Noisy Neighbors

One analytics backfill doubles its produce rate. Minutes later, every application sharing the cluster sees slower requests—even though most of them haven't changed a thing.

A broker, the Kafka server that handles requests and owns partition data, can identify the authenticated user and client ID, the caller-supplied label in the request header, behind that traffic. It cannot know that checkout matters more than a backfill. That's the noisy-neighbor problem: one workload consumes a disproportionate share of a shared resource.

My opinion: define fairness and tenant identity before an incident. Kafka quotas can enforce organizational intent, but they cannot infer it.

By the end of this post, you'll be able to:

• tell which response versions contain throttle_time_ms;
• distinguish bandwidth, request, connection, and controller quotas;
• read quota precedence without thinking in ZooKeeper paths;
• diagnose delay without mistaking a protocol field for tenant identity.

throttle_time_ms is versioned, not universal

Kafka represents a throttle delay as throttle_time_ms, a signed 32-bit integer in the response body. A value of 200 tells the client that the broker calculated a 200 ms delay. It doesn't identify the tenant, name the violated quota, or prove which resource ran hot.

An API version is a particular request and response schema revision. Clients usually discover the broker's supported range through ApiVersions, then choose a compatible version for each API. That negotiated version decides whether the response can carry a throttle field at all.

KIP-13 added the field to Produce and Fetch. KIP-124 extended request quotas across client APIs, which required another set of response-version bumps.

The table captures the quota-era version changes and two later APIs that included the field from their first version. The Kafka 4.3 protocol schemas contain many newer APIs, so clients should decode the negotiated schema rather than assume a field from the API name.

Not every response carries it. SaslHandshake and internal cluster-action responses such as LeaderAndIsr, StopReplica, UpdateMetadata, and ControlledShutdown omit the field in the Kafka 4.3 schemas. Connection limiting may also happen before Kafka has an authenticated request to answer.

Field presence also predates KIP-219's immediate-response behavior. Kafka 4.3.1's Java client adds its own pause only for Produce response v6+ and Fetch response v8+, as defined by shouldClientThrottle in the response classes. With an older negotiated version, the field can exist while the client does not add another delay; broker-side channel muting still enforces the quota.

This distinction matters during diagnosis. A missing field can mean “this response version has no field,” not “the broker applied no quota.” A present field does not by itself identify which client-side throttling behavior ran.

Four quota families protect four different bottlenecks

A quota meters an entity's resource use and delays or limits work after that entity exceeds its allowance. Kafka separates quota families because bytes, request-processing time, new connections, and metadata mutations stress different broker paths.

KIP-13's bandwidth quotas stop a producer or consumer from monopolizing network capacity. KIP-124's request quota catches clients that send many small or expensive requests without moving enough bytes to trip a bandwidth limit.

A request quota is a percentage of one thread's capacity, not a percentage of the whole broker. With num.io.threads=8 and num.network.threads=3, the two pools provide 1100% total thread capacity. A request_percentage of 25 grants 25% of one thread, or about 2.27% of that combined capacity.

KIP-124 defines the request-quota delay using observed usage O, target usage T, and measurement window W:

X = ((O - T) / T) × W
throttle = min(X, W)

Assume the broker observes 12% against a 10% target over a 1 s window. The calculated delay is (12 - 10) / 10 × 1 s, or 200 ms. The cap prevents the delay from exceeding that measurement window.

This is a deliberately bounded example, not a promise that one request always produces that delay. Kafka measures usage across rolling samples. For the same reason, you cannot derive an exact bandwidth throttle from a batch size and quota alone without knowing the broker's preceding rate samples.

KIP-612 added connection-creation-rate controls because authentication, TLS, and socket setup can overwhelm a broker before normal produce or fetch traffic dominates. These limits use IP, listener, and broker scopes rather than the user/client hierarchy used after authentication. Concurrent-connection settings are hard admission limits, not delayed client quotas, and do not produce the same throttle_time_ms response behavior.

The failure behavior differs by setting. max.connections and a listener-prefixed limit make the acceptor wait for capacity. Kafka protects the inter-broker listener from the broker-wide count; if admitting that connection pushes the total over its cap, another listener can lose its least-recently-used eligible connection. max.connections.per.ip accepts and then drops an excess socket, while a per-IP creation-rate violation delays closure for the calculated period. None of these paths returns a Produce or Fetch throttle response.

A controller is the Kafka node responsible for cluster metadata decisions. KIP-599 added controller mutation quotas for operations such as creating topics, adding partitions, and deleting topics. A quiet data plane does not make uncontrolled partition churn safe.

Request quotas still exclude work Kafka needs for cluster operation. Current Kafka 4.3.1 behavior is semantic rather than listener-wide:

• authorized APIs marked clusterAction use exempt handling, while failed cluster authorization remains throttleable;
• follower Fetch uses replication-byte accounting instead of the client request-time quota;
• Produce with acks=0 remains subject to producer bandwidth limits but cannot use response-based request throttling;
• pre-authentication ApiVersions, SASL handshake, and token exchange run inside the authenticator rather than the normal request-quota path;
• the inter-broker listener has special connection-limit protection, but it is not a blanket request-quota bypass.

The ApiKeys definitions and request handlers are the publication-version source of truth. Avoid copying a fixed list from an older KIP into an operating runbook.

KIP-219 made throttling visible without trusting the client

Kafka originally held a throttled response until its delay expired. That protected the broker, but the client learned about the throttle only after waiting and could mistake the pause for ordinary request latency.

KIP-219 changed the sequence for response versions whose clients support the new behavior:

1. The broker processes the request and calculates a delay.
2. It queues the response with throttle_time_ms without waiting out that delay.
3. It mutes the channel, meaning it temporarily stops reading from that client's socket.
4. A quota-aware client also pauses its own requests for the reported duration.

The client-side pause avoids pointless writes. The broker-side mute provides enforcement. If a client ignores throttle_time_ms, bytes may reach the socket buffer, but the broker will not process another request from that channel until it unmutes it.

The field therefore reports the broker's calculated delay; muting makes that delay stick. This split is why Kafka can communicate backpressure to cooperative clients without depending on their cooperation for isolation.

Fairness starts with a stable identity

Kafka combines two identity dimensions. A user principal is the authenticated identity established through SASL or mutual TLS. A client ID is a string the client places in its request header.

That difference matters. A principal can provide a security boundary. A client ID is self-declared metadata, useful for separating workloads that already share a trusted principal but unsafe as the only protection against an untrusted tenant.

KIP-55 defines the user/client quota precedence. Expressed as logical Admin API entities rather than metadata-store paths, the order is:

The first matching definition wins. A default user policy can therefore beat the client-ID-only quota an operator expected to apply.

In a modern KRaft cluster, configure these entities through the broker-facing Admin API, Kafka's administrative protocol surface. KIP-546 added alterClientQuotas() and describeClientQuotas(), and kafka-configs.sh --bootstrap-server exposes the same operating model without ZooKeeper addressing.

Suppose payments is the authenticated principal and checkout is its stable client ID. This command gives their exact pair a produce, fetch, and request allowance:

# ① Connect through a broker; KRaft owns the metadata.
# ② Byte-rate values apply independently on each broker.
# ③ This exact user/client pair has the highest precedence.
bin/kafka-configs.sh \
  --bootstrap-server broker-1:9092 \
  --alter \
  --add-config 'producer_byte_rate=52428800,consumer_byte_rate=104857600,request_percentage=25' \
  --entity-type users --entity-name payments \
  --entity-type clients --entity-name checkout

① keeps administration on Kafka's supported control surface. ② sets 50 MiB/s produce and 100 MiB/s fetch allowances per broker, not cluster-wide reservations. ③ ensures this pair wins over user defaults and client-ID-only entries.

My preference is to use authenticated principals for tenant ownership and client IDs for workload names such as checkout-api or nightly-backfill. If ten applications share both values, Kafka sees one quota bucket. No configuration value can reconstruct the missing organizational boundary during an incident.

Per-broker accounting makes cluster totals depend on placement

Kafka enforces client bandwidth and request quotas independently on every broker. The quota design documentation explains why: cluster-wide accounting would require brokers to coordinate usage on the request path.

A tenant with a 50 MiB/s produce quota can reach roughly 150 MiB/s across three brokers if it has enough demand and its partition leaders, the replicas accepting writes, are spread evenly across all three. If most leaders sit on one broker, that broker can throttle at 50 MiB/s while the other two remain below quota.

The configured number is therefore neither a cluster-wide cap nor reserved capacity. Adding brokers or moving partition leaders can change a tenant's attainable aggregate rate without changing its quota. Connection limits retain their broker, listener, or IP scopes, while controller mutation quotas protect controller work; do not multiply every family by broker count.

The response gives delay; labelled metrics give identity

throttle_time_ms answers “how long?” It does not answer “who?” or “which quota?” The broker already knows the authenticated user and request client ID, so its quota sensors attach those dimensions to metrics.

Self-managed operators can obtain these through JMX, Java's metrics and management interface, using the Kafka monitoring metric definitions. Exporters may rename the metrics, but they should preserve the labels needed to map a quota entity back to its owner.

A non-zero throttle is not universally an alert. A deliberately shaped backfill may spend hours throttled while meeting its completion target. A latency-sensitive service may treat sustained, unexpected throttling as an SLO risk. Alert when observed behavior violates your stated fairness policy or correlates with lag, latency, or failed administrative work—not because one threshold looked tidy on a shared dashboard.

Managed Kafka may expose a different control surface

The protocol mechanics describe Apache Kafka, but deployment ownership decides which knobs and metrics you can reach.

Managed services can enforce product limits that do not map to producer_byte_rate or return throttle_time_ms. Compare, for example, Confluent Cloud's service quotas with Amazon MSK's service limits: the available dimensions and remediation paths differ.

Before relying on a managed cluster for tenant isolation, confirm whether you can alter client quotas, which identity the service meters, and whether its exported metrics preserve tenant labels. If it exposes no suitable isolation control, separate clusters may express the boundary more honestly than client-side rate limiting.

What could go wrong

Quotas fail most often at policy boundaries, not in the delay formula.

• An old throttle workflow can amplify retries. KAFKA-6028, the operator report that motivated KIP-219, describes a MapReduce job whose clients timed out before receiving delayed responses, retried, and increased the throttle further. Immediate responses plus channel muting fixed that feedback loop in Kafka 2.0. The report gives no company, duration, or impact figures, so the mechanism is the evidence.
• Several tenants collapse into one identity. Shared principals and client IDs create one usage bucket by design. KIP-55's precedence rules cannot separate ownership that authentication never expressed.
• A broker hotspot looks like spare cluster capacity. Per-broker accounting allows one leader-heavy broker to throttle while cluster-wide averages remain comfortable. Recheck partition placement before raising the quota.
• Client telemetry misses the throttle. A response version without throttle_time_ms cannot report the delay, even though broker-side channel control still applies. Keep broker metrics in the diagnostic path.
• Exempt or unmetered work consumes the remaining capacity. Request quotas do not cap every controller, recovery, replication, disk, or connection cost. The Kafka quota design presents quotas as resource sharing, not a replacement for capacity planning.

Raising a quota may relieve the visible tenant while moving the bottleneck to disk, replication, or the controller. If the cluster lacks headroom, isolation may require workload placement or another cluster rather than a larger number.

A quota policy is executable organizational intent

The protocol details reduce to five operating rules:

• Check the negotiated response version before expecting throttle_time_ms.
• Use all four quota families where the corresponding resource is shared.
• Treat principals as tenant identity and client IDs as a workload dimension.
• Calculate bandwidth and request allowances per broker.
• Join throttle delay with labelled broker metrics before naming the noisy tenant.

Kafka can enforce the boundary you encode. It cannot decide where that boundary belongs.

Next, capture one real throttled exchange and join its response delay with the broker's labelled quota metrics before changing a limit.

Sources and References

Protocol and quota mechanics

• Apache Kafka 4.3 protocol schemas
• Apache Kafka quota design
• KIP-13: Quotas
• KIP-55: Secure Quotas for Authenticated Users
• KIP-124: Request Rate Quotas
• KIP-219: Improve Quota Communication
• KIP-546: Client Quota Admin APIs
• KIP-599: Controller Mutation Rate Quotas
• KIP-612: Connection Creation Rate Limiting

Operations and managed services

• Apache Kafka monitoring
• Confluent Cloud service quotas
• Amazon MSK service limits