Where a Kafka Request Spends Its Milliseconds

Your producer's p99 latency—the value 99% of requests complete under—jumps while broker CPU, throughput, and disk dashboards barely move. The tempting response is to add threads or change a client knob. Neither tells you where the request waited.

On the broker, the Kafka server that owns partition logs, latency isn't one stopwatch. A request can wait before a handler sees it, inside local processing, on another replica, or in the response path. Each delay has a different owner.

Kafka request latency is a map of queues, not a list of magic knobs. My rule is to follow those queues from outside inward, compare each one with a known-good run, and change configuration only after two adjacent signals agree.

The model below follows the Apache Kafka 4.3.1 broker source. It corrects three common mistakes: an Acceptor, the thread accepting new connections, doesn't process every request; the broker doesn't have four request thread pools; and a successful log append doesn't prove that bytes reached durable media.

Use this animation for ownership and ordering, not as a latency calculator. Its sample timings are illustrative. Requests on an established connection skip the Acceptor stage entirely.

The useful route is shorter than it first appears: a network thread, a bounded request queue, a request handler, an optional asynchronous wait, then the same network thread again. The sections below name the Kafka classes behind each stage.

A request doesn't cross four thread pools

A Kafka listener is a named network endpoint such as INTERNAL://:9092. For each data-plane listener, SocketServer creates one Acceptor, the thread that accepts new TCP connections, and num.network.threads network Processors, the workers that own established connections.

The broker also creates one shared data-plane pool of request handlers, threads that dequeue requests and run Kafka API logic through KafkaApis, the broker's request dispatcher. There is no separate response pool.

The defaults come from the broker configuration reference. With two data-plane listeners and default settings, a broker has two Acceptors, six network Processors, and eight shared request handlers—not two sets of eight handlers.

The Acceptor matters only when a client opens a connection. Long-lived Kafka connections don't revisit it for each Produce or Fetch request. After KIP-402, the Acceptor tries Processor connection queues without blocking before it waits for capacity. That prevents one full queue from stalling connection assignment while another Processor still has room.

Once assigned, a connection stays with its Processor. The Processor uses a Java NIO selector, an event loop that monitors many sockets from one thread, to read complete request frames. It also performs authentication and TLS, transport encryption, through Kafka's network channel before constructing a RequestChannel.Request.

The Processor then mutes that connection, places the request in RequestChannel, and returns to its selector work. It doesn't call KafkaApis, append records, or wait for replicas. Other connections assigned to that Processor can continue making progress.

The same Processor later sends the response. That symmetry matters: if encrypted traffic consumes the network stage, preserve TLS and add measured CPU or broker capacity. Security isn't a latency knob.

RequestChannel is the broker's pressure boundary

RequestChannel is the handoff between network Processors and request handlers. It isn't a thread. The current implementation stores incoming requests in a bounded ArrayBlockingQueue.

queued.max.requests controls the queue's capacity and defaults to 500. sendRequest uses a blocking put, so a full queue eventually stops a Processor from returning to its selector. That creates backpressure: downstream saturation slows the code accepting upstream work.

A larger queue can absorb a longer burst, but it doesn't increase service capacity. It can also turn overload into longer tail latency because requests spend more time waiting before a handler sees them.

The broker-wide KafkaRequestHandlerPool supplies the workers configured by num.io.threads. Each handler dequeues one request and calls the KafkaApis dispatcher, which selects the implementation for Produce, Fetch, Metadata, and the other APIs.

“I/O thread” is an awkward historical name. These workers don't form a separate disk-completion pool. They run authorization, metadata, coordination, log reads, and log appends. Some handlers finish synchronously. Others register a callback and release the worker while Kafka waits for replication or more fetch data.

Kafka already timestamps the handoffs

RequestChannel.updateRequestMetrics turns those boundaries into per-request timers:

Read these as indicators, not universal limits. RemoteTimeMs only separates cleanly when an API marks its local-completion timestamp. For simple APIs, local time can contain nearly all processing.

The broker also records ThrottleTimeMs for quota throttling, the intentional delay Kafka applies when a client exceeds a configured quota. Keep it separate from accidental queueing.

One boundary remains outside this map: Kafka starts request metrics after the network layer has assembled a complete request. Client batching, connection setup, request transmission, and some socket waiting can therefore increase client-observed latency without moving broker TotalTimeMs.

A successful append may still live only in page cache

For Produce, KafkaApis validates the request and calls ReplicaManager. The leader appends the batch through the local log, which eventually writes bytes through FileRecords.

That write normally reaches the page cache, kernel memory holding filesystem pages. A successful FileChannel.write means the kernel accepted the bytes. It doesn't mean the storage device has committed them to durable media.

The distinction appears directly in FileRecords: append writes through the channel, while flush calls channel.force(true). On Linux, the comparable durability boundary is fsync(2), which asks the kernel to flush modified file data and metadata to the storage device.

Kafka's filesystem design deliberately relies on page cache and replication instead of forcing every request to disk. Flush policies remain configurable through log.flush.interval.messages and log.flush.interval.ms, but frequent flushes change the throughput and latency envelope.

acks=all doesn't change that storage boundary. A delayed Produce completes when the partition's high watermark, the offset replicated across the current in-sync replicas (ISR), reaches the required offset. DelayedProduce checks replica progress; it doesn't require an fsync barrier on each replica.

Replication protects against the failure of an individual broker or host when replicas live elsewhere. It isn't proof that every acknowledged byte survived a simultaneous power loss across those hosts. If your failure model requires stable-media guarantees, test Kafka's flush policy, filesystem, controller, and drive cache together.

This distinction also changes diagnosis. A rise in LocalTimeMs can come from dirty-page throttling or filesystem work even when the device's average busy time looks normal. A short LocalTimeMs, meanwhile, proves only that Kafka completed its local path quickly.

Delayed operations release handlers without removing the wait

A delayed operation represents a request that has completed its immediate work but can't produce its final response yet. Kafka stores these operations in purgatory, a combination of condition watchers and timeout tracking implemented by DelayedOperationPurgatory.

For a Produce request waiting on replication, the sequence looks like this:

1. The request handler appends the batch to the leader's log.
2. ReplicaManager creates a DelayedProduce for partitions that haven't met the required offset.
3. Purgatory watches those partition keys and arms the operation's timeout.
4. Replica progress calls checkAndComplete; timeout expiry takes the alternative completion path.
5. The completion callback creates the response and sends it to RequestChannel.

The request handler returns to the shared queue after registration. It doesn't sit blocked for the full replication delay.

Fetch uses the same mechanism for a different condition. When the available data doesn't satisfy the request's minimum-byte requirement, DelayedFetch waits until more data arrives or the maximum wait expires. Purgatory turns both waits into watched state rather than sleeping request-handler threads.

The timing wheel makes timeout bookkeeping cheap

Kafka tracks delayed-operation timeouts with a hierarchical timing wheel, nested circular arrays that group deadlines into time buckets. The design follows Varghese and Lauck's timer algorithm analysis.

The current TimingWheel stores near deadlines in the lowest wheel and places later deadlines into lazily created overflow wheels. Each bucket is a doubly linked TimerTaskList. Kafka puts buckets—not every delayed request—into a DelayQueue.

As time advances, expired buckets leave the queue. Kafka either executes their tasks or moves still-future tasks into a lower wheel. Removing a task from its bucket is O(1); insertion may walk a small number of overflow-wheel levels before selecting a bucket. That narrower complexity claim follows the current implementation more closely than saying every hierarchical insertion is constant-time.

That does not make every delayed request O(1). One Produce request can watch many partition keys, and completion still has to inspect relevant replica state. The timing wheel makes deadline bookkeeping cheap; it doesn't erase replication work or lock contention.

Purgatory metrics need the same care. PurgatorySize counts watcher entries, while NumDelayedOperations estimates distinct delayed operations. A request spanning 20 partitions can contribute multiple watchers. Compare both metrics with your baseline and the request's RemoteTimeMs; don't treat either gauge as a request-latency threshold.

The response returns through the original Processor

Every RequestChannel.Request records the Processor that owns its connection. When KafkaApis or a delayed-operation callback finishes, RequestChannel.sendResponse routes the response to that Processor's queue and wakes its selector.

The Processor dequeues the response, registers the socket write, and records response-queue and send timing when the transfer completes. There is no num.response.threads setting because there is no response thread pool.

A rise in ResponseQueueTimeMs points toward a Processor that isn't draining completed work as quickly as before. A rise in ResponseSendTimeMs points farther outward: socket backpressure, constrained egress, a slow-reading client, or encryption cost can all contribute. None of those signals proves one cause by itself.

If a TLS change correlates with the shift, profile the network threads and preserve the security boundary. Reserve CPU, test the current JDK and cipher configuration, or distribute connections across more broker capacity. Disabling encryption makes a benchmark easier and the production failure model worse.

What could go wrong: JMX can stop one layer too early

Allegro's engineers hit that boundary in their 2024 Kafka latency investigation. Standard Kafka and device dashboards couldn't account for all observed tail latency, so they used eBPF, Linux kernel instrumentation that attaches probes without changing the broker, to trace filesystem and block-I/O behavior.

Their work separated time spent in Kafka from time spent below the JVM and showed why page-cache and filesystem behavior needed direct measurement. “Add more I/O threads” would have changed concurrency without identifying the slow operation.

The practical lesson is narrow: when LocalTimeMs moves but request queue, CPU, and device aggregates don't explain it, extend the same latency map into scheduler, syscall, filesystem, and block layers before changing Kafka configuration.

Follow queues from outside inward before touching a knob

Start with a healthy baseline: measurements from the same request API, broker role, listener security, payload shape, partition distribution, and comparable load during a known-good period. Friday's Fetch traffic isn't a useful baseline for Monday's Produce spike.

Kafka exposes the broker metrics through JMX (Java Management Extensions). Exporters don't always preserve units: idle “percent” metrics may appear as a ratio from zero to one or as zero to 100. Verify your exporter before comparing values.

Don't subtract client p99 from broker p99 and call the remainder “network latency.” Those percentiles can describe different requests, and quantiles don't aggregate arithmetically. Compare how distributions move, or correlate individual requests with tracing when you need an additive budget.

Walk the boundaries in this order:

The Apache monitoring reference documents the object names, but it can't supply a healthy number for your workload. A Processor idle value that is ordinary for a steady high-throughput cluster may be a regression for a bursty low-latency one.

Use this method:

1. Segment first. Compare Produce with Produce, FetchFollower with FetchFollower, and one broker with its peers. Broker-wide averages hide partition and listener skew.
2. Find the first changed boundary. If client latency moves while broker TotalTimeMs stays flat, stay outside the broker-timed path. If request-queue time moves first, continue into handler and local-time evidence.
3. Require corroboration. Queue depth without queue time can be a harmless burst. Lower idle time without worse latency can mean useful work. Two adjacent signals make a stronger case.
4. Test one change against the same load. Keep the old and new distributions, not just their averages.
5. Revert when the predicted stage doesn't improve. A knob that changes no relevant boundary wasn't the cause.

Only test more num.io.threads when RequestChannel is draining too slowly and the local work has CPU or storage headroom. Extra handlers can increase contention when the filesystem is already the limit.

Only test more num.network.threads when the network stage moved relative to baseline. Remember that Kafka applies the count per listener, so the total thread change may be larger than the config diff suggests.

Treat queued.max.requests as burst capacity, not processing capacity. For rising remote time, repair follower storage, inter-broker networking, partition skew, or ISR health while retaining the replication contract. For network encryption cost, add capacity rather than weakening security.

The first moved queue earns the next measurement

An established Kafka request enters through a network Processor, waits in RequestChannel, runs through a request handler and KafkaApis, possibly becomes a delayed operation, then returns through the original Processor. The Acceptor only owns connection setup. A local append usually reaches page cache, while replication and durable-media flush remain separate boundaries.

There is no universal 30% idle threshold, magic queue depth, or deterministic diagnosis. There is only your healthy baseline and the first stage that departed from it.

Follow queues from outside inward before changing knobs.

Producer idempotence and transactions add sequence numbers, fencing, and transaction coordination to this broker path.

Sources

Apache implementation and configuration

• SocketServer.scala — Acceptor, Processor, and response-path ownership
• RequestChannel.scala — request queue, response routing, and latency timestamps
• KafkaRequestHandler.scala — broker request-handler pool
• KafkaApis.scala — API dispatch
• ReplicaManager.scala — append and delayed-operation coordination
• Apache Kafka broker configuration — network, request-handler, queue, and flush settings
• Apache Kafka monitoring — broker metric names and definitions

Storage and delayed operations

• FileRecords.java — channel append and flush behavior
• DelayedOperationPurgatory.java — purgatory watchers and metrics
• TimingWheel.java — hierarchical timer implementation
• Purgatory Redesign Proposal — timer and watcher design rationale
• Hashed and Hierarchical Timing Wheels — Varghese and Lauck's timer analysis

Production investigation

• Allegro Tech: Kafka performance analysis — eBPF investigation of broker and kernel latency