Kafka TLS: Where the JVM Zero-Copy Fetch Path Ends

Switch a Kafka listener, a broker network endpoint, from PLAINTEXT to SSL, Kafka's listener protocol name for a TLS-protected connection. The record batch, the unit Kafka stores and fetches, does not change on disk. The component that owns its next representation does.

For a local file-backed Fetch, Kafka's response API for reading records, the plaintext transport can delegate the record bytes to Java's file-transfer API. Kafka's default encrypted transport reads those bytes into user space and passes them through Java Secure Socket Extension (JSSE), the JDK TLS API, before writing ciphertext, the protected wire bytes, to the socket.

“TLS disabled zero-copy” points at the right branch and hides the useful explanation. Transport Layer Security (TLS) creates a new authenticated representation. In stock Kafka, SSLEngine owns that transformation in user space.

TLS does not make copy avoidance impossible. Kafka 4.3.1's default transport places the TLS record layer where a file descriptor and a TLS connection cannot share one transfer operation.

Episode one established the ownership ledger. This episode follows one file-backed record range until TLS forces a new representation.

This source trace is pinned to Apache Kafka 4.3.1, OpenJDK 25.0.4 GA, Linux 6.19, and OpenSSL 3.6.3, all available by August 1, 2026. Kafka 4.3 fully supports Java 25.

The scope is one local FileRecords range in a Fetch response. Remote or tiered storage, in-memory records, message-format conversion, non-Linux systems, other JVMs, and custom Kafka transports can take different paths.

The fast path covers file-backed record bytes, not the whole response

Kafka writes Fetch metadata and protocol framing through ordinary buffers. The special path applies to the file-backed record range inside that response. Two methods establish the handoff:

// FileRecords.writeTo(...)
return (int) destChannel.transferFrom(channel, position, count); // ①
 
// PlaintextTransportLayer.transferFrom(...)
return fileChannel.transferTo(position, count, socketChannel);   // ②

① FileRecords.writeTo() passes a file, position, and count to Kafka's transport abstraction.

② PlaintextTransportLayer.transferFrom() unwraps that abstraction into a FileChannel to SocketChannel transfer. Kafka's TransferableChannel documents why the concrete destination type matters to OpenJDK's direct path.

Java's FileChannel.transferTo() offers an optimization opportunity, not a promise. OpenJDK 25.0.4 first tries a direct transfer for supported file and socket channel implementations. If that fails, it can use a mapped transfer or a buffered read/write loop, as FileChannelImpl shows.

The Linux native dispatcher has one subtle detail:

1. It first calls copy_file_range().
2. Linux copy_file_range(2) requires regular-file endpoints, so a socket destination does not match its contract.
3. OpenJDK then calls sendfile() for the socket case.

The page cache is kernel memory that caches file contents. The resulting payload path is narrow:

log file -> page cache [kernel]
         -> sendfile file-to-socket path [kernel]
         -> TCP and network device

It avoids staging those record bytes in a Java payload buffer. It does not remove Fetch framing, storage reads on a cache miss, TCP bookkeeping, retransmission state, checksums, or device work. It also does not guarantee that a different JDK, filesystem, channel type, or operating system reaches the same syscall.

Kafka earns this opportunity above the syscall. Its shared binary record format lets producers, broker log segments, and consumers use the same bytes. The Kafka compression essay shows why compressed record batches can keep this path when the broker sends them unchanged.

Stock Kafka TLS exposes two Java staging buffers

When ssl.engine.factory.class is unset, Kafka's SslFactory creates DefaultSslEngineFactory. That factory builds a Java SSLContext and creates an SSLEngine; a configured Java security provider can change the implementation behind the API. DefaultSslEngineFactory still returns the same buffer-oriented SSLEngine contract. Kafka's startHandshake() allocates netWriteBuffer with ByteBuffer.allocate().

Kafka 4.3.1's file-transfer loop can be reduced to this source-guided pseudocode:

fileChannelBuffer = ByteBuffer.allocateDirect(32768);      // ①
fileChannel.read(fileChannelBuffer, position);              // ②
sslEngine.wrap(fileChannelBuffer, netWriteBuffer);          // ③
socketChannel.write(netWriteBuffer);                        // ④

① SslTransportLayer.transferFrom() lazily allocates a 32 KiB direct buffer, user-space memory outside the Java heap, for file plaintext.

② A positioned file read copies bytes from the kernel's page-cache path into that direct user-space buffer. Using a direct destination avoids the extra temporary-direct-to-heap copy that OpenJDK uses for a heap destination, as IOUtil.read() shows.

③ SSLEngine.wrap() consumes application bytes and produces network bytes. Kafka allocates netWriteBuffer with ByteBuffer.allocate(), so this pinned path stores ciphertext in a Java heap buffer. The TLS provider may perform more internal movement; the public API does not expose a universal copy count.

④ SocketChannel.write() sends the ciphertext. In OpenJDK 25.0.4, IOUtil.write() copies a heap source into a temporary direct buffer before the native socket write. The SocketDispatcher then calls write().

The Java-visible ledger therefore contains:

page cache [kernel]
  -> direct plaintext buffer [Kafka user space]
  -> TLS ciphertext buffer [Kafka heap]
  -> temporary direct ciphertext buffer [OpenJDK]
  -> socket path [kernel]

The first arrow is a copy of the stored plaintext. The second creates a new representation and should be called a transformation, not merely another copy. The third is a same-representation staging copy in this OpenJDK release. The ordinary socket write crosses back into the kernel.

Partial socket progress can leave ciphertext in netWriteBuffer for a later writable event. That lifetime is why Kafka flushes pending network bytes before wrapping more plaintext.

TLS 1.3 makes the representation change explicit. A TLS record is the protocol unit protected for transport. RFC 8446 limits plaintext fragments to 2^14 bytes and defines Authenticated Encryption with Associated Data (AEAD), an operation that produces authenticated ciphertext from plaintext, a key, a nonce, a per-record value that must not repeat for one key, and record-header data. The output includes an outer record header, encrypted content, an inner content type, optional padding, and AEAD expansion.

Kafka cannot send its stored plaintext bytes unchanged through this stock SSLEngine path. That statement is about Kafka's owner and API boundary, not a law of TLS.

kTLS moves the record layer without removing it

Kernel TLS (kTLS) lets a user-space TLS library complete the handshake and install symmetric record-layer state in the kernel. Linux configures transmit and receive independently. The kernel or NIC then owns record framing and cryptography for the enabled direction.

A scatter list describes several memory fragments as one kernel I/O input. A network interface card (NIC) connects the host to the network. Direct memory access (DMA) lets that device read host memory without a CPU copy loop. These paths are not interchangeable:

Linux 6.19's kTLS documentation states that ordinary kTLS allocates a buffer for encrypted data. TLS_TX_ZEROCOPY_RO is a separate device-offload-only option that lets sendfile() data reach the NIC without that in-kernel copy.

The option does not mark or enforce pages as read-only. The caller promises not to modify the submitted range until transmission completes, including any TCP retransmission that still needs the original bytes. A positive sendfile() return reports bytes accepted by the TLS socket, not a per-range release event, and this API provides no MSG_ZEROCOPY-style reuse notification.

The kTLS offload document also distinguishes packet-based TLS_HW from TLS_HW_RECORD, where NIC implementation replaces the Linux TCP path. This essay means packet-based offload. Full TCP offload is a different networking architecture, not a faster setting for Kafka's existing socket path.

A package label cannot prove kTLS or NIC offload

OpenSSL provides a concrete kTLS integration, but every connection still has to pass several gates. OpenSSL 3.6.3 must be built with enable-ktls, the application must enable the option, and the negotiated protocol features must match kernel support. A cipher suite names the record cipher and hash in TLS 1.3; older TLS versions encode more choices in the suite.

The application-level check is per direction:

SSL_CTX_set_options(ctx, SSL_OP_ENABLE_KTLS);              // ①
 
if (BIO_get_ktls_send(SSL_get_wbio(ssl)) != 1) {           // ②
    /* This connection is not using kTLS for transmit. */
}

① OpenSSL's SSL_OP_ENABLE_KTLS requests the kernel data path. Compilation, cipher suite, extension, platform, and kernel support can still reject it.

② BIO_get_ktls_send() and BIO_get_ktls_recv() report transmit and receive separately. A successful TX setup says nothing about RX.

Only after TX kTLS succeeds can SSL_sendfile() send from a file descriptor. Even then, ordinary software kTLS and TLS_TX_ZEROCOPY_RO have different copy boundaries.

Verify the stack in layers:

The fallback layers are separate. If OpenSSL cannot install kTLS for the negotiated connection, the record layer remains in user space. If kTLS installs but NIC offload setup does not, Linux uses host-side TLS_SW. Even during an active TLS_HW session, rerouting, retransmission, or resynchronization can send individual packets through software handling. JVM profiles may then show less crypto even though kernel or device work increased, so observability must move with the owner.

Kafka's current design documentation states that in-kernel SSL_sendfile is not supported by Kafka. There is no broker switch that enables kTLS, NIC TLS offload, or TLS_TX_ZEROCOPY_RO for the file-backed Fetch path.

A TLS proxy cannot recover Kafka's log descriptor

A normal TLS proxy owns the sockets on each side. Kafka owns the log-segment file descriptor. Once the proxy receives bytes over TCP, it cannot turn that socket input back into Kafka's original file-to-socket transfer.

The proxy may use kTLS for its own connection, but it does not restore the broker's FileChannel.transferTo() opportunity. A source-owning Kafka transport would need to combine the log descriptor, connection keys, partial write state, and Kafka's selector lifecycle in one implementation. That is a custom transport or upstream change, not a proxy setting.

TLS termination can also change authentication identity. Kafka access-control lists (ACLs) authorize a principal, the authenticated identity attached to a connection. Kafka's TLS principal mapping derives the default SSL identity from the client certificate's X.500 name. Unless a proxy preserves an equivalent authenticated identity through another trusted mechanism, Kafka authorizes the proxy rather than the original client.

Treat TLS termination as a security-boundary design, not a copy optimization.

Historical Kafka tests explain mechanisms, not 2026 capacity

KAFKA-2517 recorded a 2015 regression on one local MacBook, one broker, and 1 KB messages. A new transport abstraction hid the concrete socket type, so plaintext FileMessageSet.writeTo() missed its direct transfer. The fix restored the opportunity. Kafka's current TransferableChannel comment preserves that lesson.

KAFKA-2561 recorded a separate 2015 laptop experiment with Java 8u60, Netty OpenSSL, JSSE, plaintext, and different cipher suites. It showed that provider, cipher, and transport branch mattered in that setup. It did not isolate one modern TLS cost, and its throughput ratios do not predict Kafka 4.3.1 on current CPUs and NICs.

Use these issues as failure records. Do not turn them into a capacity percentage.

Benchmark the branch you can name

A defensible comparison holds the same local file-backed record range, batch shape, partition placement, cache residency, client count, offered load, JDK patch, cipher suite, and authentication policy. Offered load is the rate at which clients attempt to fetch data. Change the listener protocol, then identify the first boundary that moved.

The Kafka request-path essay explains where ResponseSendTimeMs sits in the broker's queue model. Use PLAINTEXT only as an isolated diagnostic comparator when the environment can do so safely. It answers which transport branch changed, not whether production should drop encryption.

What could go wrong when transport engineering outruns evidence

• The direct API takes a fallback. FileChannel.transferTo() permits direct, mapped, and buffered implementations. Handle returned byte counts and trace the syscall before claiming a file fast path.
• An abstraction hides the destination. KAFKA-2517 is the concrete failure: the code still called a transfer method, but OpenJDK no longer saw the channel type required for its direct branch.
• A custom engine breaks Kafka's state machine. KAFKA-16305 records a Netty/OpenSSL SSLEngine handshake stall fixed in Kafka 3.7.1 and 3.8.0. Provider replacement needs handshake, closure, partial-write, and error-path tests, not only a throughput run.
• kTLS can move crypto outside the required compliance boundary. OpenSSL documents that kernel TLS operations bypass OpenSSL providers. That can be unacceptable when policy requires all cryptography to run through a specific provider such as the FIPS provider.
• NIC installation can fall back to software. Linux keeps the connection in host-side kTLS when device setup fails, and packet-level fallback can still occur during an active device session. Feature flags alone do not prove that a session or packet used the NIC.
• Read-only zero-copy can authenticate different bytes on retransmission. Linux warns that mutating a file range before transmission completes can make the original send and a retransmission differ. The peer then reports a TLS record authentication failure.
• A hand-rolled TLS 1.3 integration can mishandle key updates. Linux requires user space to provide updated TX and RX keys and explicitly says the kernel does not check for key or nonce reuse. Use a maintained TLS library unless the transport team owns that protocol state.

Copy avoidance is useful only while security, correctness, fallback, and diagnostics remain explicit.

TLS changes the owner that must touch the bytes

Kafka's plaintext file-backed Fetch payload can avoid Java staging when OpenJDK reaches sendfile(). Kafka's default TLS path reads plaintext into a direct buffer, asks SSLEngine to create ciphertext in a heap buffer, and crosses another temporary direct buffer before the native socket write in OpenJDK 25.0.4.

kTLS proves that another ownership model is possible. Software kTLS removes the user-space record path but keeps kernel crypto and encrypted buffering. Packet NIC offload moves crypto again. TLS_TX_ZEROCOPY_RO removes one more in-kernel file-data copy and adds a strict immutability contract. Stock Kafka 4.3.1 enables none of those kTLS paths.

The completed Kafka Engineering posts on compression and request latency explain the format and queues around this branch. The Valkey reply copy-avoidance implementation removes a different user-space reply copy with references and writev().

Zero-copy is earned one ownership boundary at a time.

Part three moves to the receive side, where io_uring has to loan application-visible buffers without losing the NIC's recycling contract.

Sources and References

Apache Kafka 4.3.1

• FileRecords.writeTo()
• TransferableChannel
• PlaintextTransportLayer.transferFrom()
• SslTransportLayer.write() and transferFrom()
• SslFactory and DefaultSslEngineFactory
• Kafka 4.3 design: efficiency and current TLS limitation
• Kafka 4.3 Java version support
• Kafka 4.3 TLS principal mapping

OpenJDK and TLS protocol

• Java 25 FileChannel.transferTo()
• OpenJDK 25.0.4 FileChannelImpl
• OpenJDK 25.0.4 Linux FileDispatcherImpl.c
• OpenJDK 25.0.4 IOUtil and SocketDispatcher.c
• Java 25 SSLEngine
• RFC 8446 section 5: TLS 1.3 record protocol

Linux 6.19 and OpenSSL 3.6

• sendfile(2), man-pages 6.18
• copy_file_range(2), man-pages 6.18
• Linux 6.19 kTLS interface, zero-copy option, and statistics
• Linux 6.19 kTLS software, packet NIC, and full TCP offload modes
• Linux 6.19 TLS UAPI
• OpenSSL 3.6 SSL_OP_ENABLE_KTLS and zero-copy sendfile option
• OpenSSL 3.6 BIO_get_ktls_send() and BIO_get_ktls_recv()
• OpenSSL 3.6 SSL_sendfile()
• ethtool(8)

Historical failure records

• KAFKA-2517: plaintext direct-transfer regression
• KAFKA-2561: historical Java 8 and OpenSSL experiment
• KAFKA-16305: custom SSLEngine handshake stall