Kafka Producer Idempotence, Fencing, and Transactions

Your producer reports a timeout after sending a batch. Kafka may already have appended it, but the response vanished somewhere between the broker and your client. Retrying is the only sensible move—and, without three fields in the batch header, it's also how one logical write becomes two.

Kafka calls the safe-retry property producer idempotence: the broker recognizes a retried batch and avoids appending it twice. The proof lives in 14 bytes—producer ID, producer epoch, and base sequence—inside a 61-byte fixed header.

By the end, you should be able to:

• account for every byte in a Record Batch v2 header;
• predict whether the broker accepts, deduplicates, or rejects a batch;
• trace fencing, transaction coordination, and commit or abort markers;
• state where Kafka's exactly-once semantics stop.

The batch header spends exactly 61 bytes proving who wrote what

A Record Batch v2 is the wire and log container for one or more records. Kafka introduced this format with message magic 2; the message-format specification and DefaultRecordBatch define the following byte layout.

The variable-length records begin at byte offset 61. Any layout that ends after BaseSequence is 4 bytes short because it omits RecordsCount.

The checksum boundary matters. Kafka calculates the CRC from byte 21 through the end of the batch, so it protects the producer ID, epoch, sequence, record count, and payload. The broker can assign BaseOffset and PartitionLeaderEpoch without recalculating the checksum because both fields sit before that boundary.

BaseSequence names the first record, not the whole batch. Each record carries an OffsetDelta, and Kafka derives its sequence as BaseSequence + OffsetDelta. A normal 100-record batch with base sequence 0 consumes sequences 0 through 99; the next batch for that partition starts at 100. KIP-98 specifies this per-record sequence model.

Without idempotence, Kafka writes -1 into the producer ID, epoch, and sequence fields. The batch remains valid, but the broker has no producer identity against which to check a retry.

One terminology boundary prevents a common durability mistake. When I say the leader appends a batch, I mean Kafka writes it to the log through the operating system's page cache, memory the kernel uses to buffer filesystem I/O. Kafka's filesystem design explicitly relies on that cache; a successful produce request doesn't imply a per-request fsync.

Followers also don't receive a leader push or send a dedicated ACK. They issue Fetch requests, append the returned bytes locally, then advertise their new fetch position in a later request. acks=all, which idempotence requires, waits for in-sync replica progress—not for every storage device to complete fsync. Replication and leader election determine the durability consequences later in the series.

Idempotence deduplicates retries, not events

When an idempotent producer starts, it sends InitProducerId. Kafka returns a producer ID (PID), a 64-bit identity, and a producer epoch, a 16-bit generation for that identity. The client then maintains a separate 32-bit sequence number for every topic-partition.

That last boundary is easy to miss. Kafka doesn't maintain one global sequence for a producer. A batch for orders-3 and a batch for orders-7 advance independent counters, even when one Produce request carries both.

Each partition log owns a ProducerStateManager. It tracks the current epoch and recent sequence ranges for every producer that has written to that partition. When a client retries, it sends the same batch with the same PID, epoch, and sequences.

The duplicate check compares batch metadata, not record contents. Sending the same key and value twice with two valid sequence ranges creates two records. Kafka doesn't hash business events, inspect an order ID, or infer application intent.

The broker retains metadata for the five most recent batches per producer-partition, as the ProducerStateEntry.NUM_BATCHES_TO_RETAIN constant shows. It doesn't retain five individual sequence numbers. This bounded window matches the Java producer's idempotent limit of five in-flight requests per connection.

Consider the lost-response case from the opening. The leader appends sequences 40–59, but the response disappears. The producer retries the identical batch. The broker finds the same epoch and sequence range in its recent metadata, returns the original append result, and doesn't write another copy.

That protection has limits. If the application gives up, creates a new producer, and resubmits the records under a new PID, Kafka sees a new writer. Idempotence also can't identify a business event that your application deliberately sends again.

The visualization uses one-record batches, so its sequence rises by one per batch. Multi-record batches advance by their record count. The duplicate path concerns a duplicate Kafka log append, not whether an external side effect executed.

Correctness settings belong beside the schema

Defaults need a client and version label. For the Apache Kafka 4.3 Java producer, the relevant documented defaults are:

KIP-679 changed the Java producer defaults in Kafka 3.0. Other client implementations and older Java clients can differ.

The modern default is conditional. If you omit enable.idempotence but configure acks to something other than all, set retries=0, or raise max.in.flight.requests.per.connection above 5, the Java client disables idempotence. If you explicitly set enable.idempotence=true alongside a conflicting value, it throws ConfigException instead.

My view: producer settings that change duplication, ordering, or atomic visibility are part of the data contract. A schema says what a record means; these settings say whether a retry may write it twice. Review and version the client implementation, client version, resolved settings, and transactional-ID strategy with the same care as schema compatibility.

Fencing decides which producer incarnation may continue

An idempotent producer without a transactional ID has no application-stable identity. After a restart, it normally receives a new PID. Kafka can deduplicate retries from the current session, but it can't connect the new PID to work attempted by the old process.

A transactional ID gives one logical writer a stable name across process restarts. Kafka maps that name to a PID and epoch in the transaction coordinator's state. Fencing means rejecting an older process after a newer process claims the same transactional ID.

When a replacement calls initTransactions(), the coordinator completes or aborts unfinished work from the previous instance and advances the producer epoch. The old instance can no longer complete its transaction; the Java client surfaces ProducerFencedException as a fatal error. The application must close that producer.

The epoch is a bounded 16-bit generation, not a timestamp. KIP-360 specifies PID rotation when the epoch reaches its limit, preserving the fencing model without pretending the counter grows forever.

A transactional ID therefore needs two properties: stability across restarts of the same logical task and uniqueness among tasks running concurrently. Reusing one ID for unrelated instances makes them fence each other. Generating a random ID on every restart prevents the replacement from fencing its predecessor.

Fencing still isn't business-event deduplication. A valid producer may submit the same order in two separate committed transactions, and Kafka will accept both sequence ranges.

Transactions create one Kafka outcome through coordinator state and markers

A Kafka transaction groups record batches—and optionally consumer offsets—under one commit or abort outcome. Aborting doesn't erase bytes already appended to partition logs. Instead, Kafka writes control markers that let appropriately configured consumers hide the aborted records.

The transaction coordinator is the broker leading the relevant partition of the internal __transaction_state topic. Kafka hashes the transactional ID to choose that partition. The exact request sequence depends on transaction.version, a finalized cluster feature rather than a broker property:

New Kafka 4.3.1 clusters and clusters finalized with upgrade --release-version 4.3 use level 2. A rolling binary upgrade alone does not change an existing cluster's finalized level; an absent level is treated as 0. A 4.0+ Java producer uses transaction protocol v2 only after it observes finalized level 2 through ApiVersions, and changes protocol between transactions rather than in the middle of one.

Operators can inspect and enable the level explicitly:

bin/kafka-features.sh --bootstrap-server broker-1:9092 describe
 
bin/kafka-features.sh --bootstrap-server broker-1:9092 \
  upgrade --feature transaction.version=2 --dry-run
 
bin/kafka-features.sh --bootstrap-server broker-1:9092 \
  upgrade --feature transaction.version=2

KIP-890's transaction protocol v2 moves partition registration into the first transactional Produce request and advances the producer epoch for each transaction. Older clients or clusters finalized at level 0 or 1 retain the explicit AddPartitionsToTxn path from KIP-98:

1. Discover and initialize. The client uses FindCoordinator, then sends InitProducerId with its transactional ID and transaction timeout. The coordinator returns the PID and current epoch.
2. Begin locally. beginTransaction() changes client state but sends no request.
3. Register participants. With transaction protocol v2, the first transactional Produce implicitly registers its partition. Legacy protocol versions send AddPartitionsToTxn first. In both cases, the Record Batch carries the transactional attribute bit.
4. Attach consumer progress when needed. Legacy clients use AddOffsetsToTxn before TxnOffsetCommit. Under transaction protocol v2, TxnOffsetCommit performs the server-side addition. Both place source consumer offsets under the same outcome as the output records.
5. Choose an outcome. commitTransaction() or abortTransaction() asks the coordinator to end the transaction.
6. Write markers. The coordinator durably appends PREPARE_COMMIT or PREPARE_ABORT, responds to the producer, and then drives commit or abort markers to every participant asynchronously. Transactional visibility advances as markers reach each partition; a successful commitTransaction() does not mean every partition became visible at the same wall-clock instant.

WriteTxnMarkers is a broker-to-broker request. Each partition leader appends a control batch, a Record Batch whose control flag identifies it as protocol metadata. Its control record carries COMMIT or ABORT plus the coordinator epoch. If the coordinator fails after recording the prepare state, its successor can recover the participant list and continue sending the chosen markers.

The common read-process-write case commits source offsets beside output records. The offsets map below contains the next offset to consume for each input partition, not the offset of the last processed record.

producer.initTransactions();
 
producer.beginTransaction();                                      // ①
for (ConsumerRecord<String, Order> record : records) {
    producer.send(toOutputRecord(record));
}
producer.sendOffsetsToTransaction(
    offsetsAfter(records),
    consumer.groupMetadata());                                    // ②
producer.commitTransaction();                                     // ③

① beginTransaction() only changes local producer state.
② The group metadata lets the coordinator reject offset commits from a stale consumer generation, the fencing improvement formalized in KIP-447.
③ commitTransaction() flushes pending records and waits for the durable EndTxn decision. Marker propagation can still be running when it returns. A processing failure should take the abort path; a fenced producer must close.

This example uses the ordinary producer transaction contract with transaction.two.phase.commit.enable=false. Kafka 4.3.1 documents a separate two-phase extension whose transactions do not share the ordinary timeout contract. The Kafka Transactions & Visibility Lab returns OUTSIDE_MODEL for that mode instead of importing ordinary liveness claims.

Terminal uncertainty also needs operation-specific handling. If commitTransaction() or abortTransaction() times out after a possible coordinator decision, retry the same terminal operation or close and recover. Do not switch commit to abort, or abort to commit, merely because the response was late. The pinned KafkaProducer method contracts separate those timeout paths from fatal fencing, authorization, unsupported version, and sequence errors.

The state-machine visualization uses STATE=ONGOING, STATE=PREPARE_*, and related labels as readable summaries while comparing legacy explicit registration with transaction protocol v2. Kafka actually persists transaction metadata—PID, epoch, timeout, state, and participant partitions—in __transaction_state; those labels aren't literal application records.

Every participant receives the same commit or abort outcome, but Kafka doesn't expose a cross-partition snapshot-read API. Consumers still fetch partition streams independently, and each partition advances after its own marker arrives.

The model's coordinator column can show a committed decision while the producer column remains UNKNOWN. It then retries the same commit operation; it never invents an abort.

read_committed trades visibility for a clean transactional view

A read_committed consumer returns non-transactional records and records from committed transactions while hiding ongoing and aborted transactional records. The Kafka 4.3 Java consumer default remains read_uncommitted, so a producer transaction alone doesn't give every downstream consumer a committed-only view.

For each partition, Kafka calculates the Last Stable Offset (LSO): the first offset belonging to any open transaction, or the partition's readable replicated end when none remain open. A read_committed fetch returns records below the LSO.

That creates a non-obvious form of head-of-line blocking. If a transaction starts at offset 50 and remains open, a read_committed consumer can't receive an unrelated non-transactional record at offset 51. One old transaction delays everything after it in that partition until Kafka writes a commit or abort marker.

Aborted records remain in the log. Partition leaders maintain an aborted-transaction index, and the Fetch response reports aborted producer ranges so clients can filter them. Consumers don't receive control markers as application records, and skipped aborted batches can create legitimate offset gaps.

Transactions therefore promise atomic visibility only to consumers that request it. They don't promise that aborted bytes consume no storage, that offsets remain contiguous, or that a long-running transaction has no effect on unrelated records.

Aborting a processing transaction does not rewind the consumer's fetched position. Restore or seek to the last stable committed input position before processing continues. Transactional group progress adds another recovery boundary: a replacement can observe an unresolved offset checkpoint while the earlier output/progress transaction is still completing. The 4.3.1 classic and consumer-protocol commit managers request stable offsets and handle UNSTABLE_OFFSET_COMMIT; an older stored offset is not automatically the final answer. The consumer progress reference owns the application ledger and rewind handoff.

Kafka's exactly-once boundary ends at Kafka

Kafka's exactly-once semantics (EOS) cover Kafka-managed state: input offsets, output topic records, and the Kafka-backed state used by systems such as Kafka Streams. The exactly_once_v2 processing guarantee builds on the producer transaction described above.

The protocol can't enlist an external database transaction or an HTTP call. Consider this illustrative failure sequence: a handler calls a payment endpoint, the endpoint succeeds, and the process dies before commitTransaction(). Kafka leaves the input offset uncommitted, so the next attempt may call the endpoint again. An abort marker can hide Kafka output; it can't reverse the payment.

The reverse ordering also has a gap. Committing Kafka output before updating a database can leave the record visible when the database write later fails.

External systems need their own correctness mechanism. An HTTP service can persist an idempotency key and return the prior result for a repeated operation. For a database-to-Kafka path, a transactional outbox stores the business change and outbox row in one database transaction, then publishes through change data capture; Debezium documents that pattern and its limits.

Neither mechanism enlarges a Kafka transaction. It builds a separate, explicit bridge across the boundary.

For the application-facing implementation, use A Kafka Transaction Cannot Roll Back Your HTTP Request. It keeps source intent, Kafka publication, consumer progress, sink identity, effect truth, and caller knowledge separate.

What could go wrong: producer identities can exhaust broker memory

PagerDuty's postmortem for its August 28, 2025 Kafka outage gives producer identity a concrete operational cost. A bug created a new KafkaProducer for every API request, causing Kafka to allocate roughly 4.2 million producer IDs per hour. Broker memory filled with ProducerStateManager metadata, and the resulting failures cascaded across the cluster.

The fix matched the client contract: reuse long-lived producer instances. The Java KafkaProducer API documents the client as thread-safe and recommends sharing it.

For transactional work, “share it” doesn't mean one producer should host unrelated concurrent transactions; a producer supports one active transaction at a time. Give each logical transactional task a stable owner, keep that producer alive, and replace it deliberately after a fatal fencing or sequence error—not once per request.

The payload isn't the guarantee

• Record Batch v2 has a 61-byte fixed header, including the 4-byte RecordsCount.
• PID, epoch, and per-record sequences deduplicate retried batches; they don't deduplicate business events.
• Transactional IDs fence stale producer instances, while coordinator state and control markers choose one Kafka outcome.
• read_committed hides aborted data but can stop behind the oldest open transaction.
• Kafka transactions cover Kafka records and offsets. External side effects need a separate contract.

A schema tells readers what the bytes mean. Producer correctness configuration tells them what retries are allowed to do with those bytes. Version both.

Next, replication determines which acknowledged history a replacement leader is allowed to keep.

Sources and References

Apache Kafka protocol and client behavior

• Record Batch message format
• KIP-98: Exactly Once Delivery and Transactional Messaging
• KIP-360: Improve reliability of idempotent and transactional producers
• KIP-447: Producer scalability for exactly-once semantics
• KIP-679: Enable the strongest producer guarantee by default
• KIP-890: Transactions Server-Side Defense
• Kafka 4.3 transaction protocol
• Kafka 4.3.1 TransactionVersion.java
• Kafka 4.3.1 transaction coordinator marker flow
• Kafka 4.3 producer configuration
• Kafka 4.3 consumer configuration

Source code and operations

• DefaultRecordBatch.java
• ProducerStateManager.java
• PagerDuty's August 28, 2025 Kafka outage postmortem
• Debezium outbox event router