A Successful Kafka Send Is Not a Completed Business Operation

Your request times out after calling producer.send(). The callback never arrives. The incident channel asks one question: “Did we publish the order?”

The honest answer may be unknown. Kafka can append and replicate a batch while the response disappears. It can also lose the process before the batch leaves client memory. Those histories need different recovery actions even though the application sees the same missing result.

This reference uses Apache Kafka and the Java client 4.3.1. It separates five boundaries:

1. the application accepts a business intent;
2. the producer accepts a record into local memory;
3. a partition leader appends it;
4. the selected acknowledgement policy completes;
5. the business effect finishes.

The Java producer configuration and KafkaProducer method contracts at tag 4.3.1 define the client-facing parts. The Kafka delivery design explains why delivery and processing guarantees stop at different places.

send() returning proves that the client accepted work, not that Kafka did

KafkaProducer.send() is asynchronous: after serialization, partition selection, metadata work, and buffer admission, it normally returns a future before the broker result exists. A process failure can erase a buffered record before any Produce request reaches Kafka.

That makes “the method returned” a weak publication boundary. If the application needs a recoverable acceptance point, it needs a durable intent ledger: for example, a database transaction that commits both the business change and an outbox row. The producer buffer is not that ledger.

The distinction changes what you measure:

max.block.ms bounds specific blocking inside send(), including metadata and buffer allocation waits. It does not turn every serializer or partitioner call into the same clock. delivery.timeout.ms bounds reporting the send result after send() returns, including batching, request attempts, and acknowledgement waits. An application waiting 2 seconds on the returned future adds a fourth deadline; that wait expiring does not cancel protocol work.

Acknowledgement policies answer three different questions

An acknowledgement policy tells the producer which broker evidence must exist before a Produce request succeeds.

The current in-sync replica set (ISR) matters more than replication factor alone. With replication factor three, min-ISR two, and ISR {A,B,C}, acks=all waits for A, B, and C in the stable case. If C has already left the committed ISR and the current ISR is {A,B}, A and B can satisfy the request. Min-ISR is an admission and visibility floor; it does not mean “pick any two replicas.”

Kafka 4.3.1 adds a detail that older shorthand often misses. The pinned Partition.maybeIncrementLeaderHW implementation does not advance the high watermark while the effective ISR is below min-ISR. An acks=1 append can therefore return leader-local success while normal consumers still cannot read the record below the replicated boundary. The replication article owns the deeper ISR, high-watermark, leader-epoch, and eligible-replica mechanism.

A missing response preserves uncertainty after possible transmission

Classify the failure by the strongest evidence the caller has:

This is why catch (Exception) { newProducer.send(record); } is dangerous. Creating a new producer session and sending again changes the protocol identity. It may be the right recovery action, but only after the application decides how to recognize the original business intent.

Idempotence deduplicates a retried batch, not a repeated operation

Kafka producer idempotence uses a producer ID, producer epoch, partition, and per-record sequence. The broker can recognize the same retained batch and return its previous result without appending it again. The ProducerStateEntry.NUM_BATCHES_TO_RETAIN source shows a bounded five-batch metadata window per producer-partition.

That state does not hash payloads or understand an order ID. Three actions are not equivalent:

1. the producer retries the same in-flight batch;
2. the application calls send() again in the same session;
3. a restarted application creates a new producer and resends.

The first can reuse the original sequence. The second allocates a new valid sequence. The third normally uses a new producer identity. Both later actions can append another record for one business operation.

Run P07, then compare P08 and P09. The log-append count changes because the actions change protocol identity, not because the model reads a hard-coded “duplicate” label.

Effective Java settings belong in the publication contract

The Java 4.3 producer defaults to idempotence when no conflicting settings are supplied. It also defaults to acks=all, effectively unlimited configured retries, maximum in-flight five, and linger.ms=5. These are versioned Java defaults, not production recommendations.

Idempotence requires:

• acks=all;
• retries greater than zero;
• max.in.flight.requests.per.connection no greater than five.

If you explicitly enable idempotence with incompatible values, the Java client raises ConfigException. If you omit the idempotence setting and provide a conflict, the client can resolve it to disabled. Record the resolved configuration, client family, client version, serializer, partitioner, timeout ownership, and transactional-ID strategy. A checkbox in an application config file is not enough.

Here is a publication-result shape, deliberately shown as contract pseudocode rather than a drop-in wrapper:

record PublicationResult(
    String operationId,
    String clientProfile,
    String resolvedAcks,
    boolean resolvedIdempotence,
    Status status,
    TopicPartition partition,
    Long offset) {}
 
enum Status {
    REJECTED_BEFORE_DISPATCH,
    ACKNOWLEDGED,
    UNKNOWN_AFTER_POSSIBLE_TRANSMISSION
}

operationId belongs to the application. Partition and offset belong to one log occurrence. Producer ID/epoch/sequence belong to one protocol session. Keep all three identity layers when recovery can cross a process restart.

What could go wrong

• The application times out first and resends while the producer still retries. Two retry owners create new protocol work around an unresolved batch.
• acks=all is read as “fsync every configured replica.” Kafka relies on page cache and replication; its persistence design does not promise per-request flush on every device.
• A key routes differently after a partition or partitioner change. The per-partition sequence still works, but the business-order claim moved.
• A transactional ID is treated as an operation ID. It fences writer incarnations; it does not suppress two valid committed transactions carrying the same operation.
• A compatible broker inherits Apache semantics by name. Test the exact product, mode, and client. The portability audit keeps that evidence separate.

Write these seven lines before changing retry code

1. Accepted intent: what durable state proves the application accepted the operation?
2. Client profile: which library and exact version resolves the settings?
3. Acknowledgement: what does callback success establish?
4. Unknown result: which failures preserve UNKNOWN?
5. Retry owner: producer, application, relay, or framework?
6. Stable business identity: which key survives a new send and restart?
7. Reconciliation: which durable source can resolve an ambiguous outcome?

The producer can tell you whether one protocol attempt met its contract. It cannot tell you whether your business operation is complete.

Sources and references

• Apache Kafka 4.3 producer configuration
• KafkaProducer 4.3.1 source and method contracts
• Partition.scala 4.3.1 high-watermark logic
• Kafka delivery semantics
• Kafka persistence design
• ProducerStateEntry 4.3.1