Kafka reliability / Publishing lab

Kafka Publishing Reliability Lab

After this send, timeout, retry, or failure, what exists in Kafka—and what can the producer know?

Baseline: apache-java-4.3.1 · source 26b251a451ce

This deterministic teaching model runs in your browser and never connects to Kafka. It accepts only bounded synthetic controls. Curated scenario and variant IDs may appear in a shareable URL; custom local values, payloads, brokers, credentials, and production identifiers never do. Exports include only what you request.

Deterministic teaching model · apache-java-4.3.1

P07: Lost reply and identical idempotent retry

How can two transmissions produce one append?

Scenario

Curated variant

Change bounded local inputs

Changing these controls creates a custom local example. The share link keeps only the original curated preset; exports can include your bounded local values.

Replication factor

Fictional replicas A through E.

min-ISR

Teaching profile requires 1 ≤ min-ISR ≤ replication factor.

Acknowledgements

Producer waiting policy; it does not change consumer visibility.

Idempotence setting

The model displays the Java 4.3.1 resolved mode.

Teaching retry bound

0–5 attempts beyond the first; not the Java default.

Maximum in-flight

6 deliberately exposes an idempotence conflict.

Synthetic records per batch

Sequences advance by this record count.

Current outcome · step 0 of 13

No Kafka action has occurred; model state and caller evidence both begin empty.

curated preset

Producer

Resolved idempotence: enabled; The selected Java settings support idempotent production.
Session / next sequence: 1 / p0:0 p1:0 p2:0; Sequences advance by record count independently per partition and session.
Current batch: none
Caller publication knowledge: UNKNOWN

Partition log

Leader / epoch: A / 1
LEO / HW: 10 / 10; Exclusive next-offset boundaries for partition 0.
ISR / min-ISR / RF: ABC / 2 / 3
Log entries: 0; 0 append(s) currently name operation-a.

Observation

Response: none
Pre-append rejections: 0
Routing map: v1
Process: running

Invariant results

HW stays at or below leader LEO
holds in model
The committed boundary never crosses the modeled append boundary.
Producer settings alone protect one business operation
unknown
This finite trace has at most one batch for operation-a, but producer identity is not a business-operation ledger.
Under-min-ISR state does not advance HW
holds in model
The reducer advances HW only when the declared ISR reaches min-ISR.

Ordered trace

The state table shows model truth. Each observer result says what the current caller can establish at that step.

Ordered deterministic actions through step 0. Print and exports contain the complete bounded trace.
Step	Actor	Action	Observer result
10 ms	producer	Enqueue event-a	The record is accepted into producer memory. No broker or durable application-intent evidence exists yet.
25 ms	producer	Transmit selected batch	The batch is in flight; transmission does not yet establish append or acknowledgement.
310 ms	broker A	Append on leader A	The leader model appended the batch; producer completion still depends on the selected acknowledgement path.
420 ms	broker B	Follower B appends through leader LEO	The follower appends fetched bytes locally. The leader has not learned that new position yet.
525 ms	broker B	Follower B advertises its next offset	A later follower fetch advertises its next offset, so the leader can use that position as replication evidence.
636 ms	controller	Finalize stable ISR as A and B	Committed ISR membership becomes A, B for the next declared transition.
740 ms	leader	Recalculate the high watermark	HW advances to the minimum leader-known next offset across the current ISR: 11.
845 ms	broker A	Emit the Produce response	The broker emits success after the current ISR reaches the appended end boundary.
950 ms	network	Drop the Produce response	The model knows the broker emitted a result, but the producer/application did not observe it.
1060 ms	producer	Retry the identical batch	The producer retransmits the same session, epoch, and sequence range.
1165 ms	broker A	Broker checks the retried sequence	The retained producer sequence identifies the same batch; the broker returns its prior append result without another log entry.
1270 ms	broker A	Emit retry response	The broker emits success after the current ISR reaches the appended end boundary.
1375 ms	producer	Caller observes retry response	The caller observes success under the selected acknowledgement policy. This does not establish a business effect.

Evidence and limits

Claims

Primary sources

Assumptions and known limits

Apache Kafka and Java client 4.3.1 with stable committed ISR membership between declared transitions.
Offsets are small exclusive boundaries; follower append and leader knowledge are separate actions.

No full ELR election, pending ISR expansion, reassignment, filesystem, or correlated-failure engine.

Default synthetic example loaded.

Append truth and caller knowledge belong in separate columns

The reducer separates local buffer state, transmission, leader append, follower append, leader-known follower progress, HW, response delivery, and application observation. Same-batch retry reuses protocol identity; an application resend or new producer session creates new protocol work even when the payload names the same business operation.

Assumptions and limits

Feature levels, protocol versions, client settings, and binary versions remain separate facts.
Replication examples use stable committed ISR membership between declared transitions.
The ordinary transaction profile disables transaction.two.phase.commit.enable.
Browser traces are deterministic teaching models, not broker observations or production recommendations.