A Kafka DLQ Write and Offset Commit Are Two Operations—Unless You Join Them

Offset 10 fails decoding. The handler writes a dead-letter record and commits 11. A crash between those operations decides whether the failed work disappears from normal recovery or appears in quarantine twice.

A dead-letter queue (DLQ) is only durable disposition when the quarantine record, source progress, provenance, and later redrive contract all hold. Plain KafkaConsumer does not create that contract automatically.

This guide uses a stricter recovery question: what evidence and consistency boundary must exist before we retry, quarantine, or replay? Its strongest result is “candidate for human review,” never “safe to run.”

Commit-first loses disposition; publish-first can duplicate it

Start with one failed source record at offset 10. The next source offset is 11.

The first branch skips work without disposition. The second duplicates quarantine, which is not automatically a duplicate business effect. Preserve the original cluster/topic/partition/offset, event ID, operation ID, schema, failure stage, and attempt history so later review can identify repeated quarantine.

The Kafka producer transaction API can include output records and consumed next offsets in one same-cluster transaction. The positive DLQ path needs:

• an ordinary transactional producer with transaction.two.phase.commit.enable=false;
• current consumer group metadata;
• the source next offset, not the failed offset;
• a DLQ encoding that succeeded;
• durable topic/internal-topic settings;
• committed-only readers when upstream transactions matter;
• abort recovery that restores the input position.

It does not make the later redrive effect atomic.

Compare R01, R02, R03, and R04. The state changes through reducer actions: source progress, quarantine visibility, transaction decision, caller knowledge, and restored input position remain independent.

Crash before commit and lost commit response are not the same history

If the process crashes before issuing a commit request, the bounded ordinary transaction can later resolve as abort. The DLQ record remains hidden, source progress remains 10, and recovery returns to the failed input.

If the coordinator accepts commit and the response disappears, the caller is UNKNOWN. The transaction may already have committed both participants. commitTransaction()'s method contract requires retrying the same commit operation or closing and recovering. Switching to abort because the response was late can contradict the terminal decision.

The same rule applies to an uncertain abort: repeat abort or close. Do not build one catch (KafkaException) { abort(); retryEverything(); } policy across serialization, authorization, fencing, timeout, and stale group metadata.

Retry topics trade immediate progress for ordering

A retry topic removes the failed record from the original processing path and allows later original-partition records to advance. That can be correct, but it changes the order contract.

Suppose offsets 10 and 11 share one account key:

1. offset 10 fails and moves to a 5-minute retry topic;
2. source progress advances to 11;
3. offset 11 applies immediately;
4. offset 10 returns later.

The original partition order has reversed at the business effect. Choose one policy:

Kafka does not imply a per-record delay scheduler, priority queue, or workflow state machine because a team created orders-retry-5m.

Retry budgets name owners and one absolute deadline

Broker/client retry, producer retry, application resend, handler retry, framework redelivery, and operator redrive are not automatically independent. Write one owner per boundary.

If three layers each allow two total attempts, the mechanical upper bound is 2 × 2 × 2 = 8. That bound applies only when every layer fully nests inside the next one. An absolute deadline can clip it; backpressure can prevent later attempts from starting; a non-retriable error can stop the chain.

More attempts do not resolve an unknown effect. They only create more chances to repeat it. Use the existing Retry Budget & Amplification Workbench for the bounded timeline, then return here for Kafka-specific identity, order, progress, and visibility.

Replay readiness depends on four retention systems

Retained source records do not establish a complete replay. Review:

1. topic history and compaction;
2. coordinator-stored group checkpoints and reset policy;
3. source outbox/CDC history;
4. sink identity/result retention.

The topic configuration and consumer reset configuration answer different parts. A checkpoint can expire while data remains. A compacted topic can retain current keys while omitting historical operations. An open transaction can hold read_committed visibility below the high watermark.

At the equal boundary, define the rule. If protection is valid through day 30, does replay at the start of day 30 remain covered, or does expiry happen first? Tests need one deterministic ordering. Production stores need a documented clock, time zone, restore behavior, and key-reuse policy.

Original intent and a new operation need different authority

Redriving the original business action should preserve its operation identity. Changing the key to bypass a duplicate check creates a new operation from the sink's perspective.

Sometimes a new operation is correct: an operator may approve a replacement payment, email, or credit after reconciliation. Record that as a separate authorized action with a new operation ID and a link to the original. Do not smuggle it through as “the same retry with a fresh key.”

If effect truth or current authority is unknown, stop automatic effect-capable replay. A compensation is also a new authorized effect, not a rollback that Kafka performs.

A replay manifest records questions, not commands

The planner exports a synthetic manifest shaped like this:

{
  "formatVersion": 1,
  "synthetic": true,
  "baselineId": "apache-java-4.3.1",
  "sourceIncarnation": "teaching-cluster-a/topic-a-v1",
  "ranges": [
    {
      "partition": 0,
      "startInclusive": 10,
      "endExclusive": 11,
      "isolation": "read_committed",
      "completeness": "unknown"
    }
  ],
  "intentPolicy": "preserve-original",
  "stopConditionIds": ["stop-on-identity-conflict"],
  "verificationIds": ["verify-business-effect-count"]
}

The range is half-open: include 10, stop before 11. Real Kafka offsets can exceed JavaScript's safe integer range, so broker-derived evidence must keep them as decimal strings. The browser model stays inside small teaching integers.

The export contains no kafka-consumer-groups --reset-offsets, topic delete, produce, SQL mutation, or provider call. It records:

• source and target reader versions;
• schema interpretation;
• original coordinates and identity;
• retention evidence;
• dry-run limits;
• owner and approver roles;
• canary range and rate budget;
• stop conditions;
• business-state verification after progress advances.

Use the general Replay Readiness Checklist for the cross-system review and Dead-Letter Queue Triage for the next evidence-gathering step.

Cross-cluster recovery needs another authority model

MirrorMaker can replicate records and, in a documented dedicated setup, use an EOS mode for the replication leg. The Kafka 4.3 geo-replication guide does not turn a source application transaction into one cross-cluster commit.

Failover must reconcile:

• which records arrived;
• which checkpoints translated;
• which schemas and keys arrived;
• which sink remembers operation identities;
• which cluster may accept authoritative writes;
• how failback avoids dual authority.

R12 therefore returns OUTSIDE_MODEL. It links the portability boundary audit instead of exporting a same-cluster “safe failover” plan.

The recorded F09 fixture narrows one example: pinned Kafka 4.3.1 MirrorMaker copied three synthetic records, synchronized an inactive group at next offset two, and let a target reader resume after the owned source brokers stopped. That profile enabled no target writer and tested no failback. It is broker evidence for one manual read cutover, not permission to reuse the schedule for another topology.

What could go wrong

• The DLQ payload cannot encode the poison record. Committing progress anyway recreates the commit-first loss window.
• The DLQ reader uses read_uncommitted. It can act on a quarantine record from an aborted disposition transaction.
• A retry topic keeps source progress moving but no key-order policy exists. Later changes overtake earlier ones.
• Topic retention exists but the sink's operation keys expired. Replay can repeat effects.
• A dry run reads bytes successfully and is called proof. It did not verify current authority, provider state, capacity, or complete history.
• A copied reset command targets the wrong group or cluster. This guide never emits live commands or accepts a bootstrap server.

Recovery begins when the system admits what it does not know. Keep that uncertainty visible until evidence, identity, and authority resolve it.

Sources and references

• KafkaProducer 4.3.1 transactions and terminal operations
• KafkaConsumer 4.3.1 progress and isolation
• Kafka 4.3 topic configuration
• Kafka 4.3 consumer configuration
• Kafka Streams 4.3 configuration
• Kafka Connect 4.3 user guide
• Kafka 4.3 cross-cluster mirroring