A Kafka Transaction Cannot Roll Back Your HTTP Request

A consumer charges a card, then loses its connection before it commits the Kafka offset. The group delivers the record again. The second handler cannot tell whether the first charge completed.

Aborting a Kafka transaction cannot pull the first HTTP request back across the network. Committing the offset first would avoid redelivery, but it could lose the charge when the process dies between the commit and the request.

The design question is not “Do we have exactly once?” It is: which identity and which transaction protect this specific effect?

This article uses five independent states:

1. source business intent;
2. Kafka publication occurrence;
3. consumer progress;
4. sink identity/result;
5. caller knowledge.

The positive examples protect local database state. External providers require their own retained operation-key and lookup contract.

A local outbox closes the source dual-write gap

Suppose a service updates an order row and publishes OrderAccepted.

A transactional outbox writes the business change and a publication intent in one local database transaction. The AWS pattern description states both the local atomicity benefit and the duplicate-publication obligation. It does not promise exactly one Kafka append.

A polling relay can publish, crash before marking its row complete, restart, and publish again. Producer idempotence does not join two new application sends or two producer sessions under one batch sequence. Keep a stable event and business-operation identity in the outbox row.

Change data capture can move the relay checkpoint into a database-log/connector contract. It also adds snapshots, source offsets, schema history, connector tasks, failover, and retention. The Debezium 3.3 outbox router source documentation defines event ID, aggregate key, and routing for that profile. It does not certify every connector, worker mode, or downstream sink.

Event identity and operation identity answer different questions

Use three labels:

• source occurrence: cluster/topic/partition/offset under a specific incarnation;
• event ID: one published fact or envelope;
• operation ID: one business action that may be represented by several events or attempts.

Two event IDs can request the same operation. An inbox keyed by event ID can correctly process each event once and still apply the business operation twice. The opposite error also happens: reusing one operation key for two intentional operations suppresses valid work.

The key therefore needs a semantic fingerprint. When a duplicate key arrives:

1. compare the fingerprint that defines the intended operation;
2. reuse the stored result only when the intent matches;
3. reject or reconcile a mismatch;
4. never call a mismatching key “deduplicated successfully.”

The AWS Builders' Library uses caller-provided request identity and semantic equivalence for the same reason.

An atomic inbox protects a local DB mutation

A transactional inbox stores a unique identity and the protected database mutation in one local transaction. Checking first and inserting later leaves a race; two consumers can both observe “missing” before either commits.

The following PostgreSQL-shaped example is contract pseudocode. It shows the required transaction boundary and conflict path; it is not a schema-independent library:

BEGIN;
 
INSERT INTO applied_operations (
  consumer_scope,
  operation_id,
  intent_fingerprint,
  result_json
)
VALUES (:consumer_scope, :operation_id, :fingerprint, NULL)
ON CONFLICT (consumer_scope, operation_id) DO NOTHING
RETURNING operation_id;                                      -- ①
 
-- If ① returned one row, this transaction owns the operation:
UPDATE account_balance
SET cents = cents + :delta_cents
WHERE account_id = :account_id;                              -- ②
 
UPDATE applied_operations
SET result_json = :result
WHERE consumer_scope = :consumer_scope
  AND operation_id = :operation_id;                          -- ③
 
COMMIT;

① The unique, scoped key chooses one local winner. PostgreSQL INSERT ... ON CONFLICT ... RETURNING provides the needed winner signal.

② The business mutation shares the same transaction. If the process or transaction fails before commit, PostgreSQL rolls back both the claim and mutation under its transaction semantics.

③ Store the fingerprint and result. A later duplicate must compare the fingerprint before reusing the result.

After local commit, the consumer can advance Kafka progress. A crash between the DB commit and offset commit permits redelivery, but the matching committed operation identity prevents another protected DB mutation.

Switch E03 to the check-then-insert variant. Two effects appear because the identity and mutation no longer share one winning transaction.

An inbox row cannot enclose an external provider

Move the mutation in the previous section to an HTTP API:

claim inbox identity
call provider
commit inbox
commit Kafka progress

No ordering closes every crash window:

The provider must participate through its own contract:

• stable operation key;
• atomic key + effect + stored result;
• semantic fingerprint conflict behavior;
• result lookup after timeout;
• retention long enough for every permitted delayed retry, DLQ redrive, failover, and restore;
• current authorization for the original or a separately approved new action.

That is provider idempotency, not Kafka EOS. Kafka can atomically commit output records and consumed offsets in the same cluster. The producer transaction API does not enlist an arbitrary HTTP server.

Retention has at least four independent clocks

“Dedup TTL is longer than topic retention” is not enough. Track:

1. source topic history, including compaction and transactional visibility;
2. coordinator-stored group offsets;
3. source outbox/CDC recovery history;
4. sink identity and stored-result history.

Those clocks can start at different events and restore to different points. Kafka's topic configuration and broker/group-offset configuration describe separate retention systems. A group checkpoint can disappear while source records remain. A DLQ or archive can outlive the source topic. A restored sink can forget an operation key while Kafka still retains the record.

Compaction adds another boundary. It can produce a current-state rebuild while removing older operations. It is not business-effect deduplication and does not reconstruct the historical sequence of effects.

Kafka transactions protect Kafka participants only

The protocol article on producer idempotence and transactions owns PID, epoch, sequence, coordinator decisions, markers, and LSO. For application design, use this narrower table:

The Streams configuration, Connect guide, and cross-cluster mirroring guide set different prerequisites. Do not compress them into one platform-wide claim.

What could go wrong

• The outbox event ID changes on relay retry. Duplicate publications no longer carry the same identity.
• The inbox protects event ID while two events request one operation. The database mutation repeats correctly per event but incorrectly per operation.
• A key match skips a fingerprint check. A conflicting request silently receives another operation's stored result.
• The provider retains keys for 24 hours while a DLQ retains work for 30 days. A late redrive can apply another effect.
• A sink reads aborted input. Transactional output cannot repair a decision made from an input record that should have been hidden.
• A local transaction is described as end-to-end atomicity. Kafka progress, provider truth, and caller knowledge still cross separate commits.

Choose the smallest transaction that can actually contain the effect. Then name every state outside it.

Sources and references

• Kafka 4.3 delivery semantics
• KafkaProducer 4.3.1 transactional API
• Kafka Connect 4.3 user guide
• Debezium 3.3 outbox event router source documentation
• PostgreSQL 18 INSERT
• PostgreSQL 18 transaction semantics
• AWS Builders' Library: Making retries safe with idempotent APIs
• AWS transactional outbox pattern