Kafka reliability / Consumer lab
Kafka Consumer Progress & Ownership Lab
Which offset can you commit without skipping unfinished work, and what changes when ownership moves?
Baseline: apache-java-4.3.1 · source 26b251a451ce
This deterministic teaching model runs in your browser and never connects to Kafka. It accepts only bounded synthetic controls. Curated scenario and variant IDs may appear in a shareable URL; custom local values, payloads, brokers, credentials, and production identifiers never do. Exports include only what you request.
Deterministic teaching model · apache-java-4.3.1
C02: The first unfinished record caps the frontier
Can completed offset 12 justify skipping pending 11?
Current outcome · step 0 of 6
Fetched, completed, requested, and coordinator-stored progress all begin at next offset 10.
Default synthetic example loaded.
Progress boundaries
- Partition 0
- fetched 10 / frontier 10 / stored 10
- No unfinished delivered record in the current ownership ledger.
Ownership
- Member / token
- consumer-a / epoch-1
- Assignment
- assigned
- Consumer state
- accepting
- Discarded stale completions
- 0
Effects and disposition
- In-flight records
- 0
- Bound: 6.
- Stale external effects
- 0
- Total unprotected effects
- 0
- Unsafe commits / regressions
- 0 / 0
- Poison stage
- none
- Durable disposition
- not established
Invariant results
Stored progress does not cross unfinished owned work
holds in model
Every stored next offset stays at or below the current completed frontier.
Stale completion cannot mutate the current local ledger
holds in model
Completion actions carry an ownership token; old tokens are discarded locally.
In-flight admission remains bounded
holds in model
Current in-flight work stays within 6.
Poison data advances only after an explicit durable disposition
not applicable
No poison record occurred.
Ordered trace
The state table shows model truth. Each observer result says what the current caller can establish at that step.
| Step | Actor | Action | Observer result |
|---|---|---|---|
| 10 ms | consumer thread | Poll offsets 10, 11, and 12 | Poll delivered offsets 10, 11, 12 and advanced fetched next position to 13. |
| 210 ms | worker | Complete offset 10 in partition 0 | Offset 10 completed under epoch-1. |
| 330 ms | worker | Complete offset 12 in partition 0 | Offset 12 completed under epoch-1. |
| 440 ms | consumer thread | Request commit snapshot frontier-11 | Commit snapshot frontier-11 requests next offset 11; no coordinator result has been observed yet. |
| 550 ms | coordinator | Store commit snapshot frontier-11 | Coordinator stores next offset 11 from snapshot frontier-11. |
| 620 ms | worker | Complete offset 11 in partition 0 | Offset 11 completed under epoch-1. |
Evidence and limits
Assumptions and known limits
- Traditional consumer-group progress uses next offsets and an ownership-qualified delivered-work ledger.
- The consumer-owning thread performs KafkaConsumer operations; workers only report bounded results.
- No complete group coordinator, assignor, heartbeat scheduler, or downstream resource fence.
Fetched, completed, requested, and stored progress are four states
The traditional-group reducer commits a next-offset boundary from a complete ownership-qualified delivery ledger. Sparse source offsets do not become invented handler holes. Revocation, loss, stale local completion, stale group progress, and external effects stay distinct. Share acquisition uses a separate reducer and appears only after its pinned runtime gate passes.
Assumptions and limits
- Feature levels, protocol versions, client settings, and binary versions remain separate facts.
- Replication examples use stable committed ISR membership between declared transitions.
- The ordinary transaction profile disables transaction.two.phase.commit.enable.
- Browser traces are deterministic teaching models, not broker observations or production recommendations.