Kafka Lag Is Not a Completion SLO

The dashboard says consumer lag is zero. A customer operation has still been waiting for 18 minutes.

The consumer fetched the record. It may even have committed the next offset. Neither observation proves that an asynchronous handler finished or that an external provider applied the effect.

A service-level objective (SLO) needs a declared population, start boundary, completion boundary, deadline, and observer. An offset difference supplies none of those by itself.

Every useful metric answers one narrow question

Treat each observation as a contract:

The Kafka 4.3 monitoring guide documents Java and broker metrics such as producer record errors, retries, and queue time. Those names belong to a client/version profile. They are not a cross-client standard and do not count business operations.

Offset boundaries are not record counts

Kafka's log end offset (LEO), high watermark (HW), and last stable offset (LSO) are exclusive next-offset boundaries:

logStart <= LSO <= HW <= LEO

The UnifiedLog 4.3.1 source bounds LSO by HW and the first unstable transaction. Control records occupy offsets. Aborted transactional records disappear from read_committed application output. Compaction can remove older key versions.

Therefore:

end offset - committed offset != visible record count

A difference of 1,000 can contain fewer than 1,000 application records. It can also include data that a committed-only reader cannot yet receive. Converting that difference to seconds using yesterday's average record rate hides key skew, burst shape, transaction state, and handler concurrency.

Use offsets as coordinates. Count operations with an operation observer.

Client fetch lag and stored group lag use different positions

The Java client tracks a fetched position and reports fetch lag against an end boundary. For read_committed, the KafkaConsumer implementation uses the stable boundary rather than delivering past an open transaction.

The group coordinator stores committed next offsets. A client can fetch ahead of that checkpoint. An application can process concurrently and complete behind both. The consumer progress reference keeps all four states visible:

1. fetched next;
2. completed owned frontier;
3. requested commit snapshot;
4. coordinator-stored next offset.

Choose which one your dashboard shows. Label it in the metric name and help text. “Lag” without the position and end boundary is not an operating instruction.

Completion starts when accepted intent becomes observable

For an operation deadline, define:

• population: every accepted operation in the measurement window;
• start: durable intent commit, accepted API response, or another named boundary;
• complete: business state reaches a verifiable terminal outcome;
• deadline: for example, 5 minutes from accepted intent;
• unfinished rule: overdue unfinished operations count as misses;
• observer coverage: the measurement includes stuck work, not only work that eventually completed;
• identity: one event or one business operation;
• dimensions: bounded labels that don't expose customer payloads.

If the histogram includes only completed handlers, the worst stuck operations never enter the sample. A p99 can look healthy while an unbounded tail waits forever. The Job Completion SLO Lab owns the generic cohort math and burn-rate exercise.

Kafka-specific instrumentation supplies the accepted intent, source coordinates, visibility, group progress, ownership, and retry-path context. It does not replace the completion cohort.

Partition skew defeats cluster-wide averages

Traditional groups assign one partition to one member at a time. More consumer instances than partitions do not increase partition ownership concurrency. Application handlers can add controlled concurrency within a partition, but then they need the completed-prefix ledger.

A cluster average can hide:

• one hot key on one partition;
• one partition held behind an open transaction;
• one consumer repeatedly losing ownership;
• one downstream dependency saturated by a subset of keys;
• one payload shape that exceeds normal byte budgets;
• one retry loop consuming the deadline.

The quota article owns broker/client quota mechanics. The backlog calculator owns a bounded fluid estimate. Neither tool should treat “add workers” as unlimited Kafka partition capacity.

A diagnostic packet needs units, sources, and counted identity

Use a compact incident packet:

accepted intent:
  counter: accepted_operations_total
  counted unit: business operation
  identity: operation_id
  observer: source database outbox commit
 
Kafka publication:
  client: apache-java 4.3.1
  effective acks: all
  resolved idempotence: enabled
  result: callback success / error / UNKNOWN
 
partition snapshot:
  source incarnation: cluster-a/orders-v3
  partition: 7
  logStart / LSO / HW / LEO: 120 / 140 / 150 / 153
  observed at: explicit timestamp
 
consumer:
  group protocol: classic
  fetched / completed / requested / stored: named separately
  in-flight records and bytes: bounded
  ownership incarnation: recorded
 
business completion:
  deadline: 300000 ms from accepted intent
  terminal observer: provider lookup + local result record
  unfinished overdue operations: included

Every value needs a unit, source, version, labels, observation time, and counted identity. Topic/group labels can be useful in a real operator system; this website does not collect them.

Rollouts change several contracts at once

A Kafka upgrade ticket can contain independent changes:

1. broker/client binary versions;
2. finalized feature levels;
3. traditional group protocol conversion;
4. transaction protocol activation;
5. serializer/schema meaning;
6. partitioner or key routing;
7. application concurrency/progress policy.

Canary one change with explicit stop conditions. The KRaft migration article owns metadata authority and feature finalization. The KIP-848 migration article owns group-protocol rollout. The job payload compatibility lab provides a generic mixed-reader example; Kafka adds durable old records, partition keys, transaction visibility, and replay.

What could go wrong

• Zero stored-offset lag is treated as zero unfinished work. Async handlers or external effects can remain incomplete.
• HW−LSO is paged as replication failure. A long open transaction can be the actual cause.
• Offset span becomes record count. Control, aborted, and compacted gaps break the conversion.
• Cluster average hides one hot partition. The business deadline fails for one key range while the fleet average stays calm.
• Completed-only latency excludes stuck work. The SLO rewards the observer for losing the worst operations.
• Metric names move across clients unchanged. Java JMX names and semantics are treated as a universal SDK contract.
• A rollout changes protocol and schema together. The team cannot attribute the failure or use a clean stop condition.

The first useful dashboard question is not “What is lag?” It is “Which boundary, counted identity, and unfinished population does this number observe?”

Sources and references

• Kafka 4.3 monitoring
• KafkaConsumer 4.3.1
• UnifiedLog 4.3.1 LSO calculation
• Kafka 4.3 message format
• Kafka 4.3 consumer configuration
• Kafka 4.3 producer configuration