Kafka Consumer Groups Are Platform Infrastructure

The platform team can see that a Kafka reader is falling behind. Only the application team can tell whether processing the next record again will charge a customer twice.

A Kafka consumer is the client process that reads records. A consumer group coordinates several consumers so that one member owns each assigned partition, an ordered shard of a topic. The coordination machinery is shared infrastructure. The code that turns an OrderPlaced record into a shipment or a payment is not.

My position is deliberately narrow: the platform should own the consumer fleet, while the application owns the meaning of progress. Safe defaults, client releases, migration tooling, and fleet telemetry belong in a platform contract, the supported boundary between application code and shared Kafka machinery. Processing success, duplicate handling, and each side effect, an externally visible change such as a payment or database update, stay with the application.

An offset becomes business state when it moves past a side effect

Kafka tracks progress with an offset, the position of a record within a partition. An offset commit records the next position a group intends to read. The mechanism looks operational until the commit moves relative to a database write, HTTP request, email, or payment.

Kafka's delivery semantics describe the basic trade-off. Commit before processing and a crash can lose work. Commit after processing and a crash can repeat work. The second choice is at-least-once delivery: Kafka may deliver a record again when the work succeeded but progress did not reach the coordinator.

This pseudocode shows the gap. It is not a Kafka client API.

# Pseudocode, not a Kafka API
record = receive()
apply_business_side_effect(record)  # ①
acknowledge_progress(record)         # ②

① The application changes state outside the consumer, perhaps by charging a card or updating inventory.

② The consumer records that it finished the Kafka record.

A crash between ① and ② repeats the side effect after recovery. Reversing the lines avoids that duplicate but can lose the side effect. A wrapper or proxy can move the lines. It cannot erase the failure window.

Kafka transactions can atomically combine consumed offsets with records produced back to Kafka. KIP-98 defines that boundary. It does not enlist an unrelated database or HTTP service. External effects still need an application contract such as an idempotency key, a database transaction that includes a deduplication record, or compensation. Amazon's Builders' Library explains why safe retries need caller-visible intent. The wire-level transaction boundary is covered in Kafka Producer Idempotence, Fencing, and Transactions.

The platform can own the poll loop. It cannot authorize the business side effect.

One owner should manage the mechanisms that repeat across the fleet

Central ownership pays off where the same work appears in every service:

1. Safe defaults. The platform can standardize authentication, connection handling, telemetry, group membership, and supported timeout ranges. The application must still declare its longest processing operation and its duplicate-handling strategy.
2. Client lifecycle. The platform should inventory client implementations and versions, test supported upgrade paths, publish retirement dates, and roll changes through canary cohorts.
3. Migration machinery. KIP-848 changes group coordination and moves some policy to the broker. That is a fleet rollout problem, not a README sent to every repository. A later Evolution & Migrations post owns the KIP-848 migration details.
4. Fleet observability. The platform should collect group ownership, client version, rebalance, commit, and lag signals with consistent labels. Applications define the business deadline that turns delay into an incident.

A universal alert such as "1,000 records of lag is critical" has no stable meaning. In a simple hypothetical, 1,000 small events arriving at 50,000 records per second and 1,000 expensive jobs arriving at one per minute describe different risks. The platform owns the measurement. The application owns the objective.

The boundary becomes explicit in an ownership table:

Centralization without this table makes the platform team accountable for outcomes it cannot inspect.

Uber separated transport ownership from application truth

Uber's published systems show two different control points.

Its Consumer Proxy owns Kafka consumption for queue-like workloads and pushes individual messages to application endpoints. The proxy centralizes polling, flow control, retries, offset management, and dead-letter handling. Uber also documents the failure boundary: a rebalance can make the proxy fetch acknowledged but uncommitted records again, so duplicate processing remains possible.

That is an honest platform contract. The proxy controls redelivery. The application must tolerate it.

Uber's uGroup service centralizes another concern: visibility. It reads group activity from __consumer_offsets, supplements that stream with active broker queries, and reports lag, stuck partitions, group activity, and ownership information. It does not need to run each application's handler to make the fleet observable.

The two systems support the same conclusion from opposite directions: centralize repeatable transport and evidence, not business meaning.

Choose the thinnest platform that can still move the fleet

There are three common models. A direct client lets each application use a Kafka library itself. A blessed shared library wraps a supported client with defaults and instrumentation. A consumer proxy removes the Kafka client from the application process and exposes a queue-like handler interface.

My default is the shared library, not because it is always best, but because it preserves Kafka's useful application controls while giving the platform one release vehicle. Choose a proxy when the queue-like contract is real, not as a symbol of platform maturity. Choose direct clients when partition ownership is part of the application algorithm.

The recommendation stops working when the wrapper blocks required features, when languages diverge faster than the platform can support them, or when an application needs direct assignment and offset control. Publish an escape path before teams create an unofficial one.

What could go wrong when the platform owns the client?

Centralization removes repeated work. It also concentrates mistakes.

A proxy can turn one overload into a fleet incident

A proxy sits in every record's processing path. Its capacity and retry policy therefore become shared dependencies. Google's SRE guidance shows how overload and retries can create cascading failure.

Isolate workload classes, bound retries, shed load before exhaustion, and roll proxy changes through independent cells. If the organization cannot fund that failure isolation, keep policy in a shared library and leave record movement in the application process.

A safe default can become a correlated defect

Standardization lowers variance, but one defective timeout, commit mode, or retry policy can reach the whole fleet. Google's configuration guidance treats configuration as a reliability interface, and its release guidance defines canarying as a partial, time-limited deployment.

Keep more than one supported client release during rollouts. Record the resolved configuration, not only the intended template. "Blessed" should mean tested and supportable, not deployed everywhere at once.

A wrapper can delay the feature it was meant to deliver

KIP-848 is a good example. Central ownership can test the migration once and move groups consistently. It can also block every application until the shared library or proxy exposes the new protocol.

Set a support window, test new client releases continuously, and allow a reviewed direct-client path. A platform contract without an exception process eventually becomes a bypass process.

Hidden Kafka details still surface during incidents

Applications still experience assignment changes, duplicate processing, partition-local ordering, and expired offsets even when a wrapper hides the callbacks. Preserve raw client metrics, group IDs, topic-partition offsets, and broker errors. An abstraction that removes diagnostic evidence is not simpler; it is merely harder to debug.

Write the contract before writing the wrapper

The platform contract should publish:

• supported clients, versions, and retirement dates;
• delivery modes and their failure behavior;
• default profiles, override rules, and review paths;
• fleet telemetry, ownership metadata, and migration procedures;
• rollback and incident responsibilities for shared components.

Each application should declare:

• what successful processing means;
• where progress may advance relative to every side effect;
• whether duplicate work is harmless, deduplicated, or compensated;
• what happens when a committed offset or retained record is unavailable;
• the business delay objective and the owner who responds when it is missed.

"Exactly once" is not enough. Name the transaction boundary and every system inside it. If an external effect sits outside that boundary, document its retry and duplicate contract.

Kafka consumer groups are platform infrastructure. Kafka consumers remain business programs. The ownership model works only when both statements stay true at the same time.

Next, the course moves from organizational ownership to the point of no return in a ZooKeeper to KRaft cutover.

Sources

Kafka behavior

• Kafka message delivery semantics
• KIP-98: Exactly Once Delivery and Transactional Messaging
• KIP-848: The Next Generation of the Consumer Rebalance Protocol

Platform implementations

• Uber Engineering: Kafka async queuing with Consumer Proxy
• Uber Engineering: uGroup consumer monitoring

Reliability boundaries

• Amazon Builders' Library: Making retries safe with idempotent APIs
• Google SRE: Addressing Cascading Failures
• Google SRE: Configuration Design and Best Practices
• Google SRE: Canarying Releases