Redpanda Architecture: Kafka Outside, Shards Inside

You point a Kafka client, an application that speaks Kafka's wire protocol, the byte-level request-and-response contract, at a Redpanda broker, the server that handles requests and hosts replicated logs. The client sees Kafka. The broker does not look like Kafka inside.

The client's ApiVersions request discovers the Kafka API versions the broker supports. Metadata, the map of topics, brokers, leaders, and each partition, one ordered log within a topic, arrives. A producer, the client that writes records, sends its first request through the replica authorized to accept it.

That boundary is the useful way to study Redpanda.

Redpanda is a Kafka-compatible system, not a reimplementation of Apache Kafka's process model. Its public contract is a versioned subset of the Kafka protocol. Behind that contract, Redpanda uses Seastar, an asynchronous C++ framework, sharded services, a Raft group, replicas agreeing on one ordered log, for each partition, and a separate cluster-wide Raft group for metadata.

This series pins implementation claims to Redpanda v26.1.13 at commit 90d2d87a52c31c0441e93a06986b5eda8afe77f7. Product behavior follows the documentation snapshot at bd29393970ff676dcfb60c1c34b9db91b7a694b6, dated July 22, 2026. I rechecked the evidence on August 2, 2026. Unless a section says otherwise, “Redpanda” means self-managed Redpanda.

Kafka compatibility is negotiated, not absolute

A Kafka API key identifies an operation such as Produce, Fetch, or ApiVersions. Each API has its own version range. Redpanda's ApiVersions handler builds the advertised response from the handlers compiled into the broker, then filters APIs according to enabled features such as idempotence and transactions.

The supported handler list is explicit in handlers.h. This is a better source than “100% Kafka compatible”: a client and broker must share a version for every API the application actually uses.

Redpanda's pinned compatibility guide says clients built for Kafka 0.11 and later are compatible, subject to documented exceptions. The Kafka-client exceptions include:

• one SCRAM mechanism, a password-based SASL authentication method, per user rather than simultaneous SHA-256 and SHA-512 credentials;
• no server-side KIP-890 transaction protocol, so Kafka 4.x clients fall back to the earlier transaction protocol.

Quota compatibility shows why a versioned source check matters. The generic compatibility page still groups user-scoped bandwidth and API request-rate quotas as unsupported. For v26.1.13, that combined bullet is only partly current. The tagged user_based_client_quota feature became available in 26.1.1, and the pinned throughput guide documents authenticated-user and client-ID quotas through Kafka's AlterClientQuotas and DescribeClientQuotas APIs.

The tagged client_quotas handler accepts user, client ID, a self-declared request label, client-ID-prefix, and user-plus-client entities for producer_byte_rate, consumer_byte_rate, and controller_mutation_rate. It does not expose Kafka's request_percentage key or IP quota entities. Client IDs remain self-declared labels; use authenticated-user quotas when identity matters, and remember that neither form isolates shared disk, CPU, or recovery bandwidth.

Redpanda HTTP Proxy is the product's REST produce/consume surface, not a Kafka client. Its pinned limitation is that topic and ACL CRUD are not available through the proxy. That says nothing about whether a Kafka Admin client can use the corresponding Kafka APIs.

Compatibility therefore has three levels:

For a migration, test the application's actual API trace. A successful console producer says little about transactions, quotas, consumer-group assignment, or admin tooling.

The detailed byte-level negotiation belongs to the Kafka protocol material. Here, the important point is architectural: Redpanda preserves the interface while changing the machinery behind it.

One request can cross a core before it reaches a partition

A shard is Redpanda's name for the work and state assigned to one Seastar core. The simplified request path is:

Kafka connection
  → versioned API handler
  → shard lookup
  → optional cross-shard message
  → partition state machine
  → Raft
  → local log

The handler does not own every target partition. Redpanda keeps a shard_table that maps a namespace-topic-partition to its local core. The consumer-group path documents the pattern directly: group_router maps a coordinator key to a partition, finds the owning shard, then invokes the core-local group manager.

That design avoids putting one shared, locked partition map in every request path. It does not make cross-core work free. A request that lands away from its owner pays for queueing, message transfer, and scheduling on the destination shard.

This distinction corrects two common summaries:

1. Redpanda does not guarantee that a connection always runs on the same core as every partition it accesses.
2. “Thread per core” does not mean “no coordination.” It changes coordination from implicit shared-memory access to explicit messages and sharded state.

Thread per core removes some locks and creates new obligations

Redpanda v26.1.13 pins its Seastar dependency to commit 7e67971d789c6642de7ec6225d37fabaa197b184. At that revision, Seastar describes a single-threaded fast path per core, explicit inter-core messages, and future-based I/O.

A reactor is the event loop running on one core. A sharded service creates one service instance per core; Seastar's sharded<T> model provides invoke_on() and invoke_on_all() for work that must cross those instances.

This model has a clear trade:

Seastar is cooperative, not magically non-blocking. Its pinned tutorial warns that CPU work without a yield point can cause a reactor stall. Coroutines check for preemption at co_await, and loop helpers insert yield points, but application code still has to cooperate.

The operating lesson is narrower than “more cores means linear throughput.” More cores increase available execution capacity only when partitions, connections, cross-shard traffic, disk, and network distribute the work.

The lab below makes core ownership explicit without inventing utilization figures. Choose the connection core and target partition; the board computes whether the request needs a cross-shard message.

If the connection core and partition shard differ, explicit message passing enters the path. The ownership equation identifies the hop; a shard-labelled trace still has to measure its queueing cost.

Two Raft scopes keep data and metadata separate

Each user partition forms a Raft group. That group elects one leader and replicates the partition log according to its replication factor, as the pinned architecture documentation describes.

Cluster metadata uses another group. Redpanda calls it raft0: group ID zero for the controller partition. The pinned create_raft0() source creates that group around model::controller_ntp. The architectural boundary is enough here: partition Raft owns user-log consensus; raft0 orders cluster-wide metadata. Their acknowledgement, flush, recovery, and membership boundaries remain distinct and should be diagnosed separately.

Removing ZooKeeper removes a separate product and protocol. It does not remove a control plane. Controller quorum, metadata snapshots, membership changes, and feature activation still need their own monitoring and recovery plans.

Apache Kafka reached a similar operational conclusion through KRaft. That migration history belongs to the Kafka 4.x material; repeating it here would hide what is specific to Redpanda.

The binary contains more than one product boundary

Redpanda's repository is source-available, but it is not uniformly Apache-licensed. The repository's license FAQ separates two code classes:

The pinned licensing guide lists which features require a key and what happens after expiration. Source headers make the split visible too: transform_processor.h uses the BSL header, while cloud_storage/remote.h uses the enterprise RCL header.

That table is also an operations boundary. The source paths, tunables, and Admin API runbooks in this series describe self-managed Redpanda. Redpanda Cloud is a managed Enterprise deployment, so a Kafka client may remain portable while broker configuration, upgrades, host access, and service limits do not.

“Open source Kafka replacement” erases both the license terms and the product edition. A technical evaluation should name the exact binary, license, enabled features, and support model.

What could go wrong

The source and documentation above imply four failure modes worth testing.

A client passes the smoke test and fails on the real API

ApiVersions negotiation proves a schema overlap, not feature equivalence. Run the oldest and newest supported client through authentication, metadata, produce, fetch, offset commit, groups, transactions, and every administrative operation the application needs. Keep the pinned unsupported list beside that test matrix.

Cross-shard work becomes the hidden queue

A balanced broker-level partition count can still produce a hot shard. Inspect metrics with the shard label and compare request latency with cross-core work. Adding partitions can help only when the partitioning scheme spreads the hot keys rather than multiplying metadata.

Cooperative work stops cooperating

Long computation without preemption blocks the core's event loop. Watch reactor stall logs and shard CPU beside application latency. Moving the same work to a lower scheduling group changes priority; it does not make the work disappear.

Procurement assumes a Community feature is free to operate

BSL permits production use under its additional grant but restricts offering Redpanda as a commercial streaming or queuing service. Enterprise features need an RCL-backed license key. Confirm the legal and product boundary before architecture depends on Tiered Storage, continuous balancing, or Cloud Topics.

Reproduce the source map

The following commands pin the repository and show the main boundaries used in this post:

git clone --branch v26.1.13 --depth 1 \
  https://github.com/redpanda-data/redpanda.git redpanda-v26.1.13
cd redpanda-v26.1.13
 
git grep -n "using request_types" -- src/v/kafka/server/handlers/handlers.h
git grep -n "get_supported_apis" -- src/v/kafka/server/handlers/api_versions.cc
git grep -n "owning shard" -- src/v/kafka/server/group_router.h
git grep -n "create_raft0" -- src/v/cluster/raft0_utils.h
git grep -n "user_based_client_quota" -- src/v/features/feature_table.h
git grep -n 'name = "seastar"' -- bazel/repositories.bzl

The trace does not benchmark Redpanda. It establishes which code implements the claims.

The architecture is a contract plus a queueing model

• Kafka compatibility is per API and per version, with documented feature gaps.
• Thread per core replaces much shared-memory coordination with sharded state and messages; cross-shard work and reactor stalls remain real costs.
• User data and cluster metadata use different Raft groups.
• ZooKeeper-free does not mean control-plane-free.
• Community, Enterprise, and Cloud boundaries change both licensing and operations.

See the full Redpanda reading order →

Sources

• Redpanda v26.1.13 release
• Pinned architecture docs
• Pinned Kafka compatibility reference
• Pinned throughput-quota guide
• Pinned licensing reference
• Kafka handler registry at v26.1.13
• ApiVersions implementation at v26.1.13
• Feature table at v26.1.13
• Pinned Seastar dependency
• Live Redpanda documentation — secondary, moving reference