Kafka Compatibility Is Not the Same as Portability
- Published on
- /16 mins read
Hypothetical: A staging producer accepts a new bootstrap.servers value, authenticates, writes one record, and reads it back. The migration ticket turns green.
The rollback drill asks harder questions. Does acks=all, the producer mode that waits for every current in-sync replica, prove the same durability event? Can a read_committed consumer hide the same aborted records? Did the destination receive the consumer checkpoint before the source group stopped? If record files live in your object-storage bucket, can you reconstruct their topic, partition, and offset order without the vendor's metadata service?
Kafka compatibility means that some Kafka clients and tools can communicate with a target. Portability means that a workload can move while preserving its required behavior, state, recovery objectives, operating model, and commercial boundary.
The previous acquisition post grouped an exit into four planning planes. This final audit splits the application plane into client/API and behavioral semantics, then splits operations into durability/storage and control-plane work. The result is six test layers rather than a competing taxonomy.
Compatibility is an entrance test. Portability is a rehearsed exit.
For the byte-level contract behind the first test, read Kafka Wire Protocol: How Compatibility Survives Upgrades. This post starts where a successful handshake stops.
A green client test covers one layer out of six
A wire protocol is the binary request and response contract. An API surface is the set of operations and versions a server implements. A control plane creates, configures, secures, and coordinates the service around the record path.
Recovery point objective (RPO) is the maximum acceptable data loss. Recovery time objective (RTO) is the maximum acceptable outage or degraded period.
The portability test has six layers:
| Layer | Question | Evidence that counts |
|---|---|---|
| Client and API | Can every production client, authentication path, codec, and required Admin API work? | ApiVersions, integration tests, malformed-request tests, effective-config reads |
| Behavioral semantics | Do ordering, retries, idempotence, transactions, timestamps, errors, retention, and groups preserve application invariants? | Failure tests against declared invariants |
| State | Can records, offsets, schemas, access rules, topic settings, and processing state move together? | Full-volume migration and rollback rehearsal |
| Durability and storage | What event makes a write durable, and what metadata is needed to recover it? | Acknowledgement-path analysis, fault injection, restore drill |
| Operations and control plane | Can the team monitor, scale, upgrade, diagnose, secure, and recover the target? | Runbooks executed by the owning on-call team |
| Contract and economics | Do license rights, service terms, support, commitments, data transfer, and exit cost fit? | Governing terms and a peak, replay, dual-run, and termination cost model |
A wire-compatible endpoint can remove an application rewrite. It cannot answer the other five rows.
The survey uses current first-party evidence, not a conformance badge
The product pins below reflect primary sources checked on August 1, 2026:
- Apache Kafka 4.3.1;
- Redpanda 26.2.1;
- WarpStream Agent v826;
- AutoMQ OSS 1.7.2, based on Kafka 3.9.1;
- Amazon MSK 4.2.x on Express brokers;
- Google Managed Service for Apache Kafka 3.7.1;
- current Confluent Cloud documentation for Kora;
- current Azure Event Hubs Kafka documentation.
I did not run a vendor-neutral conformance suite against these systems. Vendor tests and compatibility statements are evidence of vendor intent, not third-party certification.
The decision rule is strict: when a pinned source does not establish a required feature, or first-party pages conflict, classify the feature as unverified. For a correctness requirement, unverified means unavailable until the exact target advertises and passes it.
Different implementations preserve different parts of Kafka
The systems fall into distinct architecture classes:
| Target class | Current example | What compatibility can preserve | Portability boundary |
|---|---|---|---|
| Reference implementation | Apache Kafka 4.3.1 | The upstream protocol, coordinator, storage, and feature-level model | You still own operations, upgrades, recovery, and infrastructure |
| Managed named Kafka release | Amazon MSK 4.2.x; Google Managed Kafka 3.7.1 | A known upstream code line and broad client behavior | Provider identity, storage access, configs, versions, metrics, and service APIs |
| Independent compatible broker | Redpanda 26.2.1 | Core Kafka clients, produce, consume, classic groups, and transactions | Different replication, storage, feature set, configs, and license |
| Object-storage-first engine | WarpStream Agent v826 | Kafka clients, groups, transactions in normal mode, and selected Admin APIs | Vendor metadata defines offsets and ordering; operating modes can change semantics |
| Kafka-derived storage replacement | AutoMQ OSS 1.7.2 | Large parts of Kafka 3.9.1 compute and coordinator behavior | S3Stream, write-ahead log, object metadata, and a 3.9.1 source base |
| Proprietary managed Kafka engine | Confluent Cloud Kora | Confluent claims broad Kafka client features, including exactly-once semantics | No Apache broker release maps to the service; tier and control plane matter |
| Protocol facade over another service | Azure Event Hubs | Produce, consume, groups, offsets, and selected Kafka client features | Azure resource model, service limits, Admin gaps, and conflicting transaction docs |
The rows do not rank products. They identify which assumption must be tested next.
Running Apache Kafka does not make the service control plane portable
Amazon MSK and Google's managed service both run named Apache Kafka releases, but the surrounding contract differs.
Amazon's supported-version table lists Kafka 4.2.x on Express brokers and excludes KIP-932 Queues for Kafka. With AWS Identity and Access Management (IAM) authorization, Kafka ACL APIs can succeed while having no authorization effect on IAM identities. The request worked; the policy did not move.
Google's service overview pins brokers to Kafka 3.7.1, hides the backing Cloud Storage buckets, and does not expose Java Management Extensions (JMX) metrics. It supports IAM-backed Simple Authentication and Security Layer (SASL) and mutual TLS (mTLS), but not common mechanisms such as Salted Challenge Response Authentication Mechanism (SCRAM) or Generic Security Services API (GSSAPI), often used with Kerberos. Broker code compatibility does not transfer authentication, metrics, storage custody, or upgrade control.
For a managed named release, test the upstream APIs and the provider control plane as separate products.
Redpanda keeps familiar requests while changing broker invariants
Redpanda's current documentation says it validates Kafka 4.x-capable clients across core produce, consume, and transaction operations. Its 26.2.1 source also shows important boundaries.
The pinned handler list contains classic JoinGroup, SyncGroup, and Heartbeat handlers, but no ConsumerGroupHeartbeat or ConsumerGroupDescribe handlers. A classic group test therefore says nothing about group.protocol=consumer.
Redpanda's compatibility page states that it does not implement KIP-890's server side. Kafka 4.x clients fall back to the original transaction protocol. That may fit the workload, but the transaction generation belongs in the test record.
The 26.2.1 topic-config source accepts several Kafka property names as no-ops, including min.insync.replicas and flush.messages. A successful Admin response is not proof that the target changed runtime behavior.
This is the larger lesson: configuration-name compatibility can be more dangerous than an explicit error because it turns a rejected assumption into a green deployment.
For Redpanda, read back effective configuration, then inject the failure that the setting was meant to control.
Object ownership is not stream ownership
An object-storage engine separates record files from the metadata that gives those files Kafka meaning.
WarpStream's architecture documentation places record data in customer object storage and keeps Virtual Cluster metadata in a vendor-operated service. Its read-path documentation says files can reach object storage out of order; the metadata store maps files to topic-partition offset ranges and provides the authoritative order.
Owning the bucket therefore gives custody of bytes. It does not, by itself, give a documented procedure for rebuilding the ordered log.
WarpStream's normal mode supports Kafka transactions and exactly-once semantics. Other modes have different contracts:
- Orbit Auto Migration at Agent v826 does not support transactional produce while a topic is in
PROXYorMIGRATING. - Ripcord keeps Produce available during control-plane loss, but Fetch stops, returned offsets are zero, idempotent producers and transactions must be disabled, and external consistency is not guaranteed.
Those modes are useful because they make trade-offs explicit. They also prove that a product name is not a semantic pin. Record the exact mode.
AutoMQ creates a different object-storage boundary. The 1.7.2 release artifact is named automq-1.7.2_kafka-3.9.1.tgz, and its tagged gradle.properties sets the base version to 3.9.1. AutoMQ replaces Kafka's log storage with S3Stream, its object-backed stream library, and uses a write-ahead log (WAL) to protect the write path.
AutoMQ's documentation claims full Kafka compatibility. The source pin gives a more precise starting question: which Kafka 3.9.1 behavior remains unchanged, which later APIs did AutoMQ add, and which recovery steps depend on S3Stream data and operation metadata?
Bucket access is necessary. An executable reader for the metadata and object layout is the portability asset.
Kora and Event Hubs show two kinds of version ambiguity
Confluent Cloud runs Kora, a proprietary engine that does not track one Apache Kafka release. Confluent claims all client features since Kafka 0.10, including exactly-once semantics, and documents KIP-848 support for current Java and librdkafka clients. The semantic pin still includes the service tier: Freight does not support transactions.
Azure Event Hubs creates a different ambiguity. Microsoft describes it as a Kafka endpoint over Event Hubs rather than a Kafka broker. Topic administration uses Azure APIs, and broker configurations are unavailable.
Its first-party transaction documentation conflicts:
- the overview and transaction page say transactions are in public preview for Premium and Dedicated;
- the newer client configuration page lists transactions and exactly-once semantics as unsupported;
- the migration guide again says transactions are in preview.
Until Microsoft publishes one unambiguous, tier-specific contract and the target namespace passes failure tests, treat transactions as unsupported. Conflicting primary sources are not a tie. They are a stop signal.
Durable authority decides what an acknowledgement proves
Durable authority is the state from which a system can reconstruct an acknowledged record. Different authorities create different failure and exit paths.
| System or mode | Durable authority before acknowledgement | Metadata needed for recovery | Exit question |
|---|---|---|---|
| Apache Kafka 4.3.1 | In-sync replica progress under the selected Kafka settings; acks=all does not itself require per-request disk flush | KRaft metadata plus partition logs | Can another cluster receive records, offsets, configs, and required visibility within RPO and RTO? |
| Redpanda local storage | Partition Raft and the configured flush or write-cache policy | Controller and partition Raft state | Does the target preserve the accepted crash and storage-loss outcome? |
| WarpStream normal mode | Object write plus commit to Virtual Cluster metadata | Vendor metadata maps files to ordered offsets | Can both records and ordering metadata be exported before service loss? |
| WarpStream Ripcord | Object-storage journal before asynchronous metadata commit | Journal replay plus later control-plane recovery | Does write availability without read progress or external consistency fit? |
| AutoMQ | Configured WAL and object-storage path through S3Stream | Kafka-derived control state plus S3Stream operation metadata | Can a supported replacement interpret both object classes? |
| Managed Kafka service | Provider-managed broker and storage path | Kafka state plus provider control plane | Which state is exportable, and who controls the recovery clock? |
The same Kafka client call can sit above a local replica quorum, an object store plus metadata service, or another cloud event system. Match the failure you can tolerate, not the spelling of acks.
Migration copies checkpoints, not live coordination
A consumer offset is a stored checkpoint. A group generation identifies a coordinator's current membership and assignment epoch. Replication can copy a checkpoint. It does not move live sockets, in-flight callbacks, or a running generation into another coordinator.
A transaction marker records commit or abort state in a Kafka partition. Moving record bytes without the required marker visibility can change what a read_committed consumer observes. Active producer IDs, epochs, and open transaction-coordinator state also belong to the source.
| Migration path | Direction and stopping point | Offset behavior | Semantic boundary |
|---|---|---|---|
| Apache MirrorMaker 2 | Each flow is directional, such as A -> B; reverse needs another flow | Emits checkpoints and can synchronize translated committed offsets | Target consumers form new generations; the docs do not claim to transfer live membership or open transaction-coordinator state |
| Confluent Cluster Linking | Source to mirror topic; promote stops mirroring and offset sync for that topic | Default offset sync is 30 seconds; data lag must reach the checkpoint | Mirror topics do not support transactions or exactly-once semantics |
| WarpStream Orbit Auto Migration v826 | Source to WarpStream; COMPLETE is terminal | Source offsets must be preserved and verified for consumers | PROXY and MIGRATING do not support transactional produce |
| Amazon MSK Replicator | Asynchronous source to target; reverse is another design | Replicates consumer offsets, topic configuration, and a subset of literal topic ACLs; IAM policies and roles do not move | Documentation does not claim to move live group or transaction-coordinator state |
| MirrorMaker to Azure Event Hubs | Source to Event Hubs | Existing consumer offsets do not arrive; consumers choose a reset point | Azure resource and transaction boundaries replace Kafka broker behavior |
Before any promotion, record:
- source and destination log-start and log-end offsets per partition;
- replication lag in offsets and time;
- the committed offset for every moving group;
- whether the destination contains the record visible at that checkpoint;
- active producers, consumers, and transactions still on the source;
- the first operation after which writes cannot safely return;
- measured cutover and rollback RPO and RTO.
If the runbook cannot name the irreversible operation, it is not a rollback runbook.
Real failures reveal the hidden contract
Observed bugs and documented limits carry different weight, but both belong in the decision.
| Evidence | Primary record | What it proves |
|---|---|---|
| Upstream regression | KAFKA-19444 | A client can depend on an API version through authentication even when application code never calls it |
| Accepted no-op config | Redpanda 26.2.1 topic config source | Admin success can hide unchanged runtime behavior |
| Mode-specific semantic downgrade | WarpStream Ripcord | Availability mode can disable idempotence, transactions, reads, and ordering guarantees |
| First-party conflict | Azure transaction page and configuration page | Product documentation can be too inconsistent to support a correctness claim |
Do not rewrite a limitation as an outage or an outage as a universal product property. Use each record to design the next test.
Normalize semantics before measuring performance
Vendor benchmarks help select hypotheses. They do not provide a neutral ranking.
| Test dimension | Pin before load generation | Failure prevented |
|---|---|---|
| Durability | acks, quorum rule, flush or write cache, replication, object commit | Comparing different loss envelopes under one label |
| Producer correctness | Idempotence, transaction generation, retries, in-flight requests | Giving one target safer or cheaper retries |
| Consumer behavior | Group protocol, assignor, read_committed, offset reset | Comparing different visibility or reassignment contracts |
| Data shape | Record-size distribution, keys, entropy, codec, batch size, linger | Giving one engine easier payloads |
| Storage state | Empty, warm cache, cold object read, aged retention, compaction backlog | Publishing a hot-cache replay result |
| Load | Fixed offered-load points through saturation, achieved throughput, errors | Calling a lower-load run faster |
| Distribution | p50, p95, p99, p99.9, max, retries, sample count, repeated runs | Hiding queueing and failures behind an average |
| Recovery | Process, zone, control-plane, storage, credential, and network faults | Measuring only healthy steady state |
| Cost | Vendor meter, infrastructure, transfer, object requests, support, and dual-run month | Moving cost into an omitted invoice |
For Redpanda, setting min.insync.replicas=2 without checking effective behavior does not normalize durability. For WarpStream, Agent version and normal, Orbit, or Ripcord mode belong in the result. For AutoMQ, Kafka base, WAL type, object store, and cache state belong there. For managed services, tier and region are part of the implementation.
A tested exit is the only credible portability claim
A decision is ready only when three artifacts exist:
| Artifact | What it must prove | Invalid substitute |
|---|---|---|
| Versioned conformance report | The pinned target passes every required check across the six layers defined at the start | A vendor compatibility page or one producer-consumer smoke test |
| Full-volume migration and rollback record | Data, offsets, recovery metadata, operators, and tooling meet measured RPO and RTO under failure | A paper runbook or a sample-topic copy |
| Signed dependency record | Owners accept any unportable feature, license term, control-plane dependency, and exit cost | Calling an untested requirement "standard Kafka" |
Ordinary at-least-once transport may pass with modest work. Stateful stream processing, strict transactional visibility, cloud-native IAM, or years of retained history raise the bar. Sometimes the current platform still wins. The useful result is knowing which requirement made it win.
Kafka compatibility opens the door. Portability begins when the team has walked out and back in.
The course is complete. Turn the Kafka Engineering series into one migration rehearsal against your own workload.
Sources and References
Apache Kafka
Redpanda 26.2.1
- Redpanda 26.2.1 release
- Kafka compatibility
- Producer acknowledgement settings
- Pinned Kafka handler list
- Pinned accepted no-op topic configs
- Redpanda Business Source License at 26.2.1
WarpStream Agent v826
- Agent change log
- Protocol and feature support
- Architecture and Virtual Cluster metadata
- Read path and ordering metadata
- Orbit Auto Migration
- Ripcord
AutoMQ OSS 1.7.2
Managed services and migration
- Amazon MSK supported Kafka versions
- Amazon MSK IAM authorization
- Amazon MSK Replicator
- Google Managed Kafka overview
- Google Managed Kafka authentication
- Confluent Cloud Kafka version FAQ
- Confluent Cloud client configuration
- Confluent Cloud cluster types
- Confluent Cluster Linking limits
- Confluent Cluster Linking migration
- Azure Event Hubs Kafka overview
- Azure Event Hubs Kafka client configuration
- Azure Event Hubs Kafka transactions
- Azure Event Hubs Kafka migration
