Redpanda Data Transforms: Boundaries of In-Broker Wasm

Your pipeline, the sequence of processing stages that validates and routes each record, only needs to reject malformed JSON and send the rest to two topics. A separate stream-processing cluster, independently deployed workers that read and write streams, can do that, but it also adds a deployment, a consumer group, coordinated readers sharing partitions, offset storage, persisted partition positions used to resume processing, retries, credentials, metrics, and another failure domain, a component boundary that can fail independently.

Running the function inside a broker, the server process that owns the log, removes some of that machinery. It also moves untrusted code into that process.

WebAssembly (Wasm) is a portable bytecode format executed inside a sandboxed runtime. Redpanda Data Transforms are a narrow in-broker tool: one input record enters a core-local Wasm runtime, zero or more output records leave, and progress is committed periodically. They provide at-least-once delivery, where failure can replay an input record; they are not stateful stream processing or a transactional extension of the producer write.

This analysis uses Redpanda v26.1.13 and the pinned July 2026 Data Transforms documentation. The implementation is called Data Transforms in the product docs. Older “coprocessor” material describes a retired generation and should not be used to infer current behavior.

The tagged transform source carries a Business Source License header, not an Enterprise RCL header, and Data Transforms is absent from the current Enterprise-only feature table. Tiered Storage and Cloud Topics have a different license boundary.

The transform runs after the input write

A transform function is user code compiled to a WebAssembly module. Redpanda stores the module and metadata in internal topics, then starts transform instances where the input partition leaders run.

A Wasm virtual machine (VM) is one isolated runtime instance executing the compiled transform.

The pinned execution guide describes the order:

1. the input record is successfully written to the input topic;
2. the partition leader's shard runs the transform processor and Wasm engine;
3. Redpanda feeds records to the VM;
4. the transform emits zero or more records to configured output topics;
5. Redpanda periodically commits transform progress.

The original Produce request does not wait for the transform output. A slow or failed transform can build lag without blocking new writes to the input topic.

Kafka Produce
  → input partition commit
  → background transform read
  → Wasm callback
  → output-topic produce
  → periodic progress commit

This sequence rules out a tempting claim: the input write and every transform output are not one atomic Kafka transaction. A crash can occur after an output write and before the transform commits its input progress.

Use the lab below to place a deterministic failure after output writes but before the next progress checkpoint. Change fan-out and checkpoint cadence separately.

The duplicate exposure grows with replayed passing inputs and output count. Input commit, output produce, and progress commit remain separate boundaries.

Fan-out progress is tracked per output

Multiple outputs do not share one all-or-nothing progress marker. The tagged transform::processor loads one committed input offset for each configured output. On restart, it begins reading after the minimum of those offsets so the slowest output can catch up. An output that had already advanced suppresses replayed records until the replay reaches that output's committed offset.

This limits duplicate work for an output that was already caught up; it does not make fan-out atomic. An output write can succeed before its progress update is committed, so a crash can still repeat that output. If two destinations must change together, Data Transforms do not provide that transaction.

Leadership decides where the VM runs

Redpanda follows the partition leader. A processor is the per-transform, per-input-partition pipeline that reads source records, invokes one Wasm engine, writes outputs, and tracks progress. The pinned docs place it on the same shard, the core-local Seastar execution context, as the input partition leader.

The tagged source reflects that model:

• wasm::engine is local to the core on which it was created;
• a factory holds a compiled module and creates engines;
• one process-wide runtime owns Wasmtime and shared allocation services;
• transform::processor joins the source, VM, output queues, sinks, and progress tracker.

Each core owns a fixed-size pool of VM/engine instances, bounded by that core's transform memory reservation. The manager creates one processor for each active transform and input partition whose leader is local, as transform_manager.cc shows, and each processor uses an engine from the core-local pool. Processor count therefore scales with locally led input partitions. One deployed function can use multiple processors and engine instances when multiple input partitions are led on that core or elsewhere in the cluster; there is no single cluster-wide VM per function.

When leadership moves, the old instance stops and the new leader resumes from the last committed transform offset. The VM's in-memory state does not move with it.

This makes transforms fit functions whose result depends only on the current record and stable configuration. A process-local map, timer, or cache can disappear on restart or leadership transfer.

Wasmtime executes the module under Redpanda's budget

Redpanda v26.1.13 uses Wasmtime. The runtime compiles Wasm to machine code, allocates guarded VM stack and heap memory, and translates a configured runtime limit into Wasmtime fuel, an instruction budget.

The tagged source does not establish a universal invocation cost or a fixed ratio to native execution. Compilation, guest language runtime, record size, serialization, output fan-out, and host calls all affect cost.

The self-managed defaults in the pinned cluster-property reference are more useful:

The maximum number of simultaneously resident VM or engine instances on one core is bounded by the reserved pool divided by the per-instance limit. One deployed transform can consume several instances when several locally led input partitions run concurrently. Larger records and outputs also need room in the read and write buffers.

These are defaults, not sizing recommendations. A JSON parser and a one-to-one output have a different memory profile from one input record fanning out eight large outputs.

The host copies records across the Wasm boundary

The current transform_module parses the input batch, exposes one record at a time, and copies each record's payload into guest memory. Guest output crosses back through a host function.

That path is deliberately bounded and inspectable. It is not “zero-copy Wasm.” The broker trades copies at the sandbox boundary for isolation and a language-neutral ABI.

The module can:

• read the current record;
• emit zero or more records;
• select a configured output topic where the SDK supports write options;
• log through the provided host interface;
• use Redpanda's Schema Registry host API.

It cannot open arbitrary files or sockets. The pinned limitations state that transform functions have no external disk or network access.

Schema Registry is a controlled exception, not general networking: Redpanda provides a host-side interface to its own registry.

Supported languages are narrower than “anything that compiles to Wasm”

The pinned rpk transform init workflow supports:

• TinyGo without goroutines;
• TinyGo with goroutines;
• Rust;
• JavaScript;
• TypeScript.

The list comes from the pinned build guide. The product does not promise support for every language with a Wasm backend. SDK ABI, runtime assumptions, standard-library behavior, and tooling still matter.

JavaScript also has a product-specific limit: its SDK cannot select a specific output topic per write. Go and Rust can route to configured outputs through write options.

Input metadata differs by SDK in the tagged tree. The Go SDK exposes Offset and Timestamp on the input record. Rust's WrittenRecord exposes a timestamp but not an offset, and JavaScript's WrittenRecord exposes key, value, and headers without either field. Do not design deduplication around an input offset until the chosen SDK actually exposes it.

The latest published joint SDK tag is transform-sdk/v1.1.0, but the pinned compatibility matrix only lists v1.1.x beside Redpanda 24.2.x. The same page states that existing SDK versions remain supported by newer brokers, so v1.1.0 on 26.1.13 is covered by a compatibility policy, not by an explicit matrix row. Pin the SDK, broker, rpk, and Wasm artifact anyway, then test the exact combination before an upgrade.

At-least-once delivery defines the error strategy

Transforms commit progress periodically. If a broker fails after writing output but before committing the corresponding input offset, the replacement instance reads that input again. The docs name this at-least-once delivery.

The application consequence is direct:

Every output path must tolerate duplicates, or downstream processing must deduplicate them.

Errors create a second choice:

Redpanda does not document a built-in dead-letter-queue policy. An error topic is an application design implemented by the transform. The pinned error-handling guide explicitly warns that logging and continuing can silently discard records when logs are not monitored.

One record in, zero or more records out

The pinned limits keep the programming model intentionally small:

• one input topic;
• all partitions of that topic;
• one record processed at a time;
• zero or more output records;
• at most eight configured output topics;
• no durable transform-local state;
• no joins, windows, or cross-record aggregation;
• only committed input records when producers use Kafka transactions.

A transform can start from the latest offset, which is the default, or an operator can choose an initial offset or timestamp on first deployment using the --from-offset or --from-timestamp options. Redeploying the same transform resumes from its committed progress; reprocessing with a new starting point requires deleting and redeploying it.

Choose the smallest execution model that meets the semantics

My rule is to use a Data Transform when the logic is deterministic, bounded, and record-local. Once the design needs external I/O or durable state, moving it out of the broker is not ceremony; it is the correct failure boundary.

The sandbox does not contain these five failures

The sandbox still leaves five failures for the application or operator.

A poison record restarts forever

An uncaught error can fail the VM and replay from the last committed offset. Classify parse and validation errors inside the transform, route or skip them deliberately, and alert on execution failures and processor lag.

A “cache” becomes hidden state

Leadership changes and restarts discard VM memory. If losing a map changes the output, the transform is stateful even if the code has no database.

Fan-out exhausts the memory budget

The input and all pending outputs share bounded per-core resources. Test the largest record and maximum fan-out, not only the average event.

Duplicate output triggers a business side effect twice

At-least-once replay can repeat output records. Give downstream handlers an idempotency key or deduplication rule. Use an input offset only in an SDK that actually exposes it.

SDK and broker versions drift

The pinned compatibility table does not cover 26.x. Pin the SDK package, Wasm artifact hash, rpk version, and broker release in deployment metadata.

Pin the runtime before measuring it

git clone --branch v26.1.13 --depth 1 \
  https://github.com/redpanda-data/redpanda.git redpanda-v26.1.13
cd redpanda-v26.1.13
 
git grep -n "class engine" -- src/v/wasm/engine.h
git grep -n "millisecond_fuel_amount" -- src/v/wasm/wasmtime.cc
git grep -n "Copy out the payload" -- src/v/wasm/transform_module.cc
git grep -n "load_latest_committed" -- src/v/transform/transform_processor.cc

On a test cluster, pair that source trace with:

TRANSFORM_NAME="my-transform"
 
rpk transform list
rpk transform logs "$TRANSFORM_NAME"
rpk cluster config get data_transforms_enabled

Record transform lag, execution latency, failures, CPU time, memory use, input rate, output rate, and the artifact hash. A per-record function benchmark does not capture queueing or replay behavior.

The sandbox is only half the contract

• Data Transforms run after the input write and do not block the original Produce request.
• The Wasm engine is core-local and follows input partition leadership.
• The current SDK surface supports TinyGo, Rust, JavaScript, and TypeScript.
• Guest code has no arbitrary disk or network access.
• Delivery is at least once, so duplicate output is part of the normal failure model.
• The right use case is bounded, record-local transformation—not joins, windows, or external enrichment.

Previous: Redpanda Tiered Storage: Local Writes, Remote Reads ←

Sources

• Pinned Data Transforms behavior
• Pinned build and error-handling guide
• Pinned configuration guide
• Pinned monitoring guide
• wasm/engine.h at v26.1.13
• wasm/wasmtime.cc at v26.1.13
• transform_processor.cc at v26.1.13
• Transform SDK v1.1.0
• Live Redpanda documentation — secondary, moving reference