Agent Loops: Distributed Transactions Without Atomic Commit

The model asks your application to debit an account. The debit succeeds. The process crashes before it records the result.

After restart, the conversation still contains the tool request but no tool result. The obvious recovery path calls the tool again.

The customer is charged twice.

Here is the plain-language model. The model is a planner. The orchestrator is the application code that records the plan and dispatches work. The provider conversation is a transcript, while the application's durable ledger records what the business actually attempted. A crash can leave those records disagreeing.

An agent loop is a distributed workflow across systems that do not share one atomic commit, an all-or-nothing decision. The model proposes actions; the application owns intent, authority, side effects, recovery, and evidence.

The transaction analogy is useful because it exposes crash gaps. It is not a claim that model providers, application databases, external application programming interfaces (APIs), queues, and humans participate in one database transaction, XA-style two-phase commit, where a coordinator asks every participant to prepare and then commit, or a universal rollback protocol.

Use the simulator below to reproduce the opening failure. Trigger a crash after the external debit but before the local result record, then compare a blind retry with checking the downstream receipt or replaying under the same operation key. This application-owned identifier names one business intent.

The lesson is not “never retry.” It is “do not guess.” After the lost response, the application owns an UNKNOWN outcome until reconciliation establishes what happened. Reconciliation means querying the downstream system or safely repeating the same declared intent under a contract that deduplicates the same key.

Provider call IDs correlate messages, not business intent

A provider call ID correlates one proposal and result inside the provider's message protocol. A business intent is the application-level action the user meant to perform, even if several execution attempts follow.

Tool APIs encode the proposal/result exchange. Field names differ:

Those identifiers join provider messages. They are not automatically stable across a regenerated model turn, provider migration, replay, or application retry.

Create application-owned identities:

interface ToolIntent {
  readonly runId: string
  readonly providerCallId: string
  readonly logicalCallId: string
  readonly operationKey: string
  readonly toolName: string
  readonly argumentsHash: string
  readonly principalId: string
  readonly policyVersion: string
  readonly deadlineAt: string
}

logicalCallId identifies one proposed step in your workflow. operationKey identifies one business intent across execution attempts. argumentsHash binds that key to canonical arguments. principalId identifies the user or service whose authority the application will evaluate.

Canonicalization is domain work. Currency needs an explicit unit and scale; resource IDs need a tenant, the customer or account isolation boundary; unordered sets need stable ordering; timestamps need a zone. Do not log secrets merely to make the hash reproducible.

The workflow is transaction-shaped, not transactional

PostgreSQL's transaction tutorial describes an all-or-nothing boundary inside one database. A tool-using run usually crosses:

• model provider;
• orchestrator process;
• local database;
• queue or scheduler;
• external APIs;
• human approval;
• notification channels.

No general rollback spans that set.

The analogy earns its keep only when it forces six questions:

1. Where does intent become durable?
2. Which effects may repeat?
3. What does a timeout mean for this dependency?
4. Who has authority at the instant of dispatch?
5. Which worker owns the run, and how is a stale worker fenced?
6. Which evidence lets an operator reconstruct the result?

Conversation history is protocol state, not the business ledger

Provider messages are necessary for model continuation. They are not enough to rebuild application state or prove which effect occurred.

A durable history might append:

RunCreated
ModelRequestStarted
ToolProposalRecorded
ArgumentsCanonicalized
AuthorizationAllowed
DispatchIntentRecorded
LeaseAcquired
ToolAttemptStarted
ToolOutcomeMarkedUnknown
ToolEffectReconciled
ToolResultRecorded
CompensationRequested
RunCompleted

At this point, a lease means temporary worker ownership, and a compensation means a new action intended to offset an earlier effect. Neither term promises rollback.

Each event needs more than a timestamp:

Event sourcing means application state derives from retained events. A folder of request logs is not event sourcing. A reducer applies events in order; the projection is the current state it rebuilds.

Two useful invariants are:

replay(events, reducer_version) -> the same projection hash

no external dispatch before durable DispatchIntentRecorded

Temporal's event-history documentation is one implementation example: the service appends workflow events for crash recovery and debugging, while also imposing history-size limits. The pattern transfers; its exact event model does not.

A timeout creates an unknown outcome

Consider this sequence:

1. orchestrator sends debit
2. wallet commits debit
3. response is lost
4. orchestrator reaches its deadline

The caller observed no result. The effect may exist.

Mark the call:

UNKNOWN

not:

FAILED

Recovery depends on the downstream contract:

1. query by operation key or provider receipt;
2. repeat only when the same key is idempotent for the same arguments;
3. submit a separately authorized compensation;
4. stop for operator repair when the outcome cannot be established.

The AWS timeout and retry guidance explains why timeouts and retries can increase load. gRPC is a remote procedure call framework; its deadline guide distinguishes an absolute deadline from a relative timeout and describes deadline propagation.

Cancellation has the same boundary. Cancelling a local future proves that the caller stopped waiting. It does not prove that a remote effect stopped.

Idempotency binds retries to one declared intent

Idempotency means that repeating one declared operation does not create an additional protected effect.

The AWS idempotent API guidance prefers caller-provided request identifiers because identical parameters can represent either a retry or two intentional operations.

A useful record is:

interface IdempotencyRecord {
  readonly operationKey: string
  readonly argumentsHash: string
  readonly scope: string
  readonly status: "in_progress" | "completed" | "failed" | "unknown"
  readonly resultRef?: string
  readonly downstreamReceipt?: string
  readonly expiresAt?: string
}

Apply four rules:

• same key, same arguments, completed: return the retained result;
• same key, same arguments, in progress or unknown: wait or reconcile;
• same key, terminal failure before any effect: retry only under the recorded dependency contract;
• same key, different arguments: reject;
• distinct business intent: issue a new key.

Stripe's idempotent-request contract showed two boundaries when checked on 2026-08-03: it compared parameters on key reuse, and it could prune keys after 24 hours. Your workflow must not assume that a downstream key lives forever.

Idempotency also has a scope. A payment API may deduplicate a charge while a notification subsystem still emits two emails. Record which effect the key protects.

The goal is not exactly one execution attempt. Retries intentionally create several attempts. The useful target is one observable business effect within a named scope, backed by downstream idempotency or reconciliation.

Retry ownership and deadlines prevent a failure storm

If a software development kit (SDK) attempts three times, your service attempts three times, and a queue redelivers three times, one logical call can produce:

3 × 3 × 3 = 27 attempts

Assign one retry owner for each dependency. Disable or account for hidden SDK retries.

Every retry decision needs:

• retryable failure classification;
• remaining attempt budget;
• remaining run deadline;
• idempotent execution or a reconciliation path;
• backoff with jitter, an increasing randomized delay;
• dependency health and circuit-breaker state, whether calls are temporarily blocked after repeated failures.

Use one absolute run deadline:

attempt_timeout =
min(per_attempt_cap, run_deadline - now - recovery_reserve)

A fresh 30-second timeout on every attempt can turn a 30-second product promise into several minutes of hidden work.

Measure:

retry_amplification = attempts / logical_calls

Also retain queue delay, backoff delay, timeout class, and deadline remaining. An attempt count without those fields does not explain the pressure.

A lease chooses an owner; a fencing value rejects stale owners

Optional architect path: if one process owns each run from start to finish, skip this section and continue at “Authority is evaluated at dispatch.” Once queue redelivery, failover, or long pauses can create two workers, a lease needs a fencing value or a stale worker can still write after takeover.

Two workers can replay the same run after a pause, partition, or slow failover. A lease grants ownership until an expiry time. It reduces overlap, but an expired worker may wake up and continue after a new worker takes ownership. A fencing value is a monotonic ownership generation that the protected resource checks before accepting a write.

The Chubby lock-service paper uses a sequencer, a value tied to the acquired lock state, so protected resources can reject delayed requests from stale holders.

If dispatch uses an outbox, a local table of messages waiting for delivery, the fence belongs there too.

Use the same idea for run ownership:

interface RunLease {
  readonly runId: string
  readonly ownerId: string
  readonly expiresAt: string
  readonly fence: number
}

On each successful takeover:

1. atomically assign the owner;
2. advance the monotonic fence;
3. include the fence in local state transitions and outbox records;
4. reject a write whose fence is lower than the latest accepted value.

A fence works only where the protected resource checks it. Most third-party APIs do not accept your workflow fence. For those effects, route dispatch through a fenced local outbox, reuse a downstream idempotency key, and reconcile ambiguous outcomes.

Do not renew a lease indefinitely without a run deadline. Ownership is not permission to loop forever.

Authority is evaluated at dispatch, not when the model suggested the call

JSON Schema is a vocabulary for validating the shape of structured data. It can answer:

Is this object shaped like a debit request?

It cannot answer:

May this principal debit this tenant's account for this amount now?

The JSON Schema 2020-12 core specification defines structural constraints and annotations. Business authority belongs to the application.

Bind an authorization decision to:

• principal and tenant;
• action and resource;
• canonical arguments;
• policy version;
• approval or delegation chain;
• amount, scope, or rate limit;
• issue and expiry times.

Reauthorize immediately before dispatch. A queued call can outlive a user session, approval, policy version, or spending limit.

My default rule is stricter for irreversible or high-impact tools: the model never holds ambient authority, broad permission inherited from the application process. It produces a proposal that a policy layer reduces to a narrow, expiring capability.

Parallel calls create partial completion, not one result

A model turn can propose several tools:

reserve inventory -> succeeded
debit wallet      -> succeeded
schedule shipment -> failed

Track each logical call independently:

PROPOSED
  -> VALIDATED
  -> AUTHORIZED
  -> INTENT_RECORDED
  -> DISPATCHING
  -> SUCCEEDED | FAILED | UNKNOWN

The run cannot complete while a required call remains UNKNOWN.

Correlation uses call IDs, never array position. Completion order can differ from proposal order, and a retry can produce a new provider message ID.

Compensation is a new effect, not rollback

A saga splits a long workflow into committed steps and separately committed compensating steps. The original Saga paper describes this shape for long-lived transactions.

For the example above:

release reservation
refund debit

Each compensation needs:

• its own operation key;
• current authorization;
• a deadline and retry policy;
• reconciliation;
• an outcome retained in the audit history.

Compensation can fail. The original effect may already be visible to a user or downstream system. Calling it rollback erases that fact.

Some effects have no honest compensation: an email was read, a secret was exposed, a pull request triggered automation, or a market order executed. Those tools need prevention, approval, or containment before dispatch.

The transactional outbox closes one local dual-write gap

Suppose the application needs to:

mark order fulfilled
publish OrderFulfilled

Writing the row and publishing the message separately leaves a crash window.

The transactional outbox writes the business change and message intent in one local database transaction. A relay publishes the outbox row later. AWS's outbox pattern documentation also calls out duplicate messages and ordering.

The relay can crash after publish but before acknowledging the outbox row. Consumers still deduplicate by event ID.

A pattern closes one named atomicity gap. It does not grant global atomicity.

Tool output remains untrusted data

Structured output can remove syntax errors. It cannot make arguments authorized, current, or semantically valid.

Validate proposals in this order:

1. schema and size limits;
2. known tool and version;
3. canonical arguments;
4. tenant and resource binding;
5. business invariants;
6. authorization and approval;
7. unresolved-call and lease state;
8. dispatch budget.

Validate results too. Tool output can contain stale data, malicious instructions, oversized payloads, or content controlled by another user. Anthropic's tool documentation explicitly warns that tool results can carry indirect prompt injection, instructions hidden inside tool-controlled data. Keep results in an untrusted data channel and promote only parsed fields that the next step needs.

Termination is a reliability control

An agent can stay syntactically healthy while making no progress.

Bound every run with:

The stop reason belongs in the event history. “Budget exhausted” is different from “tool failed,” “authorization denied,” and “outcome unknown.”

The workshop artifact is a fault protocol

Before implementing the workshop, predict the state after each injected crash and name the component that owns recovery. If your prediction says FAILED after a lost response, revisit the timeout section before writing code.

The downloadable JSON defines a seed, run deadline, attempt limits, injected crash points, and assertions. The browser simulator is a deterministic teaching model. Neither is a supplied production workflow engine.

A useful local implementation can use SQLite, a scripted planner, and a virtual clock:

Acceptance properties include:

attempt_count >= 2
business_effect_count == 1

after crash and replay;

projection_hash_before_restart
==
projection_hash_after_replay

after rebuilding projections; and

dispatch_count == 0

for invalid, stale-fence, expired, or unauthorized proposals.

A green assertion is not the whole result. The lab write-up should explain why the invariant held, which durable record made replay possible, and who repairs an outcome that remains UNKNOWN.

What could go wrong

• Intent is logged after dispatch.
• Provider call IDs are reused as business idempotency keys.
• The same operation key is accepted with changed arguments.
• A timeout is marked failed without reconciliation.
• Several layers retry the same dependency.
• Relative timeouts reset on every attempt.
• A lease expires but the resource does not check a fence.
• Authorization happens at proposal time, then expires in a queue.
• Parallel results are correlated by position.
• Compensation is assumed to succeed.
• Outbox delivery duplicates a consumer effect.
• Tool output is inserted into a privileged instruction channel.
• The event history cannot identify the principal, policy, worker, or receipt.
• The loop has no turn, cost, or history limit.

The runtime becomes dependable when failure is a state transition with an owner, not an exception escaping a while loop.

The model may choose the next proposal. It never owns the commit.

Workshop three moves from workflow state to model state: context becomes an allocation, retention, retrieval, and information-loss problem. The course roadmap shows how that storage boundary connects to the other workshops.

Sources and references

Tool protocols and validation

• OpenAI function calling
• Anthropic tool-result handling
• JSON Schema 2020-12 core

Workflow reliability

• PostgreSQL transactions
• Sagas
• The Chubby lock service
• AWS: Making retries safe with idempotent APIs
• AWS: Timeouts, retries, and backoff with jitter
• gRPC deadlines
• Stripe idempotent requests
• Temporal event history
• Transactional outbox