Synthetic teaching case · not client work
Async Reliability Review: sample handoff
One virtual debit succeeds, then the worker crashes before it records the result. This sample shows how a bounded review could separate observed evidence, assumptions, risks, and owned next actions.
Every fact below comes from the deterministic public model. It does not describe a client engagement, GitHub production behavior, or a measured reliability outcome.
Scope and evidence boundary
- Flow
- One virtual debit operation and one injected crash
- Failure point
- After the modeled debit, before the result is recorded
- Evidence
- Two deterministic traces generated by lib/async-failures.ts
- Decision
- Whether retry may repeat the effect or must reconcile first
Model assumptions
- One virtual debit operation and a single injected fault; no real payment, broker, or network is used.
- The durable case assumes a stable operation key and a downstream contract that can deduplicate or look up that same intent.
- The downstream idempotency record is retained throughout the modeled retry window and is atomically tied to the effect.
- Operation lookup eventually succeeds, storage is available, and authorization is checked before dispatch.
- Outages of the lookup service, expired deduplication records, concurrency, and timing distributions are not modeled.
Failure map
| Boundary | Evidence in the model | Review question |
|---|---|---|
| Dispatch intent | Recorded only in the durable trace | Which stable identity survives worker replacement? |
| External effect | The virtual debit happens before the injected crash | Can the downstream system find or deduplicate the intent? |
| Recorded result | Absent when the worker restarts | Does the runtime represent UNKNOWN separately from FAILED? |
| Replay | Naive retry repeats; durable recovery looks up the key | Which evidence and owner authorize another dispatch? |
Evidence summary
Naive loop
- Attempts
- 2
- Effects
- 2
- Status
- Duplicate debit
- 01ToolCalled
- 02DebitApplied
- 03ProcessCrashed
- 04ToolCalledAgain
- 05DebitAppliedAgain
Durable runtime
- Attempts
- 2
- Effects
- 1
- Status
- Recovered by operation key
- 01DispatchIntentRecorded
- 02DebitApplied
- 03ProcessCrashed
- 04RunReplayed
- 05OperationLookupFound
- 06ToolResultRecorded
The model produces two attempts and two effects for the naive loop. With the stated durable-runtime assumptions, it produces two attempts and one modeled effect, then reports “Recovered by operation key.”
Prioritized recommendations
- P0Carry one operation key across every delivery attemptThe worker, downstream effect, lookup path, and replay tooling need the same stable identity. A delivery ID alone does not identify the business intent.
- P0Represent an unknown outcome instead of guessingAfter a crash or timeout, reconcile the operation key before another effect-capable call. Missing acknowledgement is not proof that the effect failed.
- P1Make retention at least as long as the replay windowA deduplication record that expires before delayed retries or manual replay can turn an old operation into a new effect.
- P1Name the recovery owner and required evidenceThe runbook should state who can pause dispatch, inspect the operation record, approve replay, and verify the final business state.
Replay and ownership handoff
Before replay
- Pause or bound new dispatch for the affected operation.
- Look up the stable operation key at the effect boundary.
- Confirm deduplication retention still covers the replay.
- Record who authorizes replay and verifies final state.
What this sample cannot decide
- Your real provider's idempotency semantics.
- Your replay window, storage availability, or concurrency.
- Who has operational authority in your organization.
- Whether a real incident had the same failure sequence.
Use the shape, not the synthetic conclusion
A real review replaces this model with your authorized evidence, names the unknowns, and leaves implementation and operational ownership with your team.