Background jobs / SLO workbench
Job Completion SLO & Alert Designer
Did accepted work finish before its deadline, including the jobs that never produced a completion event?
Interactive calculations run in your browser; the initial example is pre-rendered. There are no accounts, uploads, or live queue connections. Inputs stay in page memory; the site does not persist them, put them in URLs, or send them to analytics. A worksheet download includes only what you explicitly export.
Completion evidence, not acknowledgement counts
Long-window completion SLI
55.56%
5 met / 9 matured deadlines
Unfinished overdue jobs
2
Across the observed trace, including outside the alert window
Long-window burn rate
44.44x
Miss fraction / allowed miss fraction
Two-window alert
Would fire
Only for the declared example metric contract
| Window | Deadlines | Met | Missed | Burn |
|---|---|---|---|---|
| 3600 s long window | 9 | 5 | 4 | 44.44x |
| 300 s short window | 3 | 2 | 1 | 33.33x |
At this volume, the long-window allowance is 0.09 misses (fractional planning budget); remaining budget is -3.91. 1 accepted jobs are not yet due. The oldest unfinished overdue job is 360 seconds past its deadline. Do not read a non-firing alert as proof of health: volume, delayed observation, or absent telemetry can suppress it.
Example PromQL: two-window deadline alert
Required metric contract
- background_job_deadlines_total{queue,outcome="met|missed"}: initialize both counters, then increment exactly one outcome once for every accepted job at its deadline; late success cannot rewrite a miss.
- background_job_deadline_observer_lag_seconds{queue}: age of the oldest deadline not yet assessed; a stalled observer invalidates the apparent SLO.
- background_jobs_oldest_overdue_age_seconds{queue}: seconds past the oldest unfinished job's deadline; zero when none exist.
- Use a bounded queue label, never job IDs, tenant identities, operation keys, or payloads. Monitor exporter availability and accepted-job/deadline accounting independently.
| Job / accepted | Deadline | Trace terminal time | Observed state | Deadline assessment |
|---|---|---|---|---|
| job-1 / 0 s | 300 s | 30 s | Success within deadline | Met |
| job-2 / 60 s | 360 s | 180 s | Success within deadline | Met |
| job-3 / 120 s | 420 s | 730 s | Success after deadline | Missed |
| job-4 / 180 s | 480 s | 210 s | Terminal failure | Missed |
| job-5 / 240 s | 540 s | None (stuck) | Unfinished, deadline missed | Missed |
| job-6 / 300 s | 600 s | 330 s | Success within deadline | Met |
| job-7 / 360 s | 660 s | 390 s | Success within deadline | Met |
| job-8 / 420 s | 720 s | 450 s | Success within deadline | Met |
| job-9 / 480 s | 780 s | None (stuck) | Unfinished, deadline missed | Missed |
| job-10 / 880 s | 1180 s | 910 s (future) | In flight, not yet due | Not yet in cohort |
Choose a denominator that cannot forget unfinished jobs
A completion-only histogram can look healthy while stuck jobs disappear from it. This example evaluates every accepted job once its deadline arrives and counts late or missing success as a miss. The generated query belongs to that explicit deadline-observer contract, not to arbitrary broker metrics. Observer freshness, accepted-job accounting, and exporter health remain independent requirements.
Assumptions and limits
- This is an exact, small synthetic trace, not telemetry ingestion. CSV rows are accepted-second, queue-wait, work, retry-delay, outcome (success, failure, or stuck). Durations are whole seconds.
- Every accepted job belongs to the cohort ending at its deadline, even if it fails early. The lookback interval is (observation time - window, observation time], using deadline time, not completion time.
- A job meets the objective only if success is recorded by its deadline, including equality. Failed, stuck, or late-success jobs miss it. Rejected submissions are not accepted jobs in this model.
- Jobs whose deadlines have not arrived are not yet in a deadline cohort. Unfinished overdue work is counted as missed rather than disappearing from completion-only statistics.
- Burn rate is the missed fraction divided by the allowed missed fraction. A 100% objective has no finite burn rate; its example alert uses any observed miss instead.
- The sample alert requires BOTH windows to breach and meet the minimum count. Its burn threshold consumes the chosen fraction of a period's budget in the long window, assuming representative volume.
- The metric contract needs a durable deadline observer, initialized outcome counters, and independent freshness/completeness monitoring. PromQL increase extrapolates between scrapes; exact trace counts here do not.