Background jobs / Scheduling simulator
Tenant Fairness & Backpressure
Can a small interactive tenant get timely service while another tenant fills the queue with long-running work?
Interactive calculations run in your browser; the initial example is pre-rendered. There are no accounts, uploads, or live queue connections. Inputs stay in page memory; the site does not persist them, put them in URLs, or send them to analytics. A worksheet download includes only what you explicitly export.
Same workload, different access to workers
| Policy | Completed / accepted | Unfinished | Completed jobs/s | Rejected | Not yet arrived | Missed / matured deadlines | P95 completed latency | Worker-seconds | Worker starts |
|---|---|---|---|---|---|---|---|---|---|
| FIFO | 36 / 36 | 0 | 0.15 | 0 | 0 | 24 / 36 | 92 s | 720 | 3 |
| Tenant round robin | 36 / 36 | 0 | 0.15 | 0 | 0 | 24 / 36 | 108 s | 720 | 3 |
| Strict priority | 36 / 36 | 0 | 0.15 | 0 | 0 | 25 / 36 | 110 s | 720 | 3 |
| Round robin + tenant cap | 36 / 36 | 0 | 0.15 | 0 | 0 | 26 / 36 | 190 s | 720 | 3 |
- FIFO
- Tenant round robin
- Strict priority
- Round robin + tenant cap
| Policy / tenant | Completed / accepted | Pending / rejected | P95 completed latency | P95 wait of started jobs | Oldest unstarted wait |
|---|---|---|---|---|---|
| FIFO / bulk | 24 / 24 | 0 / 0 | 66 s | 52 s | None waiting |
| FIFO / interactive | 6 / 6 | 0 / 0 | 73 s | 71 s | None waiting |
| FIFO / reports | 6 / 6 | 0 / 0 | 92 s | 72 s | None waiting |
| Tenant round robin / bulk | 24 / 24 | 0 / 0 | 108 s | 98 s | None waiting |
| Tenant round robin / interactive | 6 / 6 | 0 / 0 | 33 s | 31 s | None waiting |
| Tenant round robin / reports | 6 / 6 | 0 / 0 | 52 s | 32 s | None waiting |
| Strict priority / bulk | 24 / 24 | 0 / 0 | 110 s | 96 s | None waiting |
| Strict priority / interactive | 6 / 6 | 0 / 0 | 17 s | 15 s | None waiting |
| Strict priority / reports | 6 / 6 | 0 / 0 | 46 s | 26 s | None waiting |
| Round robin + tenant cap / bulk | 24 / 24 | 0 / 0 | 190 s | 188 s | None waiting |
| Round robin + tenant cap / interactive | 6 / 6 | 0 / 0 | 2 s | 0 s | None waiting |
| Round robin + tenant cap / reports | 6 / 6 | 0 / 0 | 100 s | 80 s | None waiting |
Round robin gives turns, not equal processing time. Long jobs can still occupy every worker. A concurrency cap creates room for another tenant but may reduce utilization. Try admission overload: a fair scheduler cannot rescue a quiet tenant whose work was rejected before entering the queue.
Inspect the reproducible traces
FIFO: job and clock trace
| Tenant | Completed / accepted | Pending / rejected | P95 completed latency | P95 wait of started jobs | Oldest unstarted wait |
|---|---|---|---|---|---|
| bulk | 24 / 24 | 0 / 0 | 66 s | 52 s | None waiting |
| interactive | 6 / 6 | 0 / 0 | 73 s | 71 s | None waiting |
| reports | 6 / 6 | 0 / 0 | 92 s | 72 s | None waiting |
| Job | Arrives / service | Starts | Finishes | State at cutoff |
|---|---|---|---|---|
| bulk-1 | 0 s / 20 s | 0 s | 20 s | completed |
| bulk-2 | 0 s / 2 s | 0 s | 2 s | completed |
| bulk-3 | 0 s / 2 s | 0 s | 2 s | completed |
| bulk-4 | 0 s / 20 s | 2 s | 22 s | completed |
| bulk-5 | 0 s / 2 s | 2 s | 4 s | completed |
| bulk-6 | 0 s / 2 s | 4 s | 6 s | completed |
| bulk-7 | 0 s / 20 s | 6 s | 26 s | completed |
| bulk-8 | 0 s / 2 s | 20 s | 22 s | completed |
| bulk-9 | 0 s / 2 s | 22 s | 24 s | completed |
| bulk-10 | 0 s / 20 s | 22 s | 42 s | completed |
| bulk-11 | 0 s / 2 s | 24 s | 26 s | completed |
| bulk-12 | 0 s / 2 s | 26 s | 28 s | completed |
| bulk-13 | 0 s / 20 s | 26 s | 46 s | completed |
| bulk-14 | 0 s / 2 s | 28 s | 30 s | completed |
| bulk-15 | 0 s / 2 s | 30 s | 32 s | completed |
| bulk-16 | 0 s / 20 s | 32 s | 52 s | completed |
| bulk-17 | 0 s / 2 s | 42 s | 44 s | completed |
| bulk-18 | 0 s / 2 s | 44 s | 46 s | completed |
| bulk-19 | 0 s / 20 s | 46 s | 66 s | completed |
| bulk-20 | 0 s / 2 s | 46 s | 48 s | completed |
| Time | Queued / running | Ready / starting | Retiring / live | Oldest wait | Completed / rejected |
|---|---|---|---|---|---|
| 0 s | 21 / 3 | 3 / 0 | 0 / 3 | 0 s | 0 / 0 |
| 1 s | 22 / 3 | 3 / 0 | 0 / 3 | 1 s | 0 / 0 |
| 2 s | 21 / 3 | 3 / 0 | 0 / 3 | 2 s | 2 / 0 |
| 3 s | 21 / 3 | 3 / 0 | 0 / 3 | 3 s | 2 / 0 |
| 4 s | 20 / 3 | 3 / 0 | 0 / 3 | 4 s | 3 / 0 |
| 5 s | 21 / 3 | 3 / 0 | 0 / 3 | 5 s | 3 / 0 |
| 6 s | 21 / 3 | 3 / 0 | 0 / 3 | 6 s | 4 / 0 |
| 7 s | 21 / 3 | 3 / 0 | 0 / 3 | 7 s | 4 / 0 |
| 8 s | 21 / 3 | 3 / 0 | 0 / 3 | 8 s | 4 / 0 |
| 9 s | 22 / 3 | 3 / 0 | 0 / 3 | 9 s | 4 / 0 |
| 10 s | 23 / 3 | 3 / 0 | 0 / 3 | 10 s | 4 / 0 |
| 11 s | 23 / 3 | 3 / 0 | 0 / 3 | 11 s | 4 / 0 |
| 12 s | 23 / 3 | 3 / 0 | 0 / 3 | 12 s | 4 / 0 |
| 13 s | 24 / 3 | 3 / 0 | 0 / 3 | 13 s | 4 / 0 |
| 14 s | 25 / 3 | 3 / 0 | 0 / 3 | 14 s | 4 / 0 |
| 15 s | 25 / 3 | 3 / 0 | 0 / 3 | 15 s | 4 / 0 |
| 16 s | 25 / 3 | 3 / 0 | 0 / 3 | 16 s | 4 / 0 |
| 17 s | 26 / 3 | 3 / 0 | 0 / 3 | 17 s | 4 / 0 |
| 18 s | 27 / 3 | 3 / 0 | 0 / 3 | 18 s | 4 / 0 |
| 19 s | 27 / 3 | 3 / 0 | 0 / 3 | 19 s | 4 / 0 |
Tenant round robin: job and clock trace
| Tenant | Completed / accepted | Pending / rejected | P95 completed latency | P95 wait of started jobs | Oldest unstarted wait |
|---|---|---|---|---|---|
| bulk | 24 / 24 | 0 / 0 | 108 s | 98 s | None waiting |
| interactive | 6 / 6 | 0 / 0 | 33 s | 31 s | None waiting |
| reports | 6 / 6 | 0 / 0 | 52 s | 32 s | None waiting |
| Job | Arrives / service | Starts | Finishes | State at cutoff |
|---|---|---|---|---|
| bulk-1 | 0 s / 20 s | 0 s | 20 s | completed |
| bulk-2 | 0 s / 2 s | 0 s | 2 s | completed |
| bulk-3 | 0 s / 2 s | 0 s | 2 s | completed |
| bulk-4 | 0 s / 20 s | 4 s | 24 s | completed |
| bulk-5 | 0 s / 2 s | 22 s | 24 s | completed |
| bulk-6 | 0 s / 2 s | 26 s | 28 s | completed |
| bulk-7 | 0 s / 20 s | 42 s | 62 s | completed |
| bulk-8 | 0 s / 2 s | 50 s | 52 s | completed |
| bulk-9 | 0 s / 2 s | 62 s | 64 s | completed |
| bulk-10 | 0 s / 20 s | 64 s | 84 s | completed |
| bulk-11 | 0 s / 2 s | 66 s | 68 s | completed |
| bulk-12 | 0 s / 2 s | 68 s | 70 s | completed |
| bulk-13 | 0 s / 20 s | 70 s | 90 s | completed |
| bulk-14 | 0 s / 2 s | 74 s | 76 s | completed |
| bulk-15 | 0 s / 2 s | 76 s | 78 s | completed |
| bulk-16 | 0 s / 20 s | 78 s | 98 s | completed |
| bulk-17 | 0 s / 2 s | 84 s | 86 s | completed |
| bulk-18 | 0 s / 2 s | 86 s | 88 s | completed |
| bulk-19 | 0 s / 20 s | 88 s | 108 s | completed |
| bulk-20 | 0 s / 2 s | 90 s | 92 s | completed |
| Time | Queued / running | Ready / starting | Retiring / live | Oldest wait | Completed / rejected |
|---|---|---|---|---|---|
| 0 s | 21 / 3 | 3 / 0 | 0 / 3 | 0 s | 0 / 0 |
| 1 s | 22 / 3 | 3 / 0 | 0 / 3 | 1 s | 0 / 0 |
| 2 s | 21 / 3 | 3 / 0 | 0 / 3 | 2 s | 2 / 0 |
| 3 s | 21 / 3 | 3 / 0 | 0 / 3 | 3 s | 2 / 0 |
| 4 s | 20 / 3 | 3 / 0 | 0 / 3 | 4 s | 3 / 0 |
| 5 s | 21 / 3 | 3 / 0 | 0 / 3 | 5 s | 3 / 0 |
| 6 s | 22 / 3 | 3 / 0 | 0 / 3 | 6 s | 3 / 0 |
| 7 s | 22 / 3 | 3 / 0 | 0 / 3 | 7 s | 3 / 0 |
| 8 s | 22 / 3 | 3 / 0 | 0 / 3 | 8 s | 3 / 0 |
| 9 s | 23 / 3 | 3 / 0 | 0 / 3 | 9 s | 3 / 0 |
| 10 s | 24 / 3 | 3 / 0 | 0 / 3 | 10 s | 3 / 0 |
| 11 s | 24 / 3 | 3 / 0 | 0 / 3 | 11 s | 3 / 0 |
| 12 s | 24 / 3 | 3 / 0 | 0 / 3 | 12 s | 3 / 0 |
| 13 s | 25 / 3 | 3 / 0 | 0 / 3 | 13 s | 3 / 0 |
| 14 s | 26 / 3 | 3 / 0 | 0 / 3 | 14 s | 3 / 0 |
| 15 s | 26 / 3 | 3 / 0 | 0 / 3 | 15 s | 3 / 0 |
| 16 s | 26 / 3 | 3 / 0 | 0 / 3 | 16 s | 3 / 0 |
| 17 s | 27 / 3 | 3 / 0 | 0 / 3 | 17 s | 3 / 0 |
| 18 s | 28 / 3 | 3 / 0 | 0 / 3 | 18 s | 3 / 0 |
| 19 s | 28 / 3 | 3 / 0 | 0 / 3 | 19 s | 3 / 0 |
Strict priority: job and clock trace
| Tenant | Completed / accepted | Pending / rejected | P95 completed latency | P95 wait of started jobs | Oldest unstarted wait |
|---|---|---|---|---|---|
| bulk | 24 / 24 | 0 / 0 | 110 s | 96 s | None waiting |
| interactive | 6 / 6 | 0 / 0 | 17 s | 15 s | None waiting |
| reports | 6 / 6 | 0 / 0 | 46 s | 26 s | None waiting |
| Job | Arrives / service | Starts | Finishes | State at cutoff |
|---|---|---|---|---|
| bulk-1 | 0 s / 20 s | 0 s | 20 s | completed |
| bulk-2 | 0 s / 2 s | 0 s | 2 s | completed |
| bulk-3 | 0 s / 2 s | 0 s | 2 s | completed |
| bulk-4 | 0 s / 20 s | 4 s | 24 s | completed |
| bulk-5 | 0 s / 2 s | 46 s | 48 s | completed |
| bulk-6 | 0 s / 2 s | 48 s | 50 s | completed |
| bulk-7 | 0 s / 20 s | 50 s | 70 s | completed |
| bulk-8 | 0 s / 2 s | 64 s | 66 s | completed |
| bulk-9 | 0 s / 2 s | 66 s | 68 s | completed |
| bulk-10 | 0 s / 20 s | 66 s | 86 s | completed |
| bulk-11 | 0 s / 2 s | 68 s | 70 s | completed |
| bulk-12 | 0 s / 2 s | 70 s | 72 s | completed |
| bulk-13 | 0 s / 20 s | 70 s | 90 s | completed |
| bulk-14 | 0 s / 2 s | 72 s | 74 s | completed |
| bulk-15 | 0 s / 2 s | 74 s | 76 s | completed |
| bulk-16 | 0 s / 20 s | 76 s | 96 s | completed |
| bulk-17 | 0 s / 2 s | 86 s | 88 s | completed |
| bulk-18 | 0 s / 2 s | 88 s | 90 s | completed |
| bulk-19 | 0 s / 20 s | 90 s | 110 s | completed |
| bulk-20 | 0 s / 2 s | 90 s | 92 s | completed |
| Time | Queued / running | Ready / starting | Retiring / live | Oldest wait | Completed / rejected |
|---|---|---|---|---|---|
| 0 s | 21 / 3 | 3 / 0 | 0 / 3 | 0 s | 0 / 0 |
| 1 s | 22 / 3 | 3 / 0 | 0 / 3 | 1 s | 0 / 0 |
| 2 s | 21 / 3 | 3 / 0 | 0 / 3 | 2 s | 2 / 0 |
| 3 s | 21 / 3 | 3 / 0 | 0 / 3 | 3 s | 2 / 0 |
| 4 s | 20 / 3 | 3 / 0 | 0 / 3 | 4 s | 3 / 0 |
| 5 s | 21 / 3 | 3 / 0 | 0 / 3 | 5 s | 3 / 0 |
| 6 s | 22 / 3 | 3 / 0 | 0 / 3 | 6 s | 3 / 0 |
| 7 s | 22 / 3 | 3 / 0 | 0 / 3 | 7 s | 3 / 0 |
| 8 s | 22 / 3 | 3 / 0 | 0 / 3 | 8 s | 3 / 0 |
| 9 s | 23 / 3 | 3 / 0 | 0 / 3 | 9 s | 3 / 0 |
| 10 s | 24 / 3 | 3 / 0 | 0 / 3 | 10 s | 3 / 0 |
| 11 s | 24 / 3 | 3 / 0 | 0 / 3 | 11 s | 3 / 0 |
| 12 s | 24 / 3 | 3 / 0 | 0 / 3 | 12 s | 3 / 0 |
| 13 s | 25 / 3 | 3 / 0 | 0 / 3 | 13 s | 3 / 0 |
| 14 s | 26 / 3 | 3 / 0 | 0 / 3 | 14 s | 3 / 0 |
| 15 s | 26 / 3 | 3 / 0 | 0 / 3 | 15 s | 3 / 0 |
| 16 s | 26 / 3 | 3 / 0 | 0 / 3 | 16 s | 3 / 0 |
| 17 s | 27 / 3 | 3 / 0 | 0 / 3 | 17 s | 3 / 0 |
| 18 s | 28 / 3 | 3 / 0 | 0 / 3 | 18 s | 3 / 0 |
| 19 s | 28 / 3 | 3 / 0 | 0 / 3 | 19 s | 3 / 0 |
Round robin + tenant cap: job and clock trace
| Tenant | Completed / accepted | Pending / rejected | P95 completed latency | P95 wait of started jobs | Oldest unstarted wait |
|---|---|---|---|---|---|
| bulk | 24 / 24 | 0 / 0 | 190 s | 188 s | None waiting |
| interactive | 6 / 6 | 0 / 0 | 2 s | 0 s | None waiting |
| reports | 6 / 6 | 0 / 0 | 100 s | 80 s | None waiting |
| Job | Arrives / service | Starts | Finishes | State at cutoff |
|---|---|---|---|---|
| bulk-1 | 0 s / 20 s | 0 s | 20 s | completed |
| bulk-2 | 0 s / 2 s | 20 s | 22 s | completed |
| bulk-3 | 0 s / 2 s | 22 s | 24 s | completed |
| bulk-4 | 0 s / 20 s | 24 s | 44 s | completed |
| bulk-5 | 0 s / 2 s | 44 s | 46 s | completed |
| bulk-6 | 0 s / 2 s | 46 s | 48 s | completed |
| bulk-7 | 0 s / 20 s | 48 s | 68 s | completed |
| bulk-8 | 0 s / 2 s | 68 s | 70 s | completed |
| bulk-9 | 0 s / 2 s | 70 s | 72 s | completed |
| bulk-10 | 0 s / 20 s | 72 s | 92 s | completed |
| bulk-11 | 0 s / 2 s | 92 s | 94 s | completed |
| bulk-12 | 0 s / 2 s | 94 s | 96 s | completed |
| bulk-13 | 0 s / 20 s | 96 s | 116 s | completed |
| bulk-14 | 0 s / 2 s | 116 s | 118 s | completed |
| bulk-15 | 0 s / 2 s | 118 s | 120 s | completed |
| bulk-16 | 0 s / 20 s | 120 s | 140 s | completed |
| bulk-17 | 0 s / 2 s | 140 s | 142 s | completed |
| bulk-18 | 0 s / 2 s | 142 s | 144 s | completed |
| bulk-19 | 0 s / 20 s | 144 s | 164 s | completed |
| bulk-20 | 0 s / 2 s | 164 s | 166 s | completed |
| Time | Queued / running | Ready / starting | Retiring / live | Oldest wait | Completed / rejected |
|---|---|---|---|---|---|
| 0 s | 23 / 1 | 3 / 0 | 0 / 3 | 0 s | 0 / 0 |
| 1 s | 23 / 2 | 3 / 0 | 0 / 3 | 1 s | 0 / 0 |
| 2 s | 23 / 3 | 3 / 0 | 0 / 3 | 2 s | 0 / 0 |
| 3 s | 23 / 2 | 3 / 0 | 0 / 3 | 3 s | 1 / 0 |
| 4 s | 23 / 2 | 3 / 0 | 0 / 3 | 4 s | 1 / 0 |
| 5 s | 23 / 3 | 3 / 0 | 0 / 3 | 5 s | 1 / 0 |
| 6 s | 24 / 3 | 3 / 0 | 0 / 3 | 6 s | 1 / 0 |
| 7 s | 24 / 2 | 3 / 0 | 0 / 3 | 7 s | 2 / 0 |
| 8 s | 24 / 2 | 3 / 0 | 0 / 3 | 8 s | 2 / 0 |
| 9 s | 24 / 3 | 3 / 0 | 0 / 3 | 9 s | 2 / 0 |
| 10 s | 25 / 3 | 3 / 0 | 0 / 3 | 10 s | 2 / 0 |
| 11 s | 25 / 2 | 3 / 0 | 0 / 3 | 11 s | 3 / 0 |
| 12 s | 25 / 2 | 3 / 0 | 0 / 3 | 12 s | 3 / 0 |
| 13 s | 25 / 3 | 3 / 0 | 0 / 3 | 13 s | 3 / 0 |
| 14 s | 26 / 3 | 3 / 0 | 0 / 3 | 14 s | 3 / 0 |
| 15 s | 26 / 2 | 3 / 0 | 0 / 3 | 15 s | 4 / 0 |
| 16 s | 26 / 2 | 3 / 0 | 0 / 3 | 16 s | 4 / 0 |
| 17 s | 26 / 3 | 3 / 0 | 0 / 3 | 17 s | 4 / 0 |
| 18 s | 27 / 3 | 3 / 0 | 0 / 3 | 18 s | 4 / 0 |
| 19 s | 27 / 2 | 3 / 0 | 0 / 3 | 19 s | 5 / 0 |
Aggregate capacity does not allocate service fairly
FIFO, job-level round robin, strict priority, and per-tenant concurrency limits allocate scarce worker slots differently. Compare pending work and rejected submissions alongside completion percentiles: a tenant that completes nothing has no reassuring latency percentile. Admission and scheduling are separate policies, and a finite trace can demonstrate lack of service without proving perpetual starvation.
Assumptions and limits
- A finite, deterministic trace runs on a one-second clock. Completions happen before arrivals, scaling, and dispatch. At the cutoff only completions are processed; new work is not started.
- Each worker runs one non-preemptible job at a time. A job occupies one downstream slot for its entire service duration. This is a concurrency bottleneck, not a universal requests-per-second model.
- Admission is tail-drop at a global unfinished-job limit, including running work. Rejection is explicit and never retried automatically; tenant scheduling does not make admission tenant-fair.
- Round robin rotates by JOB, not CPU time. Priority is strict but non-preemptive. The per-tenant cap restricts simultaneous running jobs; idle capacity can remain when every queued tenant is capped.
- Completion percentiles describe only jobs completed before cutoff. Pending and rejected counts, and oldest unstarted wait, must be read alongside them; no service before cutoff is not proof of permanent starvation.
- Deadlines use acceptance/arrival time and include waiting plus service. Only accepted jobs whose deadlines have matured by cutoff enter the deadline count; rejected work remains a separate outcome.
- Worker-seconds count booting, idle, busy, and gracefully draining workers in [0, cutoff). No price, real scheduler behavior, retry storm, network latency, or production SLO is inferred.
- Bulk jobs arrive at the chosen spacing; every third bulk job is long. Interactive jobs are short and start arriving at second 1; reports are long and start at second 2, both using the quiet-tenant interval.
- Priorities are interactive, reports, then bulk. The same trace runs through FIFO, job-round-robin, strict priority, and round-robin with a per-tenant running limit.
