A Synthetic Data Factory Needs Four Gates Before It Needs More Prompts

You wire up a for loop that calls your teacher model's API, append every response to a JSONL file, and let it run overnight against a few thousand prompts. By morning you have tens of thousands of rows and a fine-tuning job queued.

What you don't have yet: any idea how many of those rows are malformed JSON, how many are near-identical restatements of the same support ticket, or how many happen to answer a question that's sitting in your evaluation holdout.

# THE SHORTCUT
results = []
for prompt in prompts:
    response = teacher.generate(prompt)
    results.append(response)  # unfiltered, unvalidated, unversioned
with open("data.jsonl", "w") as f:
    for r in results:
        f.write(json.dumps(r) + "\n")

Nothing here checks whether the teacher's output parses, whether it repeats itself, or whether it leaked into the set you'll use to grade the student later. Distillation — training a smaller student model to reproduce a larger teacher model's behavior — is only as good as the data that crosses that boundary, and generated data is not automatically clean data.

Microsoft's Phi-1 experiment is the strongest public evidence that curated synthetic data can substitute for scale, and also the strongest warning about what happens when you skip the filtering. The team trained a 1.3B-parameter coding model for four days on eight A100 GPUs, using 6B tokens of selected web text and 1B tokens of synthetically generated "textbook and exercise" content from GPT-3.5. It scored 50.6% pass@1 — the fraction of problems solved on the first sampled attempt — on HumanEval, and 55.5% on MBPP, results that were competitive with far larger models at the time Gunasekar et al., "Textbooks Are All You Need". That's one coding benchmark for one model family, not a universal exchange rate between data quality and model size — the same paper reports Phi-1 struggled with uncommon APIs, was brittle to stylistic variation, and had a "high error rate" in some of its own GPT-3.5-generated data. The follow-up model, phi-1.5, broadened the approach to general reasoning and its own report documents hallucinations and the potential for toxic or biased generations Li et al., "Textbooks Are All You Need II". Synthetic data concentrates whatever you optimized for — including defects, if you didn't build anything to catch them.

This post builds that catch. We'll walk through a small, runnable synthetic data factory: a pipeline that turns raw teacher output into a versioned, auditable training set through four gates — schema validation, deduplication, decontamination, and a signed manifest — with a quarantine, a holding area for rejected rows that preserves the reason they failed instead of silently dropping them.

Before any of that: decide what you're generating for.

Use the panel above to change generated volume and the pass rate at admission, validation, deduplication, and release review. It keeps every rejected row in a stage-specific quarantine ledger — the same generate → gate → release loop the rest of this post builds in code.

Budget the curriculum before you budget the API calls

A pile of generated rows tells you how much data you have. It says nothing about whether you have the right data. A curriculum — a versioned allocation of generation budget across the capabilities, difficulty levels, and languages your product actually needs — turns "generate more" into "generate this, specifically, because we don't have enough of it yet."

Start with the smallest useful axes: intent (what the customer wants), difficulty, language, and risk. Each combination is a cell. Give every cell a minimum (a policy claim: below this count, the release is knowingly incomplete) and a weight (a spending preference for whatever budget is left over). Those are different numbers and conflating them is a real failure mode — a rare safety cell can have a low historical weight and still need a nonzero minimum.

Turning integer weights into integer quotas is a classic largest-remainder apportionment problem:

from dataclasses import dataclass
 
 
@dataclass(frozen=True)
class CurriculumCell:
    cell_id: str
    minimum: int
    weight: int
 
 
def allocate_budget(
    cells: tuple[CurriculumCell, ...],
    budget: int,
) -> dict[str, int]:
    total_weight = sum(cell.weight for cell in cells)
    if total_weight == 0:
        raise ValueError("at least one cell needs positive weight")
 
    quotas = {
        cell.cell_id: budget * cell.weight // total_weight  # ①
        for cell in cells
    }
    remaining = budget - sum(quotas.values())
 
    ranked = sorted(
        cells,
        key=lambda c: (-(budget * c.weight % total_weight), c.cell_id),  # ②
    )
    for cell in ranked[:remaining]:
        quotas[cell.cell_id] += 1
 
    for cell in cells:
        if quotas[cell.cell_id] < cell.minimum:                          # ③
            print(
                f"WARNING: {cell.cell_id} underfunded "
                f"({quotas[cell.cell_id]} of {cell.minimum} minimum)"
            )
    return quotas

① Integer division gives every cell its deterministic base share. ② Whatever's left over goes to the cells with the largest fractional remainder first, and cell_id breaks ties so two runs with the same inputs produce the same allocation. ③ The function doesn't silently pad under-budgeted cells to their minimum — it surfaces the gap. An "underfunded" warning is more useful than a quietly wrong total, because it tells a human exactly where to spend the next budget increase.

This is deliberately a diagnostic, not an enforcement mechanism: it tells you where the curriculum is thin. It's on you to decide whether an underfunded safety cell blocks the release or gets a time-boxed waiver.

Gate 1: schema validation catches the JSON that never should have shipped

Once generation starts, every teacher response is untrusted input, full stop. The cheapest, highest-value check is shape: does the payload parse as JSON, and does it have every field the student training loop expects?

pip install pydantic

import hashlib
import json
import time
from typing import Optional
 
from pydantic import BaseModel, Field, ValidationError
 
 
class SupportTicket(BaseModel):
    ticket_id: str
    category: str = Field(..., description="billing | technical | escalation")
    customer_query: str
    reasoning_trace: str = Field(
        ..., description="Step-by-step rationale from the teacher"
    )
    resolution: str
 
 
class QuarantineRecord(BaseModel):
    raw_payload: str
    gate_name: str
    failure_reason: str
    timestamp: float = Field(default_factory=time.time)

reasoning_trace is worth calling out: asking the teacher for a rationale before the final answer, not just the answer itself, is the difference between a demonstration row and an explanation row (more on that distinction below). It costs nothing extra to request and it's a required field here, so a response missing it fails the gate instead of silently training a student that only ever sees conclusions.

def validate_schema(raw_text: str) -> tuple[SupportTicket | None, QuarantineRecord | None]:
    try:
        parsed = json.loads(raw_text)
        return SupportTicket(**parsed), None
    except (json.JSONDecodeError, ValidationError) as e:
        return None, QuarantineRecord(
            raw_payload=raw_text,
            gate_name="Gate-1-Schema",
            failure_reason=f"SCHEMA_VALIDATION_ERROR: {e}",
        )

Pydantic's ValidationError gives you the exact missing or malformed field, which is worth preserving verbatim in the quarantine record — "resolution field required" is actionable; "row 3 failed" is not.

Gate 2: exact deduplication is the floor, not the ceiling

The teacher will restate itself. Ask for a hundred billing complaints and you'll get a dozen genuinely different scenarios wearing a hundred different outfits. Deduplication removes rows that teach effectively the same lesson twice.

def content_hash(category: str, query: str) -> str:
    normalized = f"{category}:{query}".strip().lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

This catches exact and near-exact restatements after normalization. It will not catch a paraphrase — "I got billed twice for invoice #8841" and "Invoice 8841 shows a duplicate charge" hash to different values but teach the same lesson. Production pipelines add a second, looser check: token-set Jaccard similarity against already-accepted rows (using the standard intersection / union set overlap), or a full MinHash locality-sensitive-hashing index when the accepted set gets too large to compare pairwise — see datasketch for a maintained implementation. Whichever you pick, watch the threshold: too loose and cosmetic rewrites inflate your row count; too tight and you erase real diversity, like a legitimately rare phrasing of a boundary case.

Lee and coauthors found extensive duplication and train/test overlap in common language-model training sets, and that deduplicating those sets reduced memorized output and made evaluation more reliable. Their experiments were on web-scraped corpora, not synthetic support tickets, but the release lesson transfers directly: duplicate control and contamination control are separate checks, and skipping either one hides real problems behind a healthy-looking row count.

Gate 3: decontamination protects the set you'll grade the student on

This is the gate the naive loop skips entirely, and it's the one that produces the most embarrassing failure: a student that scores suspiciously well on your eval set because a paraphrase of the eval set was in its training data.

Decontamination checks generated rows against a holdout fingerprint set — hashes of your evaluation examples — before those rows are eligible for training.

def check_contamination(
    category: str,
    query: str,
    hidden_eval_hashes: set[str],
) -> Optional[QuarantineRecord]:
    row_hash = content_hash(category, query)                       # ①
    if row_hash in hidden_eval_hashes:                              # ②
        return QuarantineRecord(
            raw_payload=f"{category}:{query}",
            gate_name="Gate-3-Decontamination",
            failure_reason=f"HOLDOUT_OVERLAP: hash={row_hash[:8]}",
        )
    return None                                                     # ③

① Reuse the same normalized hash from the dedup gate — one canonicalization function, two comparison sets. ② The generation code never sees the actual eval prompts, only their fingerprints; that's what keeps the holdout uncontaminated even when the same team writes both the eval set and the generation prompts. ③ A pass here only rules out exact overlap. A paraphrased eval question — the harder, more common contamination path — needs the same near-duplicate machinery from Gate 2, run against the holdout instead of the training set.

Gate 4: the manifest is the only thing training should trust

A dataset becomes a release — something a training job is allowed to consume — when its content, its rejection history, and an explicit sign-off are bound together in one artifact. Copying the accepted rows into a file is not a release; it's a rumor.

class DatasetManifest(BaseModel):
    version: str
    created_at: float
    total_generated: int
    passed_samples: int
    quarantined_samples: int
    category_distribution: dict[str, int]
    content_hash: str
 
 
def build_manifest(
    version: str,
    passed: list[SupportTicket],
    quarantined: list[QuarantineRecord],
) -> DatasetManifest:
    dataset_str = "\n".join(t.model_dump_json() for t in passed)
    distribution: dict[str, int] = {}
    for t in passed:
        distribution[t.category] = distribution.get(t.category, 0) + 1
 
    return DatasetManifest(
        version=version,
        created_at=time.time(),
        total_generated=len(passed) + len(quarantined),
        passed_samples=len(passed),
        quarantined_samples=len(quarantined),
        category_distribution=distribution,
        content_hash=hashlib.sha256(dataset_str.encode("utf-8")).hexdigest(),
    )

The content_hash is what a training job checks before reading a single row: recompute the hash of the file you were handed, compare it to the manifest, and refuse to train if they don't match. That's the difference between "we have a training set" and "we have a training set we can prove hasn't been quietly edited since release."

Running the factory end to end

def run_factory(raw_outputs: list[str], hidden_eval_hashes: set[str]) -> None:
    passed: list[SupportTicket] = []
    quarantined: list[QuarantineRecord] = []
    seen_hashes: set[str] = set()
 
    for raw in raw_outputs:
        ticket, failure = validate_schema(raw)
        if failure:
            quarantined.append(failure)
            continue
 
        row_hash = content_hash(ticket.category, ticket.customer_query)
 
        contamination = check_contamination(
            ticket.category, ticket.customer_query, hidden_eval_hashes
        )
        if contamination:
            quarantined.append(contamination)
            continue
 
        if row_hash in seen_hashes:
            quarantined.append(
                QuarantineRecord(
                    raw_payload=raw,
                    gate_name="Gate-2-Deduplication",
                    failure_reason=f"DUPLICATE: hash={row_hash[:8]}",
                )
            )
            continue
 
        seen_hashes.add(row_hash)
        passed.append(ticket)
 
    manifest = build_manifest("v0.1.0", passed, quarantined)
    print(manifest.model_dump_json(indent=2))

Note the gate order: schema first (cheapest check, catches structurally broken rows before they cost you anything else), then contamination (protect the holdout before anything else touches it), then deduplication (only compare rows that already cleared the first two gates). Reordering this matters less for correctness than for cost — always run the cheapest gate first.

Choosing what the teacher generates, not just how much

A quality gate filters what the teacher already produced. A generation mode decides what you ask for in the first place, and the choice has real cost and validation trade-offs:

Start with direct demonstrations, the cheapest mode to validate. Add explanations for hidden rules the student keeps missing, perturbations to test whether it holds the right label under rewording, and counterexamples specifically for boundary cases (refusals, escalations). Self-Instruct demonstrated the broader bootstrapping version of this loop — generate instructions, generate inputs and outputs, then filter invalid or overly similar samples before they're admitted. WizardLM's Evol-Instruct goes further, deliberately rewriting instructions toward more depth or breadth and eliminating evolutions that fail a quality check. Best-of-N is the expensive mode: it's only worth its cost when you've measured real variance across candidates and have a judge you trust to pick correctly — otherwise you're just paying N times to reinforce the teacher's favorite mistake.

What could go wrong

• Aggressive near-duplicate thresholds erase real diversity. A rare, legitimately short "where's my refund" can register as a near-duplicate of a longer, similar-topic ticket and get quarantined for the wrong reason. Calibrate the threshold per curriculum cell, not globally.
• Exact-hash decontamination misses paraphrased leakage. Gate 3 above only catches identical fingerprints; the harder problem — a generated row that paraphrases an eval question — needs the same similarity machinery used for near-duplicates, applied against the holdout.
• Rights and licensing gaps hide behind a clean pipeline. Datasheets for Datasets calls for documented motivation, composition, and collection process; the Data Provenance Initiative found frequent missing or incorrect license metadata in widely used training sets. A schema-valid, deduplicated, decontaminated row can still be a row you had no right to generate or use. None of the four gates above check permissions — that's a fifth, separate gate.
• A quarantine directory that training can read is not a quarantine. If your training loader globs every .jsonl file in a folder, a rejected row sitting next to the accepted ones is one misconfigured path away from being trainable. Keep quarantine and release physically or access-separated.

Key takeaways

• Curriculum allocation comes before generation — decide what you're short on before you spend teacher budget filling in what you already have.
• Schema validation, deduplication, decontamination, and a hash-bound manifest are four separate questions. Collapsing them into one "quality score" hides which one actually failed.
• Phi-1's results are evidence that curated synthetic data can substitute for scale on one coding benchmark — not a license to skip filtering because "the teacher is smart."
• A quarantine record with a reason code is more valuable than a smaller, silently-cleaner-looking dataset.

What's next

Even a perfectly gated dataset doesn't guarantee a good student — the next post in this series covers ten places distillation fails after the data is clean, from training a model that only imitates the teacher's tone to shipping a quantized artifact nobody re-evaluated.

This piece is adapted from Chapter 4, "Manufacturing the Curriculum," of Distilled: The Engineering of Small, Fast, Cheap AI Models.

Sources and References

Synthetic data papers

• Gunasekar et al., "Textbooks Are All You Need" — Phi-1 training data mix and HumanEval/MBPP results
• Li et al., "Textbooks Are All You Need II" — phi-1.5 hallucination and bias disclosure
• Wang et al., "Self-Instruct" — bootstrapped instruction generation with filtering
• Xu et al., "WizardLM" (Evol-Instruct) — instruction evolution with an elimination stage

Data quality and provenance

• Lee et al., "Deduplicating Training Data Makes Language Models Better"
• Gebru et al., "Datasheets for Datasets"
• Longpre et al., "The Data Provenance Initiative"

Tools

• Pydantic documentation
• Python hashlib documentation
• datasketch (MinHash LSH implementation)