Building the Data
Status: demonstrated — the pipeline that manufactured the synthetic world.
The synthetic world was manufactured, not curated: a staged pipeline sampled statistical patient profiles, had language models write their billing histories, reconciled every column against real codesets, and only then — as a separate, seeded layer — injected the fabricated condition. This article walks those stages and states plainly which imperfections were accepted along the way.
Stage by stage
The pipeline lives in the prism-generator module and has a deliberate shape: deterministic code at both ends, a language model exactly once in the middle.
| # | stage | what happens | model in the loop? |
|---|---|---|---|
| 1 | profile sampling | a seeded statistical sampler builds every patient: Ohio-like demographics, chronic conditions with lifecycle postures, acute-event and screening counts, behavioral and lifestyle traits | no |
| 2 | spec build | a rule layer expands each profile into expected care — medication classes and refill cadences, follow-up visits, monitoring labs | no |
| 3 | prompt assembly | each spec renders as a bespoke markdown prompt; every instruction is positive (name what to include, never what to avoid), and sections empty for a patient are omitted entirely | no |
| 4 | history generation | prompts go out in large batches to a language model, which writes each patient's multi-year billing history — a different model and configuration per pool, deliberately | yes |
| 5 | tolerant parsing | generated tables are parsed with tolerance for model quirks (missing pipes, bare codes) into a raw staging table | no |
| 6 | canonical cleaning | every column is resolved against the real codesets into the canonical patients table — the finished background world | no |
| 7 | PVE injection | a separate seeded, idempotent layer picks carriers uniformly at random and weaves the fabricated condition over them | no |
Three of those choices deserve expansion.
Everything that decides is seeded code; the language model only writes. Who exists in the 25,000-patient population (five pools of 5,000), what conditions each carries, how conscientious they are about refills and follow-ups — all of it comes from a deterministic, seeded sampler with no model in the loop, so the population reproduces exactly and is auditable after the fact. The profiles deliberately encode the mess of real billing: variable adherence, missed appointments, coverage gaps, differing provider styles. The language model enters at exactly one stage, to render a profile into a plausible timeline in the six-column format, and it never sees or touches the condition the experiment is about.
Each pool got a different generator on purpose. The five pools were written by different upstream models and configurations — a deliberate heterogeneity whose rationale belongs to the synthetic world.
Cleaning is reconciliation, not invention. Raw model output is imperfect, so a tolerant parser accepts it as written, and a per-column cleaning pass then resolves each value against the real, versioned codesets — place of service, provider taxonomy, procedures and medications, diagnoses — preferring evidence from the surrounding row over guesswork about the code alone. The result is the canonical patients table: the complete background world, containing zero phenotype content.
The injection is a separate layer
Primary Veladrin Excess arrives last, and deliberately apart from everything above. The injector selects carriers uniformly at random — this round, 50 GOOD, 150 BAD, and 300 NOPE per pool — and weaves each arc's rows into the host timeline. It is seeded and idempotent: the same parameters reproduce the same injection, and a rerun replaces its output cleanly instead of accumulating. At run time it validates that every PVE code appears in zero base patients and in no real codeset, and placement obeys this round's constraints — the TEST never in the first 20 or last 10 rows of a timeline, BAD tests at least 40 rows deep so the FAKE cut has room to move earlier. Every injected row carries a construction mark; the marks are bookkeeping no model ever sees. Keeping the injection separate, parameterized, and checked is what makes the isolated signal enforceable rather than aspirational: the one layer that creates the signal is small enough to audit, and rerunning it never disturbs the background.
One database, many views
Postgres is the coordination medium for the whole prototype: the generator writes into it, training-data extraction reads out of it, and no module calls another directly. One base view merges each patient's background history and injected rows into a single renumbered chronological timeline, and every piece of training material is a SQL view over that one view. The primary corpus — every patient's full history, for continued pre-training — is one slice; the per-kind prompt/chosen/rejected triples for GOOD, BAD, NOPE, and FAKE (three-pattern learning) are others; even the fixed two-line table header is served from a view rather than hard-coded anywhere. A small extractor renders the views to per-pool JSONL files — this round, ten of them: a primary and a secondary set for each of the five pools — and those files are everything the training runs ever consume. The discipline this buys is a single source of truth: change the injection or the slicing, rerun the views, and every downstream artifact regenerates from the same place.
Accepted imperfections
The generated world has known flaws, kept on purpose because fixing them would buy realism the experiment does not need.
| quirk | what happened | why it stands |
|---|---|---|
| hallucinated medication codes | many generated drug product codes correspond to no marketed product | codes are opaque tokens to the models and are never used as real product identifiers; the structural distribution is what matters |
| under-spec row volumes | the generating models wrote noticeably fewer rows than the profiles asked for | timelines are shorter than specified, but every experimental constraint — lead-up depth, placement windows — is enforced on the rows that exist |
| front-loaded histories | some timelines cluster events early and thin out in later years | irregular density resembles real coverage gaps as much as it violates the spec, and placement randomization does not depend on uniform density |
None of these threaten the result, for one reason: the prototype's validity rests entirely on the isolated signal, not on background realism. The claim was never that this world is clinically faithful — it is that the only thing predictive of the TEST is the injected pattern, and the read is a binary string match. A hallucinated background medication code or a thin mid-life stretch makes the background messier, not the signal easier; if anything, a noisier background raises the bar for learning the pattern at all. What the quirks do rule out is reusing this corpus for anything that requires clinical realism — and nothing in the prototype does.
See also
- The isolated signal — the non-negotiable this pipeline exists to enforce
- The synthetic world — what the pipeline produced: pools, counts, deliberate heterogeneity
- Primary Veladrin Excess — the fabricated condition the injector weaves in
- Three-pattern learning — how the views turn timelines into lessons
- Standardized codesets — the real vocabularies the cleaning pass reconciles against