Training runs
Status: demonstrated — the models this produced are the ones the results measure.
This is the as-built record of the 2026 synthetic prototype's training: the base model, the vocabulary reduction, the concrete configuration of both rounds, and the artifact chain each pool produced. For why the two rounds exist and what each one is for, see the two training rounds — this page owns the how, as it actually ran.
One caveat governs every number below: these are this round's values, not constants of the method. Context lengths, ranks, learning rates, and epoch counts have all changed between builds and will change again for real data. Treat them as a faithful lab notebook, not a specification.
The base model
All five models start from the same open base: Qwen3.5-9B-Base, loaded text-only. Its architecture is hybrid linear-attention — only 8 of its 32 layers use full attention; the other 24 are linear-attention layers whose memory cost does not grow with context length. That one fact carries most of the prototype's hardware story: the KV cache only exists for those 8 layers, so the model's full ~262k-token context fits in roughly 15 GB on a 16 GB consumer GPU. Every inference node in the prototype is exactly that — one 16 GB card running the full context (hardware notes has the measured throughput).
A smaller vocabulary
The base tokenizer ships with ~248k tokens; the PRISM corpus uses a narrow slice of them — table syntax, dates, billing codes, and their short descriptions. Before any training, the vocabulary was pruned to ~126k tokens, shrinking the embedding and output matrices and trimming both VRAM and per-token compute at training and inference alike. The prune only removes tokens: every surviving token keeps its text byte-for-byte, so the reduced models read and write the six-column format unchanged. All five pools share this one reduced vocabulary and ID space — the single deliberate commonality in a system where nothing else crosses pools.
Primary round, as run
One full-parameter SFT run per pool, on that pool's complete patient histories — continued pre-training that deliberately over-specializes the model until continuing the table is all it can do. This round's configuration: a ~96k-token training context, long enough to take each patient as one unbroken sequence; 2 epochs, with the 1-epoch state kept as a snapshot; one rented high-VRAM GPU (96 GB class) per pool, which comfortably fits a full-parameter 9B run. The five runs were fanned out in parallel, one pool per GPU, and no run ever saw another pool's data.
Secondary round, as run
Each pool then got a LoRA steering adapter, trained on that pool's own primary and nothing else. The recipe that worked is pure SFT on the chosen continuations only — the rejected column of the preference-shaped training data goes unused. This round's values: LoRA rank 32, alpha 64, dropout 0; 2 epochs; learning rate 3e-5 on a cosine schedule with a short warmup; effective batch 8; ~64k-token context with a ~1.5k-token completion budget. The adapter is kept separate from the base weights — never merged — and applied at inference per request at a strength of ~1.0 (running the ensemble covers how).
That recipe was not the plan. The design assumption was preference training over chosen/rejected pairs; it ranked the right continuation but never emitted it, and it was abandoned entirely — the most instructive detour of the prototype, told in full in the two training rounds. The operational lesson survives here as practice: adapter selection was gated on free-generation behavior at several strengths, never on training metrics — this round's winning adapter logged inverted train accuracy and margin.
The per-pool artifact chain
Each pool's training leaves a strictly vertical chain of artifacts, and each inference node serves exactly one pool's chain:
| stage | artifact (one per pool) | form | role |
|---|---|---|---|
| primary, 1 epoch | training snapshot | full weights | checkpoint kept for comparison; not deployed |
| primary, 2 epochs | the pool base | full weights | the final primary model everything downstream derives from |
| quantization | node base | 8-bit quantized (~8.5 GB) | what the pool's inference node actually serves |
| secondary | steering adapter | LoRA weights (~173 MB) | applied to the node base per request, at ~1.0 |
Nothing in this chain crosses pools — no shared training data, no shared weights, no merged or swapped adapter. That independence is not hygiene; it is the architecture, and it is what lets the results read cross-pool agreement as evidence.
None of these values are claimed as optima. They are the configuration that met the generation gate on synthetic data, at prototype scale; the real-data phase re-derives them.
See also
- The two training rounds — why primary + secondary, and the abandoned preference blend
- Hardware notes — measured throughput and the KV-cache result
- Running the ensemble — how these artifacts served the 16,250-run grid
- Results — what the models this produced actually did