Continuous Retraining
Status: vision — designed; depends on the real-data phase.
A production PRISM ensemble is never frozen and never lurches: one model retrains each day, so the whole ensemble rolls through a complete refresh on a continuous cycle while every other model keeps serving unchanged.
One model per day
The prototype trained its five models once and stopped; production cannot. Claims data accumulates daily, coding practices drift, treatments change, and — most importantly — PRISM's own suggestions produce outcomes that should feed back into training. The design answer is a rolling cycle: each day, exactly one pool's model is retrained from that pool's freshest data, validated, and swapped in. At the production-scale target of on the order of 100 pools, every model would be refreshed roughly quarterly — recent enough to track drift, staggered enough that the ensemble as a whole changes by only one voice per day.
One training job per day is also a deliberately boring workload. It is a fixed, predictable unit the cluster's management server can schedule around inference, rather than a periodic batch event that competes with screening for hardware.
The deeper reason is failure isolation. Training goes wrong in mundane ways — a bad data extract, a hardware fault, a run that converges somewhere strange. Under a rolling cycle, a bad training day produces one bad candidate for one pool; its predecessor keeps serving and the other models are untouched. A batch refresh would instead expose the whole ensemble to the same bad day at once. Since consensus across independent models is PRISM's entire unit of evidence, correlated failure is the one failure mode the architecture cannot absorb — the retraining schedule is shaped to make it structurally unlikely.
Temporal diversity as a feature
At any moment the ensemble spans the full cycle of training ages: one model trained yesterday, one nearly due for refresh, the rest spread between. This is not a compromise; it is a third axis of independence alongside disjoint data and separate weights. Freshly retrained models are the most sensitive to recent shifts; models trained months earlier anchor the vote against transients. A pattern only earns consensus if it is visible to models trained at different points in time — a one-month artifact of claims processing cannot recruit models that never saw that month.
The honest cost is bounded staleness: the oldest model in the ensemble always lags by up to one full cycle. Cycle length is a tunable trade between freshness and stability, not a constant.
The outcome feedback loop
Each retraining ingests the pool's newest claims — which include the downstream consequences of PRISM's own suggestions. The tracking code attached to every suggested test exists for compensation and cost coverage, but it has a second job here: it makes a followed suggestion identifiable in later claims, so its outcome can be labeled automatically.
| what the claims show | what it becomes in the next training set |
|---|---|
| Suggested test taken, condition confirmed, early treatment follows | a new GOOD example |
| Suggested test taken, no diagnosis results | a new NOPE example |
| Suggestion never acted on | no training signal — the outcome is unknown, and unknowns are not labeled |
These flow through the same automated three-pattern extraction that built the original training material; no manual annotation is involved. The loop this closes is unusual: the system's output, once resolved in the world, returns as labeled input. Its volume, though, depends entirely on physician uptake — early in a deployment the feedback will be a trickle, and the design has to work without it before it can improve with it.
The QA gate
No retrained model replaces its predecessor automatically. The candidate is evaluated behind a gate while the predecessor keeps serving, and the gate measures one thing: generation behavior. The candidate is fired at retrospective prompt sets with known dispositions and its firing and silence rates are read directly against the incumbent's — does it still fire on confirmed patterns, stay silent on controls, and vote at rates consistent with its pool's history?
The prototype made the reason concrete: the adapter that actually worked logged inverted training metrics — by loss and margin it looked like the worst candidate, and by generation it was the only one that did the job. Training metrics are not the signal and never gate a deployment.
A failed gate costs one day of freshness for one pool and nothing else; the failure is investigated, the incumbent stays. After deployment, voting rates remain monitored — a sudden shift in one model's firing behavior flags it for human review.
See also
The daily model cycle — the concrete train-screen-validate loop and the horizontal/vertical fleet swap this refresh runs inside