Evaluation Without a Hold-Out Set
Status: concept, demonstrated — this frame produced every number in the prototype results.
PRISM never sets aside a test set, and never will. The pool structure already makes every example an out-of-distribution test case for four of the five models — evaluation is built into the architecture, not bolted on after it.
Why there is no hold-out
Standard practice says: carve off a fraction of the data before training, never touch it, and score against it at the end. That ritual exists to answer one question — did the model learn the pattern, or memorize the examples? PRISM answers the same question with a stronger instrument that comes free with the design.
Each example trains exactly one pool's model (pools and consensus). The other four models trained on disjoint populations — in the 2026 synthetic proof of concept, populations generated by different upstream models on purpose — so to those four, the patient is out-of-distribution in the fullest sense: not merely unseen, but from a world with different generation habits. Every example is therefore simultaneously training material for its own pool and evaluation material for the other four. Nothing is held out, because everything is held out — from four-fifths of the ensemble, at all times.
Only those four out-of-distribution models count as evidence. The in-distribution model's behavior is reported for contrast, not credit. In the prototype results the two tracked each other closely, which is the strongest available sign that the pattern was learned rather than memorized — and the strip ablation, where even the in-distribution model fell silent on its own training patients once the injected rows were removed, closed the case.
The read is consensus, and it is binary
Each model emits a forced continuation — EOS banned, roughly ten rows, because PRISM recalls rather than predicts — and the model fires if the TEST code appears anywhere in that continuation. String presence, nothing subtler. The unit of evidence is then how many independent models fired on the same patient. "N of 5 surfaced this" is a flag, not a probability; individual models are noisy by design, and agreement across pools is the noise filter. Silence is not a verdict either way — it is the absence of a flag, never a clearance.
"Earlier" means FAKE, nothing else
The natural instinct — measure how many rows or months before the actual TEST a model would have fired — is deliberately banned. Earliness is operationalized through exactly one construction: FAKE. A FAKE prompt is a BAD carrier's history cut roughly twenty rows earlier than the BAD prompt — well before the point where the recorded TEST actually occurred (worked example). If a model fires on that prompt, it surfaced the test earlier than the data did. That is the entire claim, and the read stays binary: did the TEST code appear.
The austerity is the point. A graded lead-time metric would import assumptions the data cannot support — synthetic rows are not evenly spaced in clinical meaning, and the downstream decision is binary anyway: suggest the test, or stay silent. Firing on a prompt cut before the event answers exactly the question asked and nothing it can't.
Guardrails: intuitions not to import
Generic ML reflexes are not merely unnecessary here — several actively contradict the design. Applying any of these silently invalidates a result.
| do not import | why it is wrong here |
|---|---|
| train/test splits | there isn't one; every example already tests the four models that never trained on it |
| per-model precision or specificity as a verdict | consensus is the unit of evidence; any single model is expected to be noisy |
| "memorization vs. generalization gap" framing | the scoring models never saw the patient; out-of-distribution firing is the product, not leakage |
| lead-time or earliest-row-index scoring | "earlier" is handled entirely by the binary FAKE read |
| un-banning EOS so the model can "decide" to stop | a model allowed to stop would be read as predicting "no need" — a claim PRISM refuses to make |
| loss-type "best practices" and training-metric gates | the prototype's winning adapter logged inverted training accuracy; gate on generation, never on train metrics |
Retrospective ≡ prospective: evaluating the real-data phase
The same no-hold-out logic extends across time, and this is how the next phase — real claims data — will be evaluated. Truncate a real patient's history at some past date and you recreate, exactly, the prospective task: the model sees everything up to that point and must continue the table, just as it would for a live patient today. The only difference is that history already knows the answer — whether a test was eventually run, when, and what followed.
That equivalence turns every historical patient into a validation case with no waiting. A patient whose condition was caught in 2023 becomes the question: would the ensemble have fired in 2021? The known outcome grades the suggestion, complications and treatment costs included. And once the system operates, yesterday's live patient becomes tomorrow's retrospective case — the loop that feeds continuous retraining.
To be plain about status: the prototype did not run this. Its stand-in was FAKE — cut the history earlier, ask whether the model fires anyway — which is the same idea executed inside a synthetic world. Retrospective truncation on real claims is the design for the phase ahead, not a demonstrated result.
What this frame does and does not establish
The frame is demanding in one direction and silent in another. Demanding: four independent models that never saw a patient — from data generated differently on purpose — must agree before anything is flagged. Silent: it says nothing about single-model reliability, calibrated probabilities, or clinical utility, because it does not try to. The prototype's deliberately clean data makes its rates an upper bound; what carries forward to real data is the frame itself, not the numbers.