PRISM

Appearance

Hardware Notes ​

Status: demonstrated — measured on the prototype fleet; sizing input for the production vision.

The prototype's most useful infrastructure result is that one 16 GB consumer graphics card runs a complete inference node — full model, full ~262k-token context — and it produced measured throughput numbers across three GPU classes that turn the production cluster design from a sketch into arithmetic.

Why one consumer card is a whole node ​

Long-context inference is normally rationed by the KV cache: the per-token attention memory that grows with context length and pushes serious context windows onto datacenter hardware. The prototype's base model sidesteps this by architecture. It is a hybrid design in which only 8 layers use full attention; the remaining layers use a linear-attention mechanism whose memory cost does not grow with context at all. The KV cache therefore stays small no matter how long the patient history gets, and the whole serving footprint — 8-bit model weights plus the cache for the full ~262k-token context — runs in about 15 GB. A reduced ~126k-token vocabulary (pruned from the original 248k) trims the footprint further.

The consequence is worth stating plainly: a 16 GB consumer card serving the full context is a complete production node, not a budget compromise. Nothing about the production design — one model per card, racks of commodity GPUs — depends on hardware that has to be specially procured, and a failed node is replaced with a retail part.

Measured throughput ​

Throughput was measured by replaying an identical batch of prototype examples on each card under the fleet's real serving configuration: full ~262k context enabled, quantized KV cache, forced continuation with EOS banned, several concurrent request slots. Examples per hour is the metric that matters, because screening is a continuous batch workload, not an interactive one.

GPU classexamples/hrexamples/dayrelative
prototype node — previous-generation 16 GB midrange~200~4,8001×
current-generation 16 GB class~500~12,000~2.5×
current-generation 32 GB flagship, 4 concurrent streams~670~16,100~3.3×
current-generation 32 GB flagship, 8 concurrent streams~900~21,700~4.5×

The five prototype fleet nodes ran the first row's card class; the entire 16,250-run evaluation grid behind the prototype results was produced on that hardware.

Concurrency has diminishing returns ​

Doubling the flagship's concurrent streams from 4 to 8 gained about 34%, not the 2× naive scaling would suggest — and VRAM use never passed roughly 20 of its 32 GB. Generation on this workload is compute-bound: once a card's compute is saturated, additional streams mostly queue behind each other. Two things follow. Extra VRAM buys little here, which is why the 16 GB class keeps pace so well against the flagship. And per-card throughput has a ceiling, so capacity grows by adding cards rather than by buying bigger ones — exactly the shape the production design assumes.

What this sizes ​

These numbers are the sizing input for the cluster architecture, which remains vision. As explicitly back-of-envelope arithmetic: screening a million-member population once a month means each ensemble model works through on the order of 33,000 histories a day — roughly three current-generation 16 GB cards per model at the measured rate. Cadence, population, and ensemble width are all vision-stage targets rather than commitments; the point is that the arithmetic now rests on measurement instead of estimate, and it scales linearly by replication.

What these numbers do not say ​

They were measured on synthetic prototype histories, which ran shorter than intended and shorter than real claims histories may be; longer prompts shift the arithmetic. They are specific to this base model, quantization, and context discipline — change any of those and remeasure. And throughput says nothing about validity: what the ensemble's output is worth is a results question, not a hardware one.

See also ​