Keeping the model on disk
A preview release published on Hugging Face this week takes a 35-billion-parameter sparse mixture-of-experts model and runs it in under 3 GiB of active memory by leaving the weights on storage and streaming experts in as they are needed.
Edge0-35B-A3B-preview ships as a 4-bit quantised checkpoint with LoRA and prerouter adapters, under the Apache 2.0 licence. The published figures were measured on a Mac mini M4 Pro with 24 GB of memory: 14.9 to 17.7 tokens per second of decode, 113 to 140 tokens per second of prefill, and peak active memory of 2.9 GiB. There is no sharding and no upfront load of the full weights into RAM.
Why a mixture of experts makes this possible
A sparse MoE model activates only a small subset of its parameters for any given token — in this case about 3 billion of the 35 billion, which is what the A3B in the name refers to. Conventional inference still holds all of them in memory, because you do not know which experts a token will route to until you get there.
That is the problem the prerouter solves. It predicts the routing one step ahead, so the required experts can be fetched from storage while the current token is still being computed. The model card puts the improvement in decode throughput from that prediction at up to 59%.
Quantisation without the usual cost
Running at 4 bits normally costs accuracy. Edge0 uses what it calls Recover-LoRA distillation to claw some of it back, and reports an average gap of 3.9 percentage points against the fp16 base model across benchmarks including AIME, HumanEval and MMLU-Pro.
That number is the developer’s own, published on its own model card, and has not been independently verified. Nearly four points is not nothing — it is the difference between two model generations on some of these tests — but it is a very different proposition from the alternative, which on a 24 GB machine is not running a 35B model at all.
What it changes
The practical claim is that the memory ceiling on local inference is softer than it looks. If routing can be predicted far enough ahead and storage is fast enough, the constraint moves from how much RAM a machine has to how fast its SSD is — and consumer SSDs have been getting faster more cheaply than consumer RAM has been getting larger.
This is a preview, on Apple silicon, with vendor-reported quality numbers. The things to watch are whether the approach holds on commodity x86 hardware with slower storage, and whether independent evaluations reproduce the 3.9-point gap.
