Kimi K3 on 8 GB RAM is a systems lesson, not a serving plan

TL;DR: Fareed Khan’s Kimi K3 CPU demo shows that sparse MoE models can be made to fit into tiny RAM budgets, but the trade is brutal latency, disk footprint, and a very narrow practical use case.

Can a 1.56 TB model really run in 8 GB of RAM?

Yes, with a lot of caveats.

Fareed Khan’s r/LocalLLaMA post, “I pushed Kimi K3 onto one CPU with 8 GB of RAM,” is the primary source here, along with his linked repo, kimi-k3-in-c. Khan says he first deployed K3 on 32 H100s at work, then wrote a C99 inference engine so he could poke at it on his own machine.

The headline number is wild: 8.24 GB peak RSS on a dual EPYC 7763 box, with all four GPUs sitting idle. But the important part is not magic compression. It is routing.

Khan reports that 93% of the 1.56 TB checkpoint is routed experts, and only 16 of 896 experts fire per token. His engine does not keep those experts resident in RAM. It reads them from NVMe on demand and multiplies straight from their packed 4-bit form, with no dequantization step. The dense trunk gets repacked into one file, with each layer at a known offset, then streamed layer by layer.

That is the core trick. The model is not “small.” The working set is small, if you are willing to make storage and time pay the bill.

What does this teach about MoE inference?

The useful lesson is that mixture-of-experts models change the shape of the systems problem.

Dense models mostly punish you upfront. If the weights do not fit, you are stuck, or you quantize, shard, offload, or buy bigger hardware. Sparse MoE models have a different failure mode. The total checkpoint can be enormous, while the active compute path per token is much smaller.

Khan’s implementation makes that split very visible. He gives RAM as a dial. At the smallest preset, he reports about 33 seconds per token. With roughly 128 GB of RAM, it improves to about 20 seconds per token, which he says is as fast as it got. He also says output is byte-identical across budgets.

That last detail matters. This is not “sort of K3” running under a memory limit. It is the same graph, with different residency choices. The repo test path builds a 13-layer model with the same tensor graph and checks against a PyTorch reference from committed fixtures, including greedy decode, incremental decoding, KV cache behavior, and carried KDA state.

Still, this is one builder’s report on one machine, not a general benchmark. NVMe behavior, CPU memory bandwidth, packing format, routing patterns, and implementation choices all matter. I would not infer that every giant MoE can now run nicely on a mini PC. That is not what Khan claims either.

Where does the trick stop being useful?

At serving.

Khan is explicit: this is not a practical way to use K3. Half a minute per token is not a product experience. It also needs about 1.7 TB of free disk for the checkpoint plus the packed trunk. There is no BLAS, no framework, no GPU path. The whole thing is six C files, libm and OpenMP, and a 176 KB binary.

That minimalism is the point. The value is educational and diagnostic. If you want to understand what the architecture is actually doing, implementing it badly but transparently can teach more than wrapping a vendor endpoint and watching tokens arrive.

This is also a useful antidote to two common mistakes. One mistake is treating parameter count as the whole story. With MoE, total parameters and active parameters are different operational facts. The other mistake is treating “it runs” as equivalent to “it is usable.” This demo runs. It is not usable for normal inference.

For a builder, I would treat kimi-k3-in-c as a systems microscope. Run the tests before downloading anything massive. Read the packing and streaming choices. Then apply the idea where it fits: debugging model formats, understanding sparse routing, testing offload strategies, or building internal intuition for memory residency. The catch most people miss is that offloading does not remove cost. It moves cost from RAM to disk, latency, and implementation complexity.