llama.cpp Just Made 125B MoE Models 4× Faster!

llama.cpp's new MoE expert cache took a 125B model from 9.7 to 37.2 tokens a second on one 24 GB GPU. Here's where that 4× really comes from, and which cards get slower. The 4× is from lich-77's experimental expert-cache branch (llama.cpp issue #29949), measured on an RX 7900 XTX running Qwen3.8-Flash-Next. Its baseline was -ncmoe 48, which keeps every one of the model's 24,576 experts in system RAM, so the card started out holding none of them. The cache that actually shipped (PR #29887, release 0.7.0, flag --moe-cache-mib) posts 2.2× on an RTX 5090 against llama.cpp's default --fit. On the