CYBER WATCH / EQUINOXE INTELLIGENCE

← All articles

b12191: llama_moe_cache: run missed experts on CPU, add tuning script (#30966)

llama.cpp Releases · 2026-11-07 22:01:10

The update adds CPU support for missed MoE cache experts, enabling a portion of misses to be computed on the CPU via the –moe-cache-cpu-share flag. The graph now builds expert FFNs twice—once on the device from cache slots and once on the CPU from host experts—then sums the results. A new script, moe-cache-tune.py, is introduced to test various cache sizes and CPU shares, fit upload and CPU speeds, and recommend –moe-cache-mib and –moe-cache-cpu-share. Other changes include removing backend logic for the CPU part from the graph, optimizing asynchronous transfer, updating cache handling in llama_context::process_ubatch, and adding backend sync for the transfer queue. The project also assigns a new code owner (am17an) for the MoE cache.

Read original source ↗