CYBER WATCH / EQUINOXE INTELLIGENCE

← All articles

b12180: model: add Kimi‑K3.1‑Pro text model (#31779)

llama.cpp Releases · 2026-11-07 02:01:50

This release adds the Kimi‑K3.1‑Pro text model, built on K3 but replacing full‑attention MLA layers with DSA. Twenty‑four of the 93 layers use a “lightning” indexer (32 heads × 128, top_k = 2048, no rope). The rest of the K3 skeleton (cross‑layer residuals, latent MoE, etc.) remains unchanged, allowing reuse of the existing graph and converter. The indexer applies an additive mask over the entire cache, reusing the sparse flash‑attention path, and introduces four indexer tensors per DSA layer along with a hybrid memory that keeps a K‑only MLA cache, recurrent state for KDA layers, and an indexer key cache. Conversion uses KimiK31Model and KimiK31ForConditionalGeneration, with the indexer configuration exposed via attention.indexer.{head_count,key_length,top_k,types}. Indexer parameters (q_b, k, proj) are not overridden by quantization to preserve top‑k precision. The model was verified against the HF checkpoint: 38 tensors, indexer metadata, and top‑k selection match, yielding identical perplexity beyond the context length. All architecture tests pass.

Read original source ↗