CYBER WATCH / EQUINOXE INTELLIGENCE
← All articles
CUDA: few-row MMQ for BS 2‑24
llama.cpp Releases · 2026-11-13 06:01:11
A new int8 MMA kernel has been added for quantized matrix multiplications with 2‑24 source‑1 columns, enabling speculative decoding. It supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL, and IQ4_XS on NVIDIA GPUs from Ampere onward. Bug fixes include a CUDA guard for few‑row MMQ, weight flag adjustments in test_mul_mat, DS4 sum handling for quantized values, clearer HIP/MUSA abort handling, and dispatch order improvements for MMVQ and few‑row MMQ.Read original source ↗