CYBER WATCH / EQUINOXE INTELLIGENCE

← All articles

CUDA: few-row MMQ for BS 2‑24

llama.cpp Releases · 2026-11-13 06:01:11

A new int8 MMA kernel has been added for quantized matrix multiplications with 2‑24 source‑1 columns, enabling speculative decoding. It supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL, and IQ4_XS on NVIDIA GPUs from Ampere onward. Bug fixes include a CUDA guard for few‑row MMQ, weight flag adjustments in test_mul_mat, DS4 sum handling for quantized values, clearer HIP/MUSA abort handling, and dispatch order improvements for MMVQ and few‑row MMQ.

Read original source ↗