CYBER WATCH / EQUINOXE INTELLIGENCE

← All articles

b12285: CUDA – MMA FlashAttention support for head sizes 40 and 72 (#30285)

llama.cpp Releases · 2026-11-11 18:01:13

Added MMA FlashAttention support for head sizes 40 and 72, used in vision towers of Gemma 3/4 and Qwen3‑VL. Previously these sizes were excluded from all tensor‑core dispatch branches and fell back to the tile kernel. The fp16 MMA fragment covers 16 halves in the head direction, so the in‑kernel geometry is padded to the next multiple of 16 (40 → 48, 72 → 80). Register arrays, shared‑memory tile strides and MMA loop bounds use the padded size, while global‑memory loads and stores use the real one. Enabled on NVIDIA Turing and newer; Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and mma_tile_sizes specialisations are required because the explicit instantiations are compiled for every target of a translation unit. Tests added for encoder performance with head size 40. Address review comments on the head size 40 and 72 kernels.

Read original source ↗