llama.cpp CUDA get_rows Optimization Slashes Recurrent State Gather Time by 27%

In the latest llama.cpp update, a major CUDA performance improvement lands for a key operation used in recurrent neural networks. Previously, the k_get_rows_float kernel performed a slow scalar copy, recalculating row metadata for every single element. The new implementation introduces k_get_rows_float_vec, a vectorized path that copies 16 bytes (one int4) per thread when source and destination types match. This path is enabled at compile time for same-type gathers and at runtime when all base pointers are 16-byte aligned, row strides are aligned, and the row element count is a multiple of the vector size. A small but important occupancy gate prevents regression on very small gathers where vectorization would reduce thread count below what the GPU needs for full utilization. On an AMD Strix Halo (gfx1151), the DeltaNet recurrent-state gather (ne00=524288) dropped from 18.6 µs to 13.0 µs, now faster than the Vulkan backend. Overall, total get_rows time is cut by 27%, and all 47 backend operator tests pass.

Website: