llama.cpp CUDA get_rows Optimization Slashes Recurrent State Gather Time by 27%
In the latest llama.cpp update, a major CUDA performance improvement lands for a key operation used in recurrent neural networks. Previously, the k_get_rows_float kernel performed a slow scalar copy, recalculating row metadata for every single element. The new implementation introduces k_get_rows_float_vec, a vectorized path that copies 16 bytes (one int4) per thread when source and […]
llama.cpp CUDA get_rows Optimization Slashes Recurrent State Gather Time by 27% Read More »
