Inkling Model Integration Fixes
This patch addresses several hiccups that surfaced after integrating the Inkling model. One fix resolves a problem with scaled dot-product attention (SDPA) during prefill when a position bias is used and no padding is applied with the StaticCache. Another important correction ensures that assisted generation works properly for models that rely on an EncoderDecoder cache, particularly affecting the OlmoHybrid architecture. These changes smooth out the experience for anyone running the Inkling model in a self-hosted environment, preventing silent failures and improving reliability during text generation.
Performance and Multi-Device Speedups
On the optimization front, the FP8 kernels version has been bumped, which can lead to better performance for GPU-accelerated inference when using FP8 quantization. Additionally, a fix for DeepGEMM now makes it run correctly across multiple devices, removing errors that previously cropped up in multi-GPU setups. For self-hosted users who scale up inference, this means a stabler and potentially faster pipeline when employing these advanced features.
