The latest Transformers release, v5.17.0, introduces a wave of new model architectures spanning language, speech, and audio, alongside numerous performance and stability improvements. Among the highlights is HY4-Preview, a 780-billion-parameter mixture-of-experts language model that activates 49 billion parameters per token, with each MoE layer containing 256 routed experts and a shared expert, routing tokens to 8 of them. Its 1-million-token context window and novel features—Multi-head Latent Attention (MLA), DeepSeek Sparse Attention (DSA), gated MLA with learnable attention sinks, and Independent Hyper-Connections—make it a standout for long-context tasks. The release also adds VibeVoice, a next-token diffusion framework within an LLM for synthesizing high-fidelity, long-form multi-speaker audio, ideal for podcasts and audiobooks. NeoMME, a family of efficient multimodal-native foundation encoders from H Company, processes text and image patches in a single bidirectional Transformer without separate vision or language towers, and includes a retriever variant for visual document search. For speech recognition, Fun-ASR-Nano from Alibaba DAMO Academy’s FunAudioLLM team delivers state-of-the-art ASR in Chinese, English, and Japanese with only 800M parameters, supporting dialects, hotword customization, and native punctuation. Kimi Linear from Moonshot AI brings a hybrid linear attention architecture built around Kimi Delta Attention (KDA), which refines Gated DeltaNet with per-channel forget gates, combined with DeepSeek-V3-style MLA and MoE feed-forward blocks. Canary-1B-v2, a fast multilingual ASR and speech-to-text translation model from NVIDIA, reuses the Fast Conformer encoder from Parakeet and pairs it with a Transformer decoder using sinusoidal positional embeddings and cross-attention. Finally, NeuCodec, a neural audio codec extending XCodec2, achieves a 0.8 kbps bitrate at 24 kHz output using Finite Scalar Quantization with a single codebook, trained on CC data without non-commercial restrictions.
Key Improvements and Fixes
This release standardizes vision rotary embeddings (2D/3D) into a unified RoPE frequency computation module, a breaking change for custom vision models that previously relied on attention-layer-level or model-specific RoPE grid interleaving logic—migration to the new centralized implementation is required. On the generation front, the team removed an unnecessary accelerator synchronization on every decode step, reducing per-step overhead, and prevented the unconditional downloading of remote hub files during generation. Correctness fixes were applied to auto-compile cache checks for encoder-decoder models, past_key_values naming in AfMoE, and flaky export tests.
Cache handling received several bug fixes, including a quantized cache issue in VibeVoice, rejection of non-static cache implementations in VoxtralRealtime, and a clearer error when paged attention is called without a cache. Kernel support was improved with fixes for nested FLA kernel imports when only fla-core is installed, a warning when hub-kernel functions fall back to slower PyTorch reference implementations, and the ability to register standalone functions like RoPE in KernelConfig with optional non-inheritance of defaults. Quantization fixes address incorrect FP8 embedding handling in Qwen models, missing FP8 tensor parallelism layer overrides, and unnecessary MXFP4 dequantization on XPU devices. Across the library, numerous bug fixes and improvements cover models like HunYuanVL, Flava, YOLOS, Pix2Struct, Inkling, Mamba2, Gemma3/Gemma4 sliding-window masks, sparse TikToken tokenizers, and many integration test updates. Self-hosted users will benefit from better stability, faster generation, and broader hardware support, including XPU and CPU optimizations, gradient checkpointing offload, and memory-saving MTP hidden state capture.
Community Contributions
The v5.17.0 release was significantly shaped by the community, with many contributors providing model implementations, test fixes, documentation updates, and new utilities. A new Batch Rebalance Data Sampler improves training data distribution, DEIMv2 attribution was added, and support for GLM 5.3 Flash was contributed. Extensive work went into stabilizing continuous batching, fixing device scoping and bootstrap behavior, and improving test infrastructure for AMD MI300 and other accelerators. Documentation was enhanced with per-layer configuration, partial checkpointing, kernel customization, and a LiteRT page under community integrations. These contributions ensure that the library remains robust and accessible for self-hosted deployments across a wide range of hardware and use cases.
