Hugging Face Transformers v5.16.0 has arrived, introducing several new model architectures alongside a wave of performance and stability improvements. The headline additions include Qwen4-Exp, a hybrid text/multimodal model that combines GatedResidual connections, Qwen Sparse Attention, and Per-Layer Embedding for more efficient long-context inference; GraniteSpeech5, a lightweight ~470M-parameter conformer encoder for fast CTC-based automatic speech recognition; Step-3.7-Flash, a 198B-parameter sparse Mixture-of-Experts vision-language model with dynamic image tiling and multi-token prediction; and CohereCompass, a base architecture for small specialized language and vision-language models.
This release also introduces several breaking changes that users should review before upgrading. The legacy tensor-parallel implementation has been replaced by a DTensor-native backend, and anyone relying on the old TP API must migrate to the new interface. SDPA attention dispatch is now correctly supported for wav2vec2_conformer, wav2vec2-bert, and SeamlessM4T/v2, which may alter initialization behavior. The FuyuProcessor no longer returns the image_patch_indices field, so downstream code that depends on it needs updating. On the infrastructure side, per-layer cache configuration is now possible, allowing models with heterogeneous layers to use distinct sliding window, attention chunk size, and convolution state settings. NVFP4 quantization support has been added via HF kernels, enabling on-the-fly BF16 weight quantization with roughly 50% memory savings. A new naive pipeline parallel inference engine works seamlessly with generate(), and backward compatibility for the tensor-parallel API has been restored with a deprecation cycle for tp_plan in from_pretrained(). The default Flash Attention 2 hub kernel version has been bumped to v3 to resolve compatibility issues with newer PyTorch releases.
A large number of bug fixes and optimizations are included, touching cache handling, generation quality, attention dispatch, and quantized model loading. Specific fixes address an off-by-one error in the sliding window cache, Whisper speculative decoding corruption and position ID issues, broken image generation in Emu3, garbage output from OLMo and GPTNeoX, and generation failures on Qwen2.5-Omni and Qwen3-Omni-MoE when using compilable caches. Attention-related fixes correct a GPT2 cross-attention mask that was silently discarded, add SDPA support declaration to TimmWrapper, and fix Flash attention dispatch on ROCm RDNA GPUs and xcodec2. Quantization fixes restore CLIP weight initialization with quantized checkpoints and correct KV-cache-only quantized model loading. Parallelization improvements include a model parallel fix for the BLT model affecting beam search.
The release also incorporates numerous smaller improvements across the codebase, including video processing fixes for RGBA channel blending and frame caps, tokenizer and pipeline behavior corrections, deterministic evaluation sampling, and extensive CI and documentation updates. Community contributions were significant, with work on CI reliability, support for additional hardware accelerators like MLU and NVIDIA Spark, and model-specific test refinements. As always, the full set of changes is available in the repository, but this release underscores the ongoing focus on long-context efficiency, quantization, and multi-modal support.
