mlsanityData to 7 Oct 2026

Token Merging for Multilingual Speech Recognition: A Systematic Study Across Model Scale and Fine-Tuning

arXiv 2609.13151 ↗First posted 7 Jul 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Leading multilingual speech recognition models like Whisper transcribe diverse, low-resource languages without language-specific training but are computationally expensive to deploy. Token merging mitigates this inefficiency by dynamically combining redundant features, shortening the sequence length during inference without requiring retraining. In this paper, we systematically evaluate token merging on the Whisper model family across sixteen diverse languages and three different model sizes. We also test how token merging interacts with fine-tuning (DoRA) on low-resource languages. Our findings show that merging tokens increases computational efficiency with almost no loss in transcription accuracy across most low-resource languages and model sizes, and it works even after the model has been fine-tuned. Our results demonstrate that token merging is a highly practical method for making multilingual speech recognition faster and cheaper to deploy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 22 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

17 of 22 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2022Token Merging: Your ViT But Fastercited by 995
  5. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  6. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  7. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  8. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  9. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  10. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  11. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
  12. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
5 more
  1. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  2. 2023Accelerating Transducers through Adjacent Token Mergingcited by 14
  3. 2025BaldWhisper: Faster Whisper with Head Shearing and Layer Mergingcited by 4
  4. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Modelscited by 4
  5. 2025Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CLiFT-ASR: A Cross-Lingual Fine-Tuning Framework for Low-Resource Taiwanese Hokkien Speech Recognition
  2. 2025MLMA: Towards Multilingual ASR With Mamba-based Architectures
  3. 2026Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
  4. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
  5. 2025Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semantically
  6. 2025The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
  7. 2023Massively Multilingual Shallow Fusion with Large Language Models
  8. 2024LUPET: Incorporating Hierarchical Information Path into Multilingual ASR