mlsanityData to 7 Oct 2026

Towards Audio Token Compression in Large Audio Language Models

arXiv 2511.20973 ↗First posted 26 Nov 2025eess.AS cs.AI cs.CLcited by 4

On the map: Efficient LLM inference and serving → Audio and omni-modal token compression

Abstract (arXiv)

Large Audio Language Models (LALMs) deliver strong performance across speech and audio tasks, but their audio encoders generate high-rate token sequences (e.g., 25 tokens/s), making attention computation costly and limiting scalability. In this paper, we explore techniques such as unsupervised segmentation, uniform average pooling, etc., to reduce the number of audio tokens before they are consumed by the LLM decoder. To mitigate potential performance degradation, we employ low-rank adapters during finetuning. We evaluate our proposed models on two tasks, automatic speech recognition and speech-to-speech translation tasks, that are dependent on effectively uncovering the underlying lexical content of the input signal, and study the effect of downsampling on these tasks. Experimental results show that compressed LALMs can achieve performance closer to frame-level LALMs while reducing the input audio token count up to three times before the LLM backbone.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 37 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

32 of 37 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2025Qwen2.5-Omni Technical Reportcited by 704
  11. 2021AST: Audio Spectrogram Transformercited by 634
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
20 more
  1. 2024Qwen2-Audio Technical Reportcited by 543
  2. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  3. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  4. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  5. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  6. 2025Kimi-Audio Technical Reportcited by 210
  7. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 210
  8. 2023Listen, Think, and Understandcited by 206
  9. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  10. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  11. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  12. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  13. 2023Joint Audio and Speech Understandingcited by 99
  14. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  15. 2025Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?cited by 42
  16. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  17. 2017An embedded segmental K-means model for unsupervised segmentation and clustering of speechcited by 31
  18. 2021Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metriccited by 21
  19. 2021Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and wordscited by 18
  20. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Locality Matters for Training-Free Audio Token Compression in Audio-Language Modelscited by 2
  2. 2026VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modelingcited by 1
  3. 2026HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Modelscited by 1
  4. 2026Audio Token Attention Is Predictable Before the Language Model Runscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
  2. 2025Segmentwise Pruning in Audio-Language Models
  3. 2026TRADE: Transducer-Augmented Decoder for Speech LLM
  4. 2024Residual vector quantization for KV cache compression in large language model
  5. 2026FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
  6. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
  7. 2025A Survey of Token Compression for Efficient Multimodal Large Language Models
  8. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs