mlsanityData to 14 Oct 2026

FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model

arXiv 2410.03007 ↗First posted 3 Oct 2024eess.AS cs.AI cs.CLcited by 5

On the map: Efficient LLM inference and serving → Audio and omni-modal token compression

Abstract (arXiv)

In this study, we aim to explore Multitask Speech Language Model (SpeechLM) efficient inference via token reduction. Unlike other modalities such as vision or text, speech has unique temporal dependencies, making previous efficient inference works on other modalities not directly applicable. Furthermore, methods for efficient SpeechLM inference on long sequence and sparse signals remain largely unexplored. Then we propose FastAdaSP, a weighted token merging framework specifically designed for various speech-related tasks to improve the trade-off between efficiency and performance. Experimental results on WavLLM and Qwen-Audio show that our method achieves the state-of-the-art (SOTA) efficiency-performance trade-off compared with other baseline methods. Specifically, FastAdaSP achieved 7x memory efficiency and 1.83x decoding throughput without any degradation on tasks like Emotion Recognition (ER) and Spoken Question Answering (SQA). The code will be available at https://github.com/yichen14/FastAdaSP

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 39 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

26 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Accelerating Transducers through Adjacent Token Merging28 Jun

Other references · 25

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2022Token Merging: Your ViT But Fastercited by 1,021
  8. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  10. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  11. 2019Clotho: An Audio Captioning Datasetcited by 358
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
13 more
  1. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  2. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  3. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 115
  4. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 95
  5. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  6. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  7. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  8. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  9. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  10. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  11. 2024MLP Can Be A Good Transformer Learnercited by 17
  12. 2024SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Datacited by 3
  13. 2024Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognitioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compressioncited by 6
  2. 2025FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processingcited by 2
  3. 2026Stride-k Subsampling: Train-Free Audio Token Reduction for Whispercited by 0
  4. 2026Acoustic-to-Text KV Compression for Full-Duplex Speech Modelscited by 0
  5. 2026Audio Token Attention Is Predictable Before the Language Model Runscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
  2. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning
  3. 2026SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models
  4. 2024A Survey on Efficient Inference for Large Language Models
  5. 2025Draft-based Approximate Inference for LLMs
  6. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
  7. 2024Efficient Compression of Multitask Multilingual Speech Models
  8. 2025PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction