mlsanityData to 14 Oct 2026

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

arXiv 2601.06199 ↗First posted 8 Jan 2026eess.AS cs.AI cs.SDcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 52 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

28 of 52 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  10. 2017Mixed Precision Trainingcited by 1,156
  11. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
16 more
  1. 2024Qwen2-Audio Technical Reportcited by 560
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  3. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  4. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  5. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  6. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  7. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  8. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  9. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 50
  10. 2019KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehensioncited by 43
  11. 2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35
  12. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 23
  13. 2024SpeechPrune: Context-aware Token Pruning for Speech Information Retrievalcited by 21
  14. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  15. 2024AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLMcited by 12
  16. 2025Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Modelscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Token Compression for Efficient Multimodal Large Language Models
  2. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
  3. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
  4. 2025Towards Audio Token Compression in Large Audio Language Models
  5. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
  6. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
  7. 2025TOGGLE: Temporal Logic-Guided Large Language Model Compression for Edge
  8. 2025VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction