mlsanityData to 7 Oct 2026

Windowed SummaryMixing: An Efficient Fine-Tuning of Self-Supervised Learning Models for Low-resource Speech Recognition

arXiv 2602.09043 ↗First posted 4 Feb 2026eess.AS cs.LG cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has advanced speech processing but suffers from quadratic complexity due to self-attention. To address this, SummaryMixing (SM) has been proposed as a linear-time alternative that summarizes entire utterances using mean pooling but lacks sufficient local context. In this work, we introduce Windowed SummaryMixing (WSM), which enhances SM by integrating local neighborhood summaries alongside the global summary, maintaining efficiency while improving temporal dependencies. Additionally, we introduce a selective fine-tuning approach, replacing self-attention layers in SSL models with WSM blocks and fine-tuning only these blocks in low-resource settings. Our approach improves ASR performance while reducing peak VRAM usage by 40% in the SSL models. WSM blocks have linear-time complexity with enhanced context awareness. Selectively replacing some attention layers reduces compute, memory, and latency, making it ideal for low-resource speech recognition.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

22 of 26 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  6. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  7. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  8. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  9. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  10. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  11. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  12. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
10 more
  1. 2022Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selectioncited by 39
  2. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 38
  3. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  4. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  5. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  6. 2022HyperMixer: An MLP-based Low Cost Alternative to Transformerscited by 13
  7. 2023FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learningcited by 12
  8. 2023SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understandingcited by 11
  9. 2022Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 3
  10. 2023Generative Context-aware Fine-tuning of Self-supervised Speech Modelscited by 3

Built on by

Not cited yet.