mlsanityData to 7 Oct 2026

An Analysis of Linear Complexity Attention Substitutes with BEST-RQ

arXiv 2409.02596 ↗First posted 4 Sep 2024cs.LG cs.CL cs.SDcited by 2

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) has proven to be effective in various domains, including speech processing. However, SSL is computationally and memory expensive. This is in part due the quadratic complexity of multi-head self-attention (MHSA). Alternatives for MHSA have been proposed and used in the speech domain, but have yet to be investigated properly in an SSL setting. In this work, we study the effects of replacing MHSA with recent state-of-the-art alternatives that have linear complexity, namely, HyperMixing, Fastformer, SummaryMixing, and Mamba. We evaluate these methods by looking at the speed, the amount of VRAM consumed, and the performance on the SSL MP3S benchmark. Results show that these linear alternatives maintain competitive performance compared to MHSA while, on average, decreasing VRAM consumption by around 20% to 60% and increasing speed from 7% to 65% for input sequences ranging from 20 to 80 seconds.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

30 of 35 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  11. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  12. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
18 more
  1. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  2. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  3. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  4. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  6. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  7. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  8. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  9. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  10. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  11. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
  12. 2024Mamba in Speech: Towards an Alternative to Self-Attentioncited by 58
  13. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  14. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  15. 2022HyperMixer: An MLP-based Low Cost Alternative to Transformerscited by 13
  16. 2023SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understandingcited by 11
  17. 2024Open Implementation and Study of BEST-RQ for Speech Processingcited by 9
  18. 2023HyperConformer: Multi-head HyperMixer for Efficient Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Towards Early Prediction of Self-Supervised Speech Model Performancecited by 5
  2. 2026Polynomial Mixing for Efficient Self-supervised Speech Encoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
  2. 2022Avoid Overthinking in Self-Supervised Models for Speech Recognition
  3. 2025An Exploration of Mamba for Speech Self-Supervised Models
  4. 2026Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion
  5. 2022Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation
  6. 2025Multi-head Temporal Latent Attention
  7. 2025Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention
  8. 2024SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model