mlsanityData to 14 Oct 2026

Linear-Complexity Self-Supervised Learning for Speech Processing

arXiv 2407.13377 ↗First posted 18 Jul 2024cs.CL cs.AI eess.AScited by 2

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) models usually require weeks of pre-training with dozens of high-end GPUs. These models typically have a multi-headed self-attention (MHSA) context encoder. However, MHSA takes quadratic time and space in the input length, contributing to the high pre-training cost. Linear-complexity alternatives to MHSA have been proposed. For instance, in supervised training, the SummaryMixing model is the first to outperform MHSA across multiple speech processing tasks. However, these cheaper alternatives have not been explored for SSL yet. This paper studies a linear-complexity context encoder for SSL for the first time. With better or equivalent performance for the downstream tasks of the MP3S benchmark, SummaryMixing reduces the pre-training time and peak VRAM of wav2vec 2.0 model by 18% and by 23%, respectively, leading to the pre-training of a 155M wav2vec 2.0 model finished within one week with 4 Tesla A100 GPUs. Code is available at https://github.com/SamsungLabs/SummaryMixing.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

34 of 43 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  11. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
22 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  5. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  6. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  7. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  8. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  9. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 124
  10. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  11. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  12. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  13. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  14. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
  15. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  16. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  17. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  18. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  19. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  20. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  21. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 18
  22. 2023SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understandingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Linear Time Complexity Conformers with SummaryMixing for Streaming Speech Recognitioncited by 1
  2. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0