mlsanityData to 7 Oct 2026

On Compressing Sequences for Self-Supervised Speech Models

arXiv 2210.07189 ↗First posted 13 Oct 2022cs.CL cs.SD eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Compressing self-supervised models has become increasingly necessary, as self-supervised models become larger. While previous approaches have primarily focused on compressing the model size, shortening sequences is also effective in reducing the computational cost. In this work, we study fixed-length and variable-length subsampling along the time axis in self-supervised learning. We explore how individual downstream tasks are sensitive to input frame rates. Subsampling while training self-supervised models not only improves the overall performance on downstream tasks under certain frame rates, but also brings significant speed-up in inference. Variable-length subsampling performs particularly well under low frame rates. In addition, if we have access to phonetic boundaries, we find no degradation in performance for an average frame rate as low as 10 Hz.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 35 of its 41 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

35 of 41 references.

References · 35

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  10. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2020Efficient Transformers: A Surveycited by 784
23 more
  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  3. 2015Listen, Attend and Spellcited by 542
  4. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  5. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  6. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  7. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  8. 2021Unsupervised Speech Recognitioncited by 147
  9. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  10. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  11. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
  12. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  13. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  14. 2022Towards End-to-end Unsupervised Speech Recognitioncited by 44
  15. 2017Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LMcited by 43
  16. 2016Very Deep Convolutional Networks for End-to-End Speech Recognitioncited by 34
  17. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  18. 2015Deep convolutional acoustic word embeddings using word-pair side informationcited by 27
  19. 2020Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networkscited by 24
  20. 2021Variable-rate discrete representation learningcited by 24
  21. 2019Audio-Linguistic Embeddings for Spoken Sentencescited by 15
  22. 2021Aligned Contrastive Predictive Codingcited by 15
  23. 2022On-demand compute reduction with stochastic wav2vec 2.0cited by 10

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 2; 2024: 1.

7 papers.

  1. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  2. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 15
  3. 2023Accelerating Transducers through Adjacent Token Mergingcited by 14
  4. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 10
  5. 2022Introducing Semantics into Speech Encoderscited by 3
  6. 2022Once-for-All Sequence Compression for Self-Supervised Speech Modelscited by 2
  7. 2024SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image datacited by 2