mlsanityData to 7 Oct 2026

Wav2Vec-Aug: Improved self-supervised training with limited data

arXiv 2206.13654 ↗First posted 27 Jun 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) of speech representations has received much attention over the last few years but most work has focused on languages and domains with an abundance of unlabeled data. However, for many languages there is a shortage even in the unlabeled data which limits the effectiveness of SSL. In this work, we focus on the problem of applying SSL to domains with limited available data by leveraging data augmentation for Wav2Vec 2.0 pretraining. Further, we propose improvements to each component of the model which result in a combined relative word error rate (WER) improvement of up to 13% compared to Wav2Vec 2.0 on Librispeech test-clean / other.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 24 of its 29 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

24 of 29 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
12 more
  1. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  2. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  3. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  4. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  5. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  6. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  7. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  8. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  9. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  10. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  11. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  12. 2018wav2letter++: The Fastest Open-source Speech Recognition Systemcited by 42

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 6.

9 papers.

  1. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentationcited by 30
  2. 2023Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretrainingcited by 18
  3. 2022CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning of speech representationscited by 11
  4. 2022Efficient Utilization of Large Pre-Trained Models for Low Resource ASRcited by 4
  5. 2022data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setupcited by 4
  6. 2023Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Modelscited by 3
  7. 2023Unsupervised Pre-Training for Vietnamese Automatic Speech Recognition in the HYKIST Projectcited by 2
  8. 2023Automatic Data Augmentation for Domain Adapted Fine-Tuning of Self-Supervised Speech Representationscited by 0
  9. 2023Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language augmentation for Low Resource Self-Supervised Speech Modelscited by 0