mlsanityData to 7 Oct 2026

Wav2vec-C: A Self-supervised Model for Speech Representation Learning

arXiv 2103.08393 ↗First posted 9 Mar 2021eess.AS cs.LG cs.SDcited by 25

Not on the map. Why?

Abstract (arXiv)

Wav2vec-C introduces a novel representation learning technique combining elements from wav2vec 2.0 and VQ-VAE. Our model learns to reproduce quantized representations from partially masked speech encoding using a contrastive loss in a way similar to Wav2vec 2.0. However, the quantization process is regularized by an additional consistency network that learns to reconstruct the input features to the wav2vec 2.0 network from the quantized representations in a way similar to a VQ-VAE model. The proposed self-supervised model is trained on 10k hours of unlabeled data and subsequently used as the speech encoder in a RNN-T ASR model and fine-tuned with 1k hours of labeled data. This work is one of only a few studies of self-supervised learning on speech tasks with a large volume of real far-field labeled data. The Wav2vec-C encoded representations achieves, on average, twice the error reduction over baseline and a higher codebook utilization in comparison to wav2vec 2.0

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

18 of 23 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  12. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
6 more
  1. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  2. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  3. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  4. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  5. 2020Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognitioncited by 34
  6. 2019DiPCo -- Dinner Party Corpuscited by 18

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 7; 2023: 4; 2024: 4; 2025: 2.

25 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  3. 2022Audio Self-supervised Learning: A Surveycited by 65
  4. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  5. 2021Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognitioncited by 15
  6. 2021Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstructioncited by 12
  7. 2023Learning Speech Representation From Contrastive Token-Acoustic Pretrainingcited by 9
  8. 2025Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languagescited by 8
  9. 2021Pretext Tasks selection for multitask self-supervised speech representation learningcited by 7
  10. 2022Distilling a Pretrained Language Model to a Multilingual ASR Modelcited by 7
  11. 2024VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processingcited by 7
  12. 2024The Effect of Batch Size on Contrastive Self-Supervised Speech Representation Learningcited by 5
13 more
  1. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 4
  2. 2023ST-KeyS: Self-Supervised Transformer for Keyword Spotting in Historical Handwritten Documentscited by 3
  3. 2021Word Order Does Not Matter For Speech Recognitioncited by 2
  4. 2021Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognitioncited by 1
  5. 2022Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approachcited by 1
  6. 2022TriNet: stabilizing self-supervised learning from complete or slow collapse on ASRcited by 1
  7. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  8. 2025HuBERT-VIC: Improving Noise-Robust Automatic Speech Recognition of Speech Foundation Model via Variance-Invariance-Covariance Regularizationcited by 1
  9. 2021Label-Synchronous Speech-to-Text Alignment for ASR Using Forward and Backward Transformerscited by 0
  10. 2021Learning source-aware representations of music in a discrete latent spacecited by 0
  11. 2022Deploying self-supervised learning in the wild for hybrid automatic speech recognitioncited by 0
  12. 2022Analyzing Robustness of End-to-End Neural Models for Automatic Speech Recognitioncited by 0
  13. 2023MASR: Multi-label Aware Speech Representationcited by 0