mlsanityData to 7 Oct 2026

Guided contrastive self-supervised pre-training for automatic speech recognition

arXiv 2210.12335 ↗First posted 22 Oct 2022cs.CL cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Contrastive Predictive Coding (CPC) is a representation learning method that maximizes the mutual information between intermediate latent representations and the output of a given model. It can be used to effectively initialize the encoder of an Automatic Speech Recognition (ASR) model. We present a novel modification of CPC called Guided Contrastive Predictive Coding (GCPC). Our proposed method maximizes the mutual information between representations from a prior-knowledge model and the output of the model being pre-trained, allowing prior knowledge injection during pre-training. We validate our method on 3 ASR tasks: German, French and English. Our method outperforms CPC pre-training on all three datasets, reducing the Word Error Rate (WER) by 4.44%, 6.55% and 15.43% relative on the German, French and English (Librispeech) tasks respectively, compared to training from scratch, while CPC pre-training only brings 2.96%, 1.01% and 14.39% relative WER reduction respectively.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

19 of 31 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  8. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  9. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  10. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  11. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  12. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
7 more
  1. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  2. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  3. 2016Wav2Letter: an End-to-End ConvNet-based Speech Recognition Systemcited by 43
  4. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  5. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  6. 2019BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognitioncited by 10
  7. 2021UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Datasetcited by 7

Built on by

Not cited yet.