# Joint Masked CPC and CTC Training for ASR

arXiv: https://arxiv.org/abs/2011.00093 · HTML: https://mlsanity.com/paper/2011.00093/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Joint Unsupervised and Supervised Training for Multilingual ASR](https://mlsanity.com/paper/2111.08137/) (2021)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [JOIST: A Joint Speech and Text Streaming Model For ASR](https://mlsanity.com/paper/2210.07353/) (2022)
- [Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model](https://mlsanity.com/paper/2112.07254/) (2021)
- [Layer Pruning on Demand with Intermediate CTC](https://mlsanity.com/paper/2106.09216/) (2021)
- [Continuous Pseudo-Labeling from the Start](https://mlsanity.com/paper/2210.08711/) (2022)
- [Boosting Cross-Domain Speech Recognition with Self-Supervision](https://mlsanity.com/paper/2206.09783/) (2022)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Label Aware Speech Representation Learning For Language Identification](https://mlsanity.com/paper/2306.04374/) (2023)
- [Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Mask](https://mlsanity.com/paper/2110.04056/) (2021)
- [Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource Historical Document Transcription](https://mlsanity.com/paper/2112.08692/) (2021)
- [Word Order Does Not Matter For Speech Recognition](https://mlsanity.com/paper/2110.05994/) (2021)
- [Predicting non-native speech perception using the Perceptual Assimilation Model and state-of-the-art acoustic models](https://mlsanity.com/paper/2205.15823/) (2022)
- [SCaLa: Supervised Contrastive Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04187/) (2021)
- [Pushing the performances of ASR models on English and Spanish accents](https://mlsanity.com/paper/2212.12048/) (2022)
- [Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition](https://mlsanity.com/paper/2403.19822/) (2024)
- [MASR: Multi-label Aware Speech Representation](https://mlsanity.com/paper/2307.10982/) (2023)
- [Objective Soups: Multilingual Multi-Task Modeling for Speech Processing](https://mlsanity.com/paper/2508.09228/) (2025)

## Abstract (arXiv)

Self-supervised learning (SSL) has shown promise in learning representations of audio that are useful for automatic speech recognition (ASR). But, training SSL models like wav2vec~2.0 requires a two-stage pipeline. In this paper we demonstrate a single-stage training of ASR models that can utilize both unlabeled and labeled data. During training, we alternately minimize two losses: an unsupervised masked Contrastive Predictive Coding (CPC) loss and the supervised audio-to-text alignment loss Connectionist Temporal Classification (CTC). We show that this joint training method directly optimizes performance for the downstream ASR task using unsupervised data while achieving similar word error rates to wav2vec~2.0 on the Librispeech 100-hour dataset. Finally, we postulate that solving the contrastive task is a regularization for the supervised CTC loss.