# Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction

arXiv: https://arxiv.org/abs/2110.15430 · HTML: https://mlsanity.com/paper/2110.15430/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 20 of 20)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Learning Robust and Multilingual Speech Representations](https://mlsanity.com/paper/2001.11128/) (2020)
- [Espresso: A Fast End-to-end Neural Speech Recognition Toolkit](https://mlsanity.com/paper/1909.08723/) (2019)
- [Wav2vec-C: A Self-supervised Model for Speech Representation Learning](https://mlsanity.com/paper/2103.08393/) (2021)
- [The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans](https://mlsanity.com/paper/2012.13006/) (2020)
- [Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition](https://mlsanity.com/paper/1711.08016/) (2017)
- [BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition](https://mlsanity.com/paper/1907.00457/) (2019)
- [Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement](https://mlsanity.com/paper/2106.02896/) (2021)
- [Problem-Agnostic Speech Embeddings for Multi-Speaker Text-to-Speech with SampleRNN](https://mlsanity.com/paper/1906.00733/) (2019)
- [Exploring End-to-End Multi-channel ASR with Bias Information for Meeting Transcription](https://mlsanity.com/paper/2011.03110/) (2020)

## Built on by (most cited first, 13 of 13)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation](https://mlsanity.com/paper/2203.16104/) (2022)
- [Ensemble knowledge distillation of self-supervised speech models](https://mlsanity.com/paper/2302.12757/) (2023)
- [Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR](https://mlsanity.com/paper/2304.04974/) (2023)
- [Improving generalizability of distilled self-supervised speech processing models under distorted settings](https://mlsanity.com/paper/2210.07978/) (2022)
- [A Conformer Based Acoustic Model for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2203.00725/) (2022)
- [CORI: CJKV Benchmark with Romanization Integration -- A step towards Cross-lingual Transfer Beyond Textual Scripts](https://mlsanity.com/paper/2404.12618/) (2024)
- [HuBERT-VIC: Improving Noise-Robust Automatic Speech Recognition of Speech Foundation Model via Variance-Invariance-Covariance Regularization](https://mlsanity.com/paper/2508.12292/) (2025)
- [Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models](https://mlsanity.com/paper/2605.02715/) (2026)
- [Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2112.12522/) (2021)
- [Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement](https://mlsanity.com/paper/2211.06562/) (2022)
- [Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language augmentation for Low Resource Self-Supervised Speech Models](https://mlsanity.com/paper/2309.12763/) (2023)
- [Noise robust distillation of self-supervised speech models via correlation metrics](https://mlsanity.com/paper/2312.12153/) (2023)

## Abstract (arXiv)

Noise robustness is essential for deploying automatic speech recognition (ASR) systems in real-world environments. One way to reduce the effect of noise interference is to employ a preprocessing module that conducts speech enhancement, and then feed the enhanced speech to an ASR backend. In this work, instead of suppressing background noise with a conventional cascaded pipeline, we employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We propose to combine a reconstruction module with contrastive learning and perform multi-task continual pre-training on noisy data. The reconstruction module is used for auxiliary learning to improve the noise robustness of the learned representation and thus is not required during inference. Experiments demonstrate the effectiveness of our proposed method. Our model substantially reduces the word error rate (WER) for the synthesized noisy LibriSpeech test sets, and yields around 4.1/7.5% WER reduction on noisy clean/other test sets compared to data augmentation. For the real-world noisy speech from the CHiME-4 challenge (1-channel track), we have obtained the state of the art ASR performance without any denoising front-end. Moreover, we achieve comparable performance to the best supervised approach reported with only 16% of labeled data.