# HuBERT-VIC: Improving Noise-Robust Automatic Speech Recognition of Speech Foundation Model via Variance-Invariance-Covariance Regularization

arXiv: https://arxiv.org/abs/2508.12292 · HTML: https://mlsanity.com/paper/2508.12292/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 17 Aug 2025.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization](https://mlsanity.com/paper/2012.06659/) (2020)
- [Robust Self-Supervised Audio-Visual Speech Recognition](https://mlsanity.com/paper/2201.01763/) (2022)
- [Wav2vec-C: A Self-supervised Model for Speech Representation Learning](https://mlsanity.com/paper/2103.08393/) (2021)
- [A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2201.08930/) (2022)
- [Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning](https://mlsanity.com/paper/2210.15324/) (2022)
- [Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition](https://mlsanity.com/paper/2110.04934/) (2021)
- [Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction](https://mlsanity.com/paper/2110.15430/) (2021)
- [Joint Training of Speech Enhancement and Self-supervised Model for Noise-robust ASR](https://mlsanity.com/paper/2205.13293/) (2022)
- [deHuBERT: Disentangling Noise in a Self-supervised Model for Robust Speech Recognition](https://mlsanity.com/paper/2302.14597/) (2023)
- [Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2407.03563/) (2024)
- [I2CR: Improving Noise Robustness on Keyword Spotting Using Inter-Intra Contrastive Regularization](https://mlsanity.com/paper/2209.06360/) (2022)
- [STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models](https://mlsanity.com/paper/2312.09040/) (2023)

## Built on by (most cited first, 2 of 2)

- [StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs](https://mlsanity.com/paper/2509.22220/) (2025)
- [DASH: Dual-View Self-Distillation with Multi-Layer Hidden Representations for Robust Speech Recognition](https://mlsanity.com/paper/2606.19203/) (2026)

## Abstract (arXiv)

Noise robustness in speech foundation models (SFMs) has been a critical challenge, as most models are primarily trained on clean data and experience performance degradation when the models are exposed to noisy speech. To address this issue, we propose HuBERT-VIC, a noise-robust SFM with variance, in-variance, and covariance regularization (VICReg) objectives. These objectives adjust the statistics of noisy speech representations, enabling the model to capture diverse acoustic characteristics and improving the generalization ability across different types of noise. When applied to HuBERT, our model shows relative performance improvements of 23.3% on LibriSpeech test-clean and 13.2% on test-other, compared to the baseline model pre-trained on noisy speech.