# Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancement

arXiv: https://arxiv.org/abs/2306.08406 · HTML: https://mlsanity.com/paper/2306.08406/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [Domain Adversarial Training for Accented Speech Recognition](https://mlsanity.com/paper/1806.02786/) (2018)
- [Investigating self-supervised learning for speech enhancement and separation](https://mlsanity.com/paper/2203.07960/) (2022)
- [Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning](https://mlsanity.com/paper/2012.07178/) (2020)
- [Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learning](https://mlsanity.com/paper/2110.00165/) (2021)
- [Boosting Self-Supervised Embeddings for Speech Enhancement](https://mlsanity.com/paper/2204.03339/) (2022)
- [DRAFT: A Novel Framework to Reduce Domain Shifting in Self-supervised Learning and Its Application to Children's ASR](https://mlsanity.com/paper/2206.07931/) (2022)
- [Don't speak too fast: The impact of data bias on self-supervised speech models](https://mlsanity.com/paper/2110.07957/) (2021)
- [Noise Adaptive Speech Enhancement using Domain Adversarial Training](https://mlsanity.com/paper/1807.07501/) (2018)
- [Self-Supervised Learning Based Domain Adaptation for Robust Speaker Verification](https://mlsanity.com/paper/2108.13843/) (2021)

## Abstract (arXiv)

Large, pre-trained representation models trained using self-supervised learning have gained popularity in various fields of machine learning because they are able to extract high-quality salient features from input data. As such, they have been frequently used as base networks for various pattern classification tasks such as speech recognition. However, not much research has been conducted on applying these types of models to the field of speech signal generation. In this paper, we investigate the feasibility of using pre-trained speech representation models for a downstream speech enhancement task. To alleviate mismatches between the input features of the pre-trained model and the target enhancement model, we adopt a novel feature normalization technique to smoothly link these modules together. Our proposed method enables significant improvements in speech quality compared to baselines when combined with various types of pre-trained speech models.