# Noro: Noise-Robust One-shot Voice Conversion with Hidden Speaker Representation Learning

arXiv: https://arxiv.org/abs/2411.19770 · HTML: https://mlsanity.com/paper/2411.19770/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 29 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [Amphion: An Open-Source Audio, Music and Speech Generation Toolkit](https://mlsanity.com/paper/2312.09911/) (2023)
- [SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention](https://mlsanity.com/paper/2312.08676/) (2023)
- [How Far Are We from Robust Voice Conversion: A Survey](https://mlsanity.com/paper/2011.12063/) (2020)
- [Learning Speaker Embedding from Text-to-Speech](https://mlsanity.com/paper/2010.11221/) (2020)
- [Noise-robust voice conversion with domain adversarial training](https://mlsanity.com/paper/2201.10693/) (2022)
- [Toward Degradation-Robust Voice Conversion](https://mlsanity.com/paper/2110.07537/) (2021)
- [Learning Noise-independent Speech Representation for High-quality Voice Conversion for Noisy Target Speakers](https://mlsanity.com/paper/2207.00756/) (2022)
- [Noise-Robust Voice Conversion by Conditional Denoising Training Using Latent Variables of Recording Quality and Environment](https://mlsanity.com/paper/2406.07280/) (2024)

## Built on by (most cited first, 1 of 1)

- [AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation](https://mlsanity.com/paper/2610.14697/) (2026)

## Abstract (arXiv)

The effectiveness of one-shot voice conversion (VC) decreases in real-world scenarios where reference speeches, which are often sourced from the internet, contain various disturbances like background noise. To address this issue, we introduce Noro, a noise-robust one-shot VC system. Noro features innovative components tailored for VC using noisy reference speeches, including a dual-branch reference encoding module and a noise-agnostic contrastive speaker loss. Experimental results demonstrate that Noro outperforms our baseline system in both clean and noisy scenarios, highlighting its efficacy for real-world applications. Additionally, we investigate the hidden speaker representation capabilities of our baseline system by repurposing its reference encoder as a speaker encoder. The results show that it is competitive with several advanced self-supervised learning models for speaker representation under the SUPERB settings, highlighting the potential for advancing speaker representation learning through one-shot VC tasks.