# Phoneme Hallucinator: One-shot Voice Conversion via Set Expansion

arXiv: https://arxiv.org/abs/2308.06382 · HTML: https://mlsanity.com/paper/2308.06382/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://mlsanity.com/paper/1904.05742/) (2019)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [Voice Conversion With Just Nearest Neighbors](https://mlsanity.com/paper/2305.18975/) (2023)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [SynthASR: Unlocking Synthetic Data for Speech Recognition](https://mlsanity.com/paper/2106.07803/) (2021)
- [Empower Entity Set Expansion via Language Model Probing](https://mlsanity.com/paper/2004.13897/) (2020)
- [NRTSI: Non-Recurrent Time Series Imputation](https://mlsanity.com/paper/2102.03340/) (2021)
- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [Exchangeable Neural ODE for Set Modeling](https://mlsanity.com/paper/2008.02676/) (2020)
- [DEAAN: Disentangled Embedding and Adversarial Adaptation Network for Robust Speaker Representation Learning](https://mlsanity.com/paper/2012.06896/) (2020)
- [Partially Observed Exchangeable Modeling](https://mlsanity.com/paper/2102.06083/) (2021)
- [Transparent Single-Cell Set Classification with Kernel Mean Embeddings](https://mlsanity.com/paper/2201.07322/) (2022)
- [Distribution-based Sketching of Single-Cell Samples](https://mlsanity.com/paper/2207.00584/) (2022)

## Built on by (most cited first, 9 of 9)

- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification](https://mlsanity.com/paper/2403.00293/) (2024)
- [LinearVC: Linear transformations of self-supervised features through the lens of voice conversion](https://mlsanity.com/paper/2506.01510/) (2025)
- [Emotion Recognition and Generation: A Comprehensive Review of Face, Speech, and Text Modalities](https://mlsanity.com/paper/2502.06803/) (2025)
- [QR-VC: Leveraging Quantization Residuals for Linear Disentanglement in Zero-Shot Voice Conversion](https://mlsanity.com/paper/2411.16147/) (2024)
- [Provable Speech Attributes Conversion via Latent Independence](https://mlsanity.com/paper/2510.05191/) (2025)
- [Scaling Human and G2P Supervision for Robust Phonetic Transcription](https://mlsanity.com/paper/2606.16019/) (2026)
- [A New Approach to Voice Authenticity](https://mlsanity.com/paper/2402.06304/) (2024)
- [From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data](https://mlsanity.com/paper/2606.08843/) (2026)

## Abstract (arXiv)

Voice conversion (VC) aims at altering a person's voice to make it sound similar to the voice of another person while preserving linguistic content. Existing methods suffer from a dilemma between content intelligibility and speaker similarity; i.e., methods with higher intelligibility usually have a lower speaker similarity, while methods with higher speaker similarity usually require plenty of target speaker voice data to achieve high intelligibility. In this work, we propose a novel method textitPhoneme Hallucinator that achieves the best of both worlds. Phoneme Hallucinator is a one-shot VC model; it adopts a novel model to hallucinate diversified and high-fidelity target speaker phonemes based just on a short target speaker voice (e.g. 3 seconds). The hallucinated phonemes are then exploited to perform neighbor-based voice conversion. Our model is a text-free, any-to-any VC model that requires no text annotations and supports conversion to any unseen speaker. Objective and subjective evaluations show that textitPhoneme Hallucinator outperforms existing VC methods for both intelligibility and speaker similarity.