# Voice Filter: Few-shot text-to-speech speaker adaptation using voice conversion as a post-processing module

arXiv: https://arxiv.org/abs/2202.08164 · HTML: https://mlsanity.com/paper/2202.08164/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 28 of 28)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [High Fidelity Speech Synthesis with Adversarial Networks](https://mlsanity.com/paper/1909.11646/) (2019)
- [Sample Efficient Adaptive Text-to-Speech](https://mlsanity.com/paper/1809.10460/) (2018)
- [LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition](https://mlsanity.com/paper/2008.03687/) (2020)
- [VQVC+: One-Shot Voice Conversion by Vector Quantization and U-Net architecture](https://mlsanity.com/paper/2006.04154/) (2020)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning](https://mlsanity.com/paper/1904.06508/) (2019)
- [Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding](https://mlsanity.com/paper/2005.08484/) (2020)
- [Low-resource expressive text-to-speech using data augmentation](https://mlsanity.com/paper/2011.05707/) (2020)
- [BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization](https://mlsanity.com/paper/2002.01953/) (2020)
- [CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech](https://mlsanity.com/paper/2004.14617/) (2020)
- [Effect of data reduction on sequence-to-sequence neural TTS](https://mlsanity.com/paper/1811.06315/) (2018)
- [Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages](https://mlsanity.com/paper/2008.04549/) (2020)
- [From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint](https://mlsanity.com/paper/2005.04587/) (2020)
- [AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN](https://mlsanity.com/paper/2005.05642/) (2020)
- [Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech](https://mlsanity.com/paper/2106.12896/) (2021)
- [High quality, lightweight and adaptable TTS using LPCNet](https://mlsanity.com/paper/1905.00590/) (2019)
- [The Multi-speaker Multi-style Voice Cloning Challenge 2021](https://mlsanity.com/paper/2104.01818/) (2021)
- [Speaker verification-derived loss and data augmentation for DNN-based multispeaker speech synthesis](https://mlsanity.com/paper/2106.01789/) (2021)

## Built on by (most cited first, 8 of 8)

- [Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation](https://mlsanity.com/paper/2210.15868/) (2022)
- [Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakers](https://mlsanity.com/paper/2211.00585/) (2022)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation](https://mlsanity.com/paper/2210.17264/) (2022)
- [HyperTTS: Parameter Efficient Adaptation in Text to Speech using Hypernetworks](https://mlsanity.com/paper/2404.04645/) (2024)
- [Speech Separation with Pretrained Frontend to Minimize Domain Mismatch](https://mlsanity.com/paper/2411.03085/) (2024)
- [SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic Spaces](https://mlsanity.com/paper/2307.12445/) (2023)
- [Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation](https://mlsanity.com/paper/2504.02302/) (2025)

## Abstract (arXiv)

State-of-the-art text-to-speech (TTS) systems require several hours of recorded speech data to generate high-quality synthetic speech. When using reduced amounts of training data, standard TTS models suffer from speech quality and intelligibility degradations, making training low-resource TTS systems problematic. In this paper, we propose a novel extremely low-resource TTS method called Voice Filter that uses as little as one minute of speech from a target speaker. It uses voice conversion (VC) as a post-processing module appended to a pre-existing high-quality TTS system and marks a conceptual shift in the existing TTS paradigm, framing the few-shot TTS problem as a VC task. Furthermore, we propose to use a duration-controllable TTS system to create a parallel speech corpus to facilitate the VC task. Results show that the Voice Filter outperforms state-of-the-art few-shot speech synthesis techniques in terms of objective and subjective metrics on one minute of speech on a diverse set of voices, while being competitive against a TTS model built on 30 times more data.