# From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint

arXiv: https://arxiv.org/abs/2005.04587 · HTML: https://mlsanity.com/paper/2005.04587/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2020.

## Built on by (most cited first, 14 of 14)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech](https://mlsanity.com/paper/2103.04088/) (2021)
- [Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech](https://mlsanity.com/paper/2111.04040/) (2021)
- [AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person](https://mlsanity.com/paper/2108.04325/) (2021)
- [Speech2Video: Cross-Modal Distillation for Speech to Video Generation](https://mlsanity.com/paper/2107.04806/) (2021)
- [Voice Filter: Few-shot text-to-speech speaker adaptation using voice conversion as a post-processing module](https://mlsanity.com/paper/2202.08164/) (2022)
- [AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation](https://mlsanity.com/paper/2305.15403/) (2023)
- [SIG-VC: A Speaker Information Guided Zero-shot Voice Conversion System for Both Human Beings and Machines](https://mlsanity.com/paper/2111.03811/) (2021)
- [A Text-to-Speech Pipeline, Evaluation Methodology, and Initial Fine-Tuning Results for Child Speech Synthesis](https://mlsanity.com/paper/2203.11562/) (2022)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Karaoker: Alignment-free singing voice synthesis with speech training data](https://mlsanity.com/paper/2204.04127/) (2022)
- [The DKU-DUKEECE System for the Manipulation Region Location Task of ADD 2023](https://mlsanity.com/paper/2308.10281/) (2023)
- [The MSXF TTS System for ICASSP 2022 ADD Challenge](https://mlsanity.com/paper/2201.11400/) (2022)
- [An analysis on the effects of speaker embedding choice in non auto-regressive TTS](https://mlsanity.com/paper/2307.09898/) (2023)

## Abstract (arXiv)

High-fidelity speech can be synthesized by end-to-end text-to-speech models in recent years. However, accessing and controlling speech attributes such as speaker identity, prosody, and emotion in a text-to-speech system remains a challenge. This paper presents a system involving feedback constraint for multispeaker speech synthesis. We manage to enhance the knowledge transfer from the speaker verification to the speech synthesis by engaging the speaker verification network. The constraint is taken by an added loss related to the speaker identity, which is centralized to improve the speaker similarity between the synthesized speech and its natural reference audio. The model is trained and evaluated on publicly available datasets. Experimental results, including visualization on speaker embedding space, show significant improvement in terms of speaker identity cloning in the spectrogram level. Synthesized samples are available online for listening. (https://caizexin.github.io/mlspk-syn-samples/index.html)