# Content-Dependent Fine-Grained Speaker Embedding for Zero-Shot Speaker Adaptation in Text-to-Speech Synthesis

arXiv: https://arxiv.org/abs/2204.00990 · HTML: https://mlsanity.com/paper/2204.00990/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2022.

## Built on by (most cited first, 10 of 10)

- [Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing](https://mlsanity.com/paper/2503.12042/) (2025)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis](https://mlsanity.com/paper/2311.11745/) (2023)
- [The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge](https://mlsanity.com/paper/2410.23815/) (2024)
- [DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching](https://mlsanity.com/paper/2508.05978/) (2025)
- [ComedicSpeech: Text To Speech For Stand-up Comedies in Low-Resource Scenarios](https://mlsanity.com/paper/2305.12200/) (2023)
- [Stylebook: Content-Dependent Speaking Style Modeling for Any-to-Any Voice Conversion using Only Speech Data](https://mlsanity.com/paper/2309.02730/) (2023)
- [CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion](https://mlsanity.com/paper/2411.18918/) (2024)
- [Clip-TTS: Contrastive Text-content and Mel-spectrogram, A High-Quality Text-to-Speech Method based on Contextual Semantic Understanding](https://mlsanity.com/paper/2502.18889/) (2025)
- [Hearing the Whispers: Black-Box Membership Inference Attacks on Finetuned TTS Models](https://mlsanity.com/paper/2609.01723/) (2026)

## Abstract (arXiv)

Zero-shot speaker adaptation aims to clone an unseen speaker's voice without any adaptation time and parameters. Previous researches usually use a speaker encoder to extract a global fixed speaker embedding from reference speech, and several attempts have tried variable-length speaker embedding. However, they neglect to transfer the personal pronunciation characteristics related to phoneme content, leading to poor speaker similarity in terms of detailed speaking styles and pronunciation habits. To improve the ability of the speaker encoder to model personal pronunciation characteristics, we propose content-dependent fine-grained speaker embedding for zero-shot speaker adaptation. The corresponding local content embeddings and speaker embeddings are extracted from a reference speech, respectively. Instead of modeling the temporal relations, a reference attention module is introduced to model the content relevance between the reference speech and the input text, and to generate the fine-grained speaker embedding for each phoneme encoder output. The experimental results show that our proposed method can improve speaker similarity of synthesized speeches, especially for unseen speakers.