mlsanityData to 7 Oct 2026

Analyzing and Improving Speaker Similarity Assessment for Speech Synthesis

arXiv 2507.02176 ↗First posted 2 Jul 2025cs.SD cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Modeling voice identity is challenging due to its multifaceted nature. In generative speech systems, identity is often assessed using automatic speaker verification (ASV) embeddings, designed for discrimination rather than characterizing identity. This paper investigates which aspects of a voice are captured in such representations. We find that widely used ASV embeddings focus mainly on static features like timbre and pitch range, while neglecting dynamic elements such as rhythm. We also identify confounding factors that compromise speaker similarity measurements and suggest mitigation strategies. To address these gaps, we propose U3D, a metric that evaluates speakers' dynamic rhythm patterns. This work contributes to the ongoing challenge of assessing speaker identity consistency in the context of ever-better voice cloning systems. We publicly release our code.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 62 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

32 of 62 references.

References · 32

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  5. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  6. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  7. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  8. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  9. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  10. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  11. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  12. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
20 more
  1. 2024ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preservingcited by 42
  2. 2021Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speechcited by 37
  3. 2022Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterancescited by 32
  4. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 26
  5. 2023The Singing Voice Conversion Challenge 2023cited by 25
  6. 2024CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Modelscited by 24
  7. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  8. 2020Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissionscited by 19
  9. 2024VoxSim: A perceptual voice similarity datasetcited by 10
  10. 2024The VoxCeleb Speaker Recognition Challenge: A Retrospectivecited by 9
  11. 2024Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based Speech Language Modelcited by 8
  12. 2021The Multi-speaker Multi-style Voice Cloning Challenge 2021cited by 7
  13. 2024Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluationcited by 7
  14. 2023Rhythm Modeling for Voice Conversioncited by 6
  15. 2021SVSNet: An End-to-end Speaker Voice Similarity Assessment Modelcited by 4
  16. 2024VoxGenesis: Unsupervised Discovery of Latent Speaker Manifold for Speech Synthesiscited by 4
  17. 2025Good practices for evaluation of synthesized speechcited by 4
  18. 2025Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancementcited by 3
  19. 2022Automatic Evaluation of Speaker Similaritycited by 2
  20. 2024ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Featurescited by 0
  2. 2026DiffAnon: Diffusion-based Prosody Control for Voice Anonymizationcited by 0
  3. 2026A Cross Community Agenda for Speech AIcited by 0