Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation
Not on the map. Why?
Abstract (arXiv)
Recent advancements in personalized speech generation have brought synthetic speech increasingly close to the realism of target speakers' recordings, yet multimodal speaker generation remains on the rise. This paper introduces UniSpeaker, a unified approach for multimodality-driven speaker generation. Specifically, we propose a unified voice aggregator based on KV-Former, applying soft contrastive loss to map diverse voice description modalities into a shared voice space, ensuring that the generated voice aligns more closely with the input descriptions. To evaluate multimodality-driven voice control, we build the first multimodality-based voice control (MVC) benchmark, focusing on voice suitability, voice diversity, and speech quality. UniSpeaker is evaluated across five tasks using the MVC benchmark, and the experimental results demonstrate that UniSpeaker outperforms previous modality-specific models. Speech samples are available at urlhttps://UniSpeaker.github.io.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 11 Jan 2025.
Builds on
23 of 36 references.
References · 23
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
- 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
- 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
- 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
- 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
- 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
11 more
- 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 95
- 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
- 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
- 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 47
- 2023Imaginary Voice: Face-styled Diffusion Model for Text-to-Speechcited by 27
- 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
- 2023PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Promptscited by 18
- 2024LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioningcited by 15
- 2024Voice Attribute Editing with Text Promptcited by 9
- 2023Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignmentcited by 8
- 2024Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrievalcited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.