# 3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization

arXiv: https://arxiv.org/abs/2403.19971 · HTML: https://mlsanity.com/paper/2403.19971/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2024.

## Built on by (most cited first, 16 of 16)

- [SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation](https://mlsanity.com/paper/2507.09862/) (2025)
- [SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models](https://mlsanity.com/paper/2508.06372/) (2025)
- [Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds](https://mlsanity.com/paper/2608.23383/) (2026)
- [Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruning](https://mlsanity.com/paper/2510.04213/) (2025)
- [Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis](https://mlsanity.com/paper/2512.18699/) (2025)
- [OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation](https://mlsanity.com/paper/2604.18326/) (2026)
- [UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2505.10599/) (2025)
- [LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization](https://mlsanity.com/paper/2506.16738/) (2025)
- [Identity as Presence: Towards Appearance and Voice Personalized Joint Audio-Video Generation](https://mlsanity.com/paper/2603.17889/) (2026)
- [AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling](https://mlsanity.com/paper/2605.29488/) (2026)
- [Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models](https://mlsanity.com/paper/2506.13300/) (2025)
- [PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis](https://mlsanity.com/paper/2605.27258/) (2026)
- [Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech](https://mlsanity.com/paper/2607.08208/) (2026)
- [An Omnilingual-ASR-Based Speech-LLM System for the 2nd MLC-SLM Challenge](https://mlsanity.com/paper/2607.12468/) (2026)
- [DiaWhisper-DPO: Role-Attributed Transcription of Clinical Interviews via Failure-Mined Preference Optimization](https://mlsanity.com/paper/2609.16661/) (2026)
- [MECT: Mixture of Experts with CNN-Transformer Network for Speaker verification](https://mlsanity.com/paper/2609.24061/) (2026)

## Abstract (arXiv)

We introduce 3D-Speaker-Toolkit, an open-source toolkit for multimodal speaker verification and diarization, designed for meeting the needs of academic researchers and industrial practitioners. The 3D-Speaker-Toolkit adeptly leverages the combined strengths of acoustic, semantic, and visual data, seamlessly fusing these modalities to offer robust speaker recognition capabilities. The acoustic module extracts speaker embeddings from acoustic features, employing both fully-supervised and self-supervised learning approaches. The semantic module leverages advanced language models to comprehend the substance and context of spoken language, thereby augmenting the system's proficiency in distinguishing speakers through linguistic patterns. The visual module applies image processing technologies to scrutinize facial features, which bolsters the precision of speaker diarization in multi-speaker environments. Collectively, these modules empower the 3D-Speaker-Toolkit to achieve substantially improved accuracy and reliability in speaker-related tasks. With 3D-Speaker-Toolkit, we establish a new benchmark for multimodal speaker analysis. The toolkit also includes a handful of open-source state-of-the-art models and a large-scale dataset containing over 10,000 speakers. The toolkit is publicly available at https://github.com/modelscope/3D-Speaker.