# ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models

arXiv: https://arxiv.org/abs/2401.17230 · HTML: https://mlsanity.com/paper/2401.17230/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 57 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 30 of 46)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [Pushing the limits of raw waveform speaker recognition](https://mlsanity.com/paper/2203.08488/) (2022)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification](https://mlsanity.com/paper/1904.08104/) (2019)
- [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020)

## Built on by (most cited first, 22 of 22)

- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [SpoofCeleb: Speech Deepfake Detection and SASV In The Wild](https://mlsanity.com/paper/2409.17285/) (2024)
- [Preference Alignment Improves Language Model-Based TTS](https://mlsanity.com/paper/2409.12403/) (2024)
- [Nes2Net: A Lightweight Nested Architecture for Foundation Model Driven Speech Anti-spoofing](https://mlsanity.com/paper/2504.05657/) (2025)
- [VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis](https://mlsanity.com/paper/2412.19259/) (2024)
- [ESPnet-SpeechLM: An Open Speech Language Model Toolkit](https://mlsanity.com/paper/2502.15218/) (2025)
- [OpusLM: A Family of Open Unified Speech Language Models](https://mlsanity.com/paper/2506.17611/) (2025)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec](https://mlsanity.com/paper/2410.15764/) (2024)
- [vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders](https://mlsanity.com/paper/2409.01995/) (2024)
- [Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnet](https://mlsanity.com/paper/2508.16576/) (2025)
- [To what extent can ASV systems naturally defend against spoofing attacks?](https://mlsanity.com/paper/2406.05339/) (2024)
- [A Non-autoregressive Model for Joint STT and TTS](https://mlsanity.com/paper/2501.09104/) (2025)
- [Voice Quality Dimensions as Interpretable Primitives for Speaking Style for Atypical Speech and Affect](https://mlsanity.com/paper/2505.21809/) (2025)
- [Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks](https://mlsanity.com/paper/2601.12205/) (2026)
- [ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration](https://mlsanity.com/paper/2409.09506/) (2024)
- [Geolocation-Aware Robust Spoken Language Identification](https://mlsanity.com/paper/2508.17148/) (2025)
- [Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions](https://mlsanity.com/paper/2509.24457/) (2025)
- [Probing Low Frame Rate Degradation in Neural Audio Codecs](https://mlsanity.com/paper/2606.16969/) (2026)
- [Kiwano: A Cutting-Edge Open-Source Toolkit for Speaker Verification](https://mlsanity.com/paper/2606.22369/) (2026)
- [StreamAlign: Streaming Text-Aligned Speech Tokenization](https://mlsanity.com/paper/2609.09719/) (2026)
- [Challenges of Multi-Speaker Extraction for Real Conversational Speech Enhancement](https://mlsanity.com/paper/2609.25948/) (2026)

## Abstract (arXiv)

This paper introduces ESPnet-SPK, a toolkit designed with several objectives for training speaker embedding extractors. First, we provide an open-source platform for researchers in the speaker recognition community to effortlessly build models. We provide several models, ranging from x-vector to recent SKA-TDNN. Through the modularized architecture design, variants can be developed easily. We also aspire to bridge developed models with other domains, facilitating the broad research community to effortlessly incorporate state-of-the-art embedding extractors. Pre-trained embedding extractors can be accessed in an off-the-shelf manner and we demonstrate the toolkit's versatility by showcasing its integration with two tasks. Another goal is to integrate with diverse self-supervised learning features. We release a reproducible recipe that achieves an equal error rate of 0.39% on the Vox1-O evaluation protocol using WavLM-Large with ECAPA-TDNN.