# SVSNet: An End-to-end Speaker Voice Similarity Assessment Model

arXiv: https://arxiv.org/abs/2107.09392 · HTML: https://mlsanity.com/paper/2107.09392/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 20 Jul 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods](https://mlsanity.com/paper/1804.04262/) (2018)
- [Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder](https://mlsanity.com/paper/1610.04019/) (2016)
- [The PyTorch-Kaldi Speech Recognition Toolkit](https://mlsanity.com/paper/1811.07453/) (2018)
- [Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks](https://mlsanity.com/paper/1704.00849/) (2017)
- [Quality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model based on BLSTM](https://mlsanity.com/paper/1808.05344/) (2018)
- [MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network](https://mlsanity.com/paper/2103.00110/) (2021)
- [Deep Learning Based Assessment of Synthetic Speech Naturalness](https://mlsanity.com/paper/2104.11673/) (2021)
- [RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification](https://mlsanity.com/paper/1904.08104/) (2019)
- [Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissions](https://mlsanity.com/paper/2009.03554/) (2020)
- [STOI-Net: A Deep Learning based Non-Intrusive Speech Intelligibility Assessment Model](https://mlsanity.com/paper/2011.04292/) (2020)
- [Attention Back-end for Automatic Speaker Verification with Multiple Enrollment Utterances](https://mlsanity.com/paper/2104.01541/) (2021)
- [Multichannel Speech Enhancement by Raw Waveform-mapping using Fully Convolutional Networks](https://mlsanity.com/paper/1909.11909/) (2019)
- [A non-causal FFTNet architecture for speech enhancement](https://mlsanity.com/paper/2006.04469/) (2020)

## Built on by (most cited first, 4 of 4)

- [Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features](https://mlsanity.com/paper/2111.02363/) (2021)
- [Analyzing and Improving Speaker Similarity Assessment for Speech Synthesis](https://mlsanity.com/paper/2507.02176/) (2025)
- [Automatic Evaluation of Speaker Similarity](https://mlsanity.com/paper/2207.00344/) (2022)
- [SVSNet+: Enhancing Speaker Voice Similarity Assessment Models with Representations from Speech Foundation Models](https://mlsanity.com/paper/2406.08445/) (2024)

## Abstract (arXiv)

Neural evaluation metrics derived for numerous speech generation tasks have recently attracted great attention. In this paper, we propose SVSNet, the first end-to-end neural network model to assess the speaker voice similarity between converted speech and natural speech for voice conversion tasks. Unlike most neural evaluation metrics that use hand-crafted features, SVSNet directly takes the raw waveform as input to more completely utilize speech information for prediction. SVSNet consists of encoder, co-attention, distance calculation, and prediction modules and is trained in an end-to-end manner. The experimental results on the Voice Conversion Challenge 2018 and 2020 (VCC2018 and VCC2020) datasets show that SVSNet outperforms well-known baseline systems in the assessment of speaker similarity at the utterance and system levels.