# SVSNet+: Enhancing Speaker Voice Similarity Assessment Models with Representations from Speech Foundation Models

arXiv: https://arxiv.org/abs/2406.08445 · HTML: https://mlsanity.com/paper/2406.08445/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [Generalization Ability of MOS Prediction Networks](https://mlsanity.com/paper/2110.02635/) (2021)
- [The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods](https://mlsanity.com/paper/1804.04262/) (2018)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch](https://mlsanity.com/paper/2310.17864/) (2023)
- [Boosting Self-Supervised Embeddings for Speech Enhancement](https://mlsanity.com/paper/2204.03339/) (2022)
- [Utilizing Self-supervised Representations for MOS Prediction](https://mlsanity.com/paper/2104.03017/) (2021)
- [Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissions](https://mlsanity.com/paper/2009.03554/) (2020)
- [A Study on Incorporating Whisper for Robust Speech Assessment](https://mlsanity.com/paper/2309.12766/) (2023)
- [Fusion of Self-supervised Learned Models for MOS Prediction](https://mlsanity.com/paper/2204.04855/) (2022)
- [Pre-trained Speech Representations as Feature Extractors for Speech Quality Assessment in Online Conferencing Applications](https://mlsanity.com/paper/2210.00259/) (2022)
- [Federated Learning for ASR based on Wav2vec 2.0](https://mlsanity.com/paper/2302.10790/) (2023)
- [SVSNet: An End-to-end Speaker Voice Similarity Assessment Model](https://mlsanity.com/paper/2107.09392/) (2021)
- [Mandarin Electrolaryngeal Speech Voice Conversion using Cross-domain Features](https://mlsanity.com/paper/2306.06653/) (2023)

## Built on by (most cited first, 1 of 1)

- [Acoustic and perceptual differences between standard and accented speech and their voice clones](https://mlsanity.com/paper/2604.01562/) (2026)

## Abstract (arXiv)

Representations from pre-trained speech foundation models (SFMs) have shown impressive performance in many downstream tasks. However, the potential benefits of incorporating pre-trained SFM representations into speaker voice similarity assessment have not been thoroughly investigated. In this paper, we propose SVSNet+, a model that integrates pre-trained SFM representations to improve performance in assessing speaker voice similarity. Experimental results on the Voice Conversion Challenge 2018 and 2020 datasets show that SVSNet+ incorporating WavLM representations shows significant improvements compared to baseline models. In addition, while fine-tuning WavLM with a small dataset of the downstream task does not improve performance, using the same dataset to learn a weighted-sum representation of WavLM can substantially improve performance. Furthermore, when WavLM is replaced by other SFMs, SVSNet+ still outperforms the baseline models and exhibits strong generalization ability.