# Neural Scoring: A Refreshed End-to-End Approach for Speaker Recognition in Complex Conditions

arXiv: https://arxiv.org/abs/2410.16428 · HTML: https://mlsanity.com/paper/2410.16428/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [BUT System Description to VoxCeleb Speaker Recognition Challenge 2019](https://mlsanity.com/paper/1910.12592/) (2019)
- [End-to-End Text-Dependent Speaker Verification](https://mlsanity.com/paper/1509.08062/) (2015)
- [CN-Celeb: multi-genre speaker recognition](https://mlsanity.com/paper/2012.12468/) (2020)
- [SphereFace2: Binary Classification is All You Need for Deep Face Recognition](https://mlsanity.com/paper/2108.01513/) (2021)
- [Simple Attention Module based Speaker Verification with Iterative noisy label detection](https://mlsanity.com/paper/2110.06534/) (2021)
- [Exploring Binary Classification Loss For Speaker Verification](https://mlsanity.com/paper/2307.08205/) (2023)
- [Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation](https://mlsanity.com/paper/2309.03019/) (2023)
- [NeXt-TDNN: Modernizing Multi-Scale Temporal Convolution Backbone for Speaker Verification](https://mlsanity.com/paper/2312.08603/) (2023)
- [Build a SRE Challenge System: Lessons from VoxSRC 2022 and CNSRC 2022](https://mlsanity.com/paper/2211.00815/) (2022)
- [Towards Robust Speaker Verification with Target Speaker Enhancement](https://mlsanity.com/paper/2103.08781/) (2021)
- [Speaker Verification in Multi-Speaker Environments Using Temporal Feature Fusion](https://mlsanity.com/paper/2206.13808/) (2022)
- [Supervised Speaker Embedding De-Mixing in Two-Speaker Environment](https://mlsanity.com/paper/2001.06397/) (2020)
- [Margin-Mixup: A Method for Robust Speaker Verification in Multi-Speaker Audio](https://mlsanity.com/paper/2304.03515/) (2023)

## Abstract (arXiv)

Modern speaker verification systems primarily rely on speaker embeddings, followed by verification based on cosine similarity between the embedding vectors of the enrollment and test utterances. While effective, these methods struggle with multi-talker speech due to the unidentifiability of embedding vectors. In this paper, we propose Neural Scoring (NS), a refreshed end-to-end framework that directly estimates verification posterior probabilities without relying on test-side embeddings, making it more robust to complex conditions, e.g., with multiple talkers. To make the training of such an end-to-end model more efficient, we introduce a large-scale trial e2e training (LtE2E) strategy, where each test utterance pairs with a set of enrolled speakers, thus enabling the processing of large-scale verification trials per batch. Experiments on the VoxCeleb dataset demonstrate that NS consistently outperforms both the baseline and competitive methods across various conditions, achieving an overall 70.36% reduction in EER compared to the baseline.