# Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System

arXiv: https://arxiv.org/abs/1804.05160 · HTML: https://mlsanity.com/paper/1804.05160/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2018.

## Built on by (most cited first, 30 of 47)

- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Self-Supervised Speaker Verification with Simple Siamese Network and Self-Supervised Regularization](https://mlsanity.com/paper/2112.04459/) (2021)
- [On the Robustness of Arabic Speech Dialect Identification](https://mlsanity.com/paper/2306.03789/) (2023)
- [Deepfake audio detection by speaker verification](https://mlsanity.com/paper/2209.14098/) (2022)
- [Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learning](https://mlsanity.com/paper/2207.05506/) (2022)
- [Multi-task Voice Activated Framework using Self-supervised Learning](https://mlsanity.com/paper/2110.01077/) (2021)
- [Multi-View Self-Attention Based Transformer for Speaker Recognition](https://mlsanity.com/paper/2110.05036/) (2021)
- [Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation](https://mlsanity.com/paper/2309.03019/) (2023)
- [USAT: A Universal Speaker-Adaptive Text-to-Speech Approach](https://mlsanity.com/paper/2404.18094/) (2024)
- [SIG-VC: A Speaker Information Guided Zero-shot Voice Conversion System for Both Human Beings and Machines](https://mlsanity.com/paper/2111.03811/) (2021)
- [Multi-Frequency Information Enhanced Channel Attention Module for Speaker Representation Learning](https://mlsanity.com/paper/2207.04540/) (2022)
- [A Compact End-to-End Model with Local and Global Context for Spoken Language Identification](https://mlsanity.com/paper/2210.15781/) (2022)
- [Large-scale learning of generalised representations for speaker recognition](https://mlsanity.com/paper/2210.10985/) (2022)
- [Synthetic Speech Source Tracing using Metric Learning](https://mlsanity.com/paper/2506.02590/) (2025)
- [Real Additive Margin Softmax for Speaker Verification](https://mlsanity.com/paper/2110.09116/) (2021)
- [Improving Transformer-based Networks With Locality For Automatic Speaker Verification](https://mlsanity.com/paper/2302.08639/) (2023)
- [Towards spoken dialect identification of Irish](https://mlsanity.com/paper/2307.07436/) (2023)
- [Speech Rhythm-Based Speaker Embeddings Extraction from Phonemes and Phoneme Duration for Multi-Speaker Speech Synthesis](https://mlsanity.com/paper/2402.07085/) (2024)
- [Explainable AI in Speaker Recognition -- Making Latent Representations Understandable](https://mlsanity.com/paper/2604.23354/) (2026)
- [Binary Neural Network for Speaker Verification](https://mlsanity.com/paper/2104.02306/) (2021)
- [Pretraining Approaches for Spoken Language Recognition: TalTech Submission to the OLR 2021 Challenge](https://mlsanity.com/paper/2205.07083/) (2022)
- [A Study on Bias and Fairness In Deep Speaker Recognition](https://mlsanity.com/paper/2303.08026/) (2023)
- [VIFS: An End-to-End Variational Inference for Foley Sound Synthesis](https://mlsanity.com/paper/2306.05004/) (2023)
- [RADIO: Reference-Agnostic Dubbing Video Synthesis](https://mlsanity.com/paper/2309.01950/) (2023)
- [Enhancing Risk Assessment in Transformers with Loss-at-Risk Functions](https://mlsanity.com/paper/2411.02558/) (2024)
- [DRASP: A Dual-Resolution Attentive Statistics Pooling Framework for Automatic MOS Prediction](https://mlsanity.com/paper/2508.21407/) (2025)
- [Explainable AI in Speaker Recognition -- Attention Map Visualisation and Evaluation](https://mlsanity.com/paper/2606.22901/) (2026)
- [Adaptive Margin Circle Loss for Speaker Verification](https://mlsanity.com/paper/2106.08004/) (2021)
- [GC-TTS: Few-shot Speaker Adaptation with Geometric Constraints](https://mlsanity.com/paper/2108.06890/) (2021)

## Abstract (arXiv)

In this paper, we explore the encoding/pooling layer and loss function in the end-to-end speaker and language recognition system. First, a unified and interpretable end-to-end system for both speaker and language recognition is developed. It accepts variable-length input and produces an utterance level result. In the end-to-end system, the encoding layer plays a role in aggregating the variable-length input sequence into an utterance level representation. Besides the basic temporal average pooling, we introduce a self-attentive pooling layer and a learnable dictionary encoding layer to get the utterance level representation. In terms of loss function for open-set speaker verification, to get more discriminative speaker embedding, center loss and angular softmax loss is introduced in the end-to-end system. Experimental results on Voxceleb and NIST LRE 07 datasets show that the performance of end-to-end learning system could be significantly improved by the proposed encoding layer and loss function.