# Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition

arXiv: https://arxiv.org/abs/2603.25377 · HTML: https://mlsanity.com/paper/2603.25377/

## Key facts

- Cited by 2 papers in the map (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 19 of 19)

- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [Connecting Speech Encoder and Large Language Model for ASR](https://mlsanity.com/paper/2309.13963/) (2023)
- [An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verification](https://mlsanity.com/paper/2305.12838/) (2023)
- [Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR](https://mlsanity.com/paper/2110.03151/) (2021)
- [SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models](https://mlsanity.com/paper/2508.06372/) (2025)
- [VIBEVOICE-ASR Technical Report](https://mlsanity.com/paper/2601.18184/) (2026)
- [Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approach](https://mlsanity.com/paper/2309.05248/) (2023)
- [Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions](https://mlsanity.com/paper/2409.08596/) (2024)
- [Speaker Diarization with Lexical Information](https://mlsanity.com/paper/2004.06756/) (2020)
- [Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio](https://mlsanity.com/paper/2511.16046/) (2025)
- [TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding](https://mlsanity.com/paper/2601.06896/) (2026)
- [TellWhisper: Tell Whisper Who Speaks When](https://mlsanity.com/paper/2601.03712/) (2026)

## Built on by (most cited first, 2 of 2)

- [SoulX-Transcriber: A Robust End-to-End Framework for Multi-Speaker Speech Transcription](https://mlsanity.com/paper/2606.02400/) (2026)
- [DiaScriber: A Speech LLM for Joint Diarization and Transcription in Multi-Speaker Scenarios](https://mlsanity.com/paper/2608.22796/) (2026)

## Abstract (arXiv)

Large Audio-Language Models (LALMs) have demonstrated remarkable performance in end-to-end speaker diarization and recognition. However, their speaker discriminability remains limited due to the scarcity of large-scale conversational data and the absence of explicit speaker representation optimization. To address this, we propose GLSC-SDR, a paradigm that jointly trains speaker classification with diarization and recognition. We further introduce a Global-Local Speaker Classification strategy, which uses clustered speakers as global labels and re-encoded intra-cluster speakers as local labels. This hierarchical design enhances fine-grained speaker discrimination while preserving semantic transcription accuracy. Experiments on AliMeeting, AISHELL-4, and AMI-SDM demonstrate that GLSC-SDR achieves competitive or superior performance compared to simulation-based and multi-encoder approaches, without relying on large-scale real conversational data.