# SoulX-Transcriber: A Robust End-to-End Framework for Multi-Speaker Speech Transcription

arXiv: https://arxiv.org/abs/2606.02400 · HTML: https://mlsanity.com/paper/2606.02400/

## Key facts

- Cited by 5 papers in the map (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 21 of 21)

- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [Connecting Speech Encoder and Large Language Model for ASR](https://mlsanity.com/paper/2309.13963/) (2023)
- [Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR](https://mlsanity.com/paper/2110.03151/) (2021)
- [3D-Speaker: A Large-Scale Multi-Device, Multi-Distance, and Multi-Dialect Corpus for Speech Representation Disentanglement](https://mlsanity.com/paper/2306.15354/) (2023)
- [SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models](https://mlsanity.com/paper/2508.06372/) (2025)
- [VIBEVOICE-ASR Technical Report](https://mlsanity.com/paper/2601.18184/) (2026)
- [Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approach](https://mlsanity.com/paper/2309.05248/) (2023)
- [Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions](https://mlsanity.com/paper/2409.08596/) (2024)
- [Speaker Diarization with Lexical Information](https://mlsanity.com/paper/2004.06756/) (2020)
- [Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio](https://mlsanity.com/paper/2511.16046/) (2025)
- [Summary on The Multilingual Conversational Speech Language Model Challenge: Datasets, Tasks, Baselines, and Methods](https://mlsanity.com/paper/2509.13785/) (2025)
- [WenetSpeech-Chuan: A Large-Scale Sichuanese Corpus with Rich Annotation for Dialectal Speech Processing](https://mlsanity.com/paper/2509.18004/) (2025)
- [TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding](https://mlsanity.com/paper/2601.06896/) (2026)
- [Can Audio Large Language Models Verify Speaker Identity?](https://mlsanity.com/paper/2509.19755/) (2025)
- [Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR](https://mlsanity.com/paper/2604.03074/) (2026)
- [Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition](https://mlsanity.com/paper/2603.25377/) (2026)

## Built on by (most cited first, 5 of 5)

- [The SLT 2026 SmartGlasses Challenge: Benchmarking Egocentric Multi-Talker Speech Recognition and Understanding with Audio-Language Models](https://mlsanity.com/paper/2608.12034/) (2026)
- [DiaScriber: A Speech LLM for Joint Diarization and Transcription in Multi-Speaker Scenarios](https://mlsanity.com/paper/2608.22796/) (2026)
- [VibeVoice-ASR-Streaming Technical Report](https://mlsanity.com/paper/2609.02812/) (2026)
- [SmoothConv and DuplexConv: Complementary Mandarin Multi-Party Conversational Speech Corpora for Speech Interaction](https://mlsanity.com/paper/2610.11150/) (2026)
- [Frontend-Assisted Large Audio-Language Models for SLT2026 SmartGlasses Challenge](https://mlsanity.com/paper/2610.13282/) (2026)

## Abstract (arXiv)

Recent advances in Automatic Speech Recognition (ASR) and Large Language Models (LLMs) have significantly improved speech understanding capabilities. However, multi-speaker speech transcription remains challenging task, constrained by highly similar speaker voices, rapid turn-taking transitions, overlapping utterances and inaccurate speaker boundary segmentation. These challenges become particularly pronounced in real-world conversational audio, where speaker dynamics and acoustic conditions are highly variable. This technical report presents SoulX-Transcriber, a unified multi-speaker transcription system that jointly models speaker diarization (SD) and ASR within an LLM-based framework. SoulX-Transcriber adopts a two-stage training strategy to improve both speaker discrimination and transcription robustness. In the first stage, speaker-aware multi-task continuous pre-training enhances speaker representation learning and boundary perception. In the second stage, supervised fine-tuning further optimizes the model for accurate end-to-end speaker-attributed transcription under complex multi-speaker conditions. SoulX-Transcriber delivers strong performance and robustness across multiple public benchmarks, including AliMeeting, AISHELL-4, and AMI, while maintaining high adaptability to multi-domain scenarios.