# Diarization-Aware Multi-Speaker Automatic Speech Recognition via Large Language Models

arXiv: https://arxiv.org/abs/2506.05796 · HTML: https://mlsanity.com/paper/2506.05796/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2025.

## Built on by (most cited first, 2 of 2)

- [VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models](https://mlsanity.com/paper/2601.19956/) (2026)
- [Word-Level Text Unmixing via Evidence-Preserving Ownership Routing with Language Models](https://mlsanity.com/paper/2610.06603/) (2026)

## Abstract (arXiv)

Multi-speaker automatic speech recognition (MS-ASR) faces significant challenges in transcribing overlapped speech, a task critical for applications like meeting transcription and conversational analysis. While serialized output training (SOT)-style methods serve as common solutions, they often discard absolute timing information, limiting their utility in time-sensitive scenarios. Leveraging recent advances in large language models (LLMs) for conversational audio processing, we propose a novel diarization-aware multi-speaker ASR system that integrates speaker diarization with LLM-based transcription. Our framework processes structured diarization inputs alongside frame-level speaker and semantic embeddings, enabling the LLM to generate segment-level transcriptions. Experiments demonstrate that the system achieves robust performance in multilingual dyadic conversations and excels in complex, high-overlap multi-speaker meeting scenarios. This work highlights the potential of LLMs as unified back-ends for joint speaker-aware segmentation and transcription.