mlsanityData to 19 Oct 2026

Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning

arXiv 2606.18134 ↗First posted 16 Jun 2026eess.AScited by 0

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

We propose diarization-conditioned spoken language models (SLMs), a strategy for extending SLMs to far-field multi-talker audio. Rather than adapting the decoder via Serialized Output Training, which risks catastrophic forgetting, we condition the acoustic encoder on diarization masks to extract target-speaker representations, keeping the decoder frozen. We instantiate this as Dixtral, integrating a Diarization Conditioned Whisper (DiCoW) encoder into the Voxtral SLM. On AMI, NOTSOFAR-1, LibriSpeechMix, and Mixer6, Dixtral outperforms Gemini 3.0 Flash, VibeVoice, and Voxtral Mini Transcribe V2 on speaker-attributed transcription by 29.0%, 19.8%, and 16.0% absolute cpWER respectively. On a novel long-form multi-speaker QA benchmark, zero-shot Dixtral matches Gemini on far-field content understanding, and when fine-tuned surpasses both Gemini and Voxtral operating on close-talk across all tasks.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 39 of its 50 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

39 of 50 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper27 Jan
  2. 2026TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding11 Jan
  3. 2025Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition4 Oct
  4. 2025Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognition1 Sep
  5. 2025Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges24 Jul
  6. 2025Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR27 Jun
  7. 2025Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization30 May
  8. 2024SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASR7 Dec
  9. 2024META-CAT: Speaker-Informed Speech Embeddings via Meta Information Concatenation for Multi-talker ASR18 Sep
  10. 2024Target Speaker ASR with Whisper14 Sep
  11. 2024NTT Multi-Speaker ASR System for the DASR Task of CHiME-8 Challenge9 Sep
  12. 2024The USTC-NERCSLIP Systems for the CHiME-8 NOTSOFAR-1 Challenge3 Sep
8 more
  1. 2024SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR4 Mar
  2. 2024NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcription16 Jan
  3. 2023Extending Whisper with prompt tuning to target-speaker ASR13 Dec
  4. 2023End-to-End Joint Target and Non-Target Speakers ASR4 Jun
  5. 2023BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR23 May
  6. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddings1 Nov
  7. 2022Streaming Target-Speaker ASR with Neural Transducer9 Sep
  8. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Training2 Feb

Other references · 19

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,936
  3. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 663
  4. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 263
  5. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 231
  6. 2020Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenariocited by 133
  7. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 131
  8. 2026Ministral 3cited by 118
  9. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 103
  10. 2020VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognitioncited by 54
  11. 2023Neural Target Speech Extraction: An Overviewcited by 52
  12. 2023MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systemscited by 49
7 more
  1. 2025Voxtralcited by 48
  2. 2026VIBEVOICE-ASR Technical Reportcited by 29
  3. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 27
  4. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 20
  5. 2024STCON System for the CHiME-8 Challengecited by 9
  6. 2026Voxtral Realtimecited by 8
  7. 2025Summary of the NOTSOFAR-1 Challenge: Highlights and Learningscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SEAL: Speaker Error Correction using Acoustic-conditioned Large Language Models
  2. 2024DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition
  3. 2026Balancing ASR and diarization in end-to-end LLMs for multi-talker speech recognition
  4. 2025BUT System for the MLC-SLM Challenge
  5. 2026DM-ASR: Diarization-aware Multi-speaker ASR with Large Language Models
  6. 2021A Review of Speaker Diarization: Recent Advances with Deep Learning
  7. 2025Pushing the Limits of End-to-End Diarization
  8. 2025Diarization-Aware Multi-Speaker Automatic Speech Recognition via Large Language Models