mlsanityData to 14 Oct 2026

TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding

arXiv 2601.06896 ↗First posted 11 Jan 2026eess.AS cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We present TagSpeech, a unified LLM-based framework that utilizes Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The framework is built on two key designs: (1) decoupled semantic and speaker streams fine-tuned via Serialized Output Training (SOT) to learn turn-taking dynamics; and (2) an interleaved time anchor mechanism that not only supports fine-grained timestamp prediction but also acts as a synchronization signal between semantic understanding and speaker tracking. Compared to previous works that primarily focus on speaker-attributed ASR or implicit diarization, TagSpeech addresses the challenge of fine-grained speaker-content alignment and explicitly models "who spoke what and when" in an end-to-end manner. Experiments on AMI and AliMeeting benchmarks demonstrate that our method achieves consistent improvements in Diarization Error Rate (DER) over strong end-to-end baselines, including Qwen-Omni and Gemini, particularly in handling complex speech overlaps. Moreover, TagSpeech employs a parameter-efficient training paradigm in which the LLM backbone is frozen and only lightweight projectors are trained, resulting in strong performance with low computational cost.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 11 Jan 2026.

Builds on

32 of 46 references.

References · 32

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 191
  5. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 132
  6. 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 111
  7. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  8. 2024VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Groundingcited by 85
  9. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  10. 2019End-to-End Neural Speaker Diarization with Self-attentioncited by 77
  11. 2020End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractorscited by 71
  12. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 70
20 more
  1. 2025TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMscited by 50
  2. 2020Integrating end-to-end neural and clustering-based diarization: Getting the best of both worldscited by 35
  3. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  4. 2021Lhotse: a speech data representation library for the modern deep learning ecosystemcited by 33
  5. 2018A Purely End-to-end System for Multi-speaker Speech Recognitioncited by 30
  6. 2023MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systemscited by 28
  7. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  8. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddingscited by 18
  9. 2025FLAM: Frame-Wise Language-Audio Modelingcited by 18
  10. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  11. 2017Single-Channel Multi-talker Speech Recognition with Permutation Invariant Trainingcited by 16
  12. 2024Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systemscited by 15
  13. 2025Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audiocited by 10
  14. 2024PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordingscited by 8
  15. 2025Efficient Scaling for LLM-based ASRcited by 8
  16. 2025DNCASR: End-to-End Training for Speaker-Attributed ASRcited by 5
  17. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 5
  18. 2024META-CAT: Speaker-Informed Speech Embeddings via Meta Information Concatenation for Multi-talker ASRcited by 4
  19. 2025Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understandingcited by 3
  20. 2025A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026VIBEVOICE-ASR Technical Reportcited by 16
  2. 2026G-STAR: End-to-End Global Speaker-Tracking Attributed Recognitioncited by 0
  3. 2026Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metricscited by 0
  4. 2026MURMUR: An Efficient Inference System for Long-Form ASRcited by 0
  5. 2026Inference-Time Target Speaker Unlearning in LLM-Based Automatic Speech Recognitioncited by 0