mlsanityData to 19 Oct 2026

VibeVoice-ASR-Streaming Technical Report

arXiv 2609.02812 ↗First posted 2 Sep 2026eess.AScited by 0

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 25 of its 31 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

25 of 31 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026SoulX-Transcriber: A Robust End-to-End Framework for Multi-Speaker Speech Transcription1 Jun
  2. 2026G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition11 Mar
  3. 2025Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio20 Nov
  4. 2025Summary on The Multilingual Conversational Speech Language Model Challenge: Datasets, Tasks, Baselines, and Methods17 Sep
  5. 2025SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models8 Aug
  6. 2025Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering24 Jul
  7. 2024Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems10 Sep
  8. 2024On Speaker Attribution with SURT28 Jan
  9. 2023Frame-wise streaming end-to-end speaker diarization with non-autoregressive self-attention-based attractors25 Sep
  10. 2023SURT 2.0: Advances in Transducer-based Multi-talker Speech Recognition18 Jun
  11. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings30 Mar
  12. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Training2 Feb

Other references · 13

  1. 2024Qwen2.5 Technical Reportcited by 5,212
  2. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 497
  3. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 497
  4. 2021AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenariocited by 125
  5. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 51
  6. 2023MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systemscited by 49
  7. 2020Streaming end-to-end multi-talker speech recognitioncited by 41
  8. 2025VibeVoice Technical Reportcited by 38
  9. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 25
  10. 2020BW-EDA-EEND: Streaming End-to-End Neural Speaker Diarization for a Variable Number of Speakerscited by 24
  11. 2024Efficient Streaming LLM for Speech Recognitioncited by 19
  12. 2026Voxtral Realtimecited by 8
1 more
  1. 2026Uni-ASR: Unified LLM-Based Architecture for Non-Streaming and Streaming Automatic Speech Recognitioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VIBEVOICE-ASR Technical Report
  2. 2026Pushing the Boundaries of Streaming Multi-Speaker ASR: A Systematic Study of Architectural Trade-offs
  3. 2024Target Speaker ASR with Whisper
  4. 2021A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio
  5. 2025Unifying Streaming and Non-streaming Zipformer-based ASR
  6. 2026HINTT Submission to the 2nd MLC-SLM Challenge: Comparing Cascaded and Unified Approaches to Diarization and ASR
  7. 2023Updated Corpora and Benchmarks for Long-Form Speech Recognition
  8. 2026DM-ASR: Diarization-aware Multi-speaker ASR with Large Language Models