# Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio

arXiv: https://arxiv.org/abs/2511.16046 · HTML: https://mlsanity.com/paper/2511.16046/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2025.

## Built on by (most cited first, 10 of 10)

- [VIBEVOICE-ASR Technical Report](https://mlsanity.com/paper/2601.18184/) (2026)
- [TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding](https://mlsanity.com/paper/2601.06896/) (2026)
- [MOSS Transcribe Diarize Technical Report](https://mlsanity.com/paper/2601.01554/) (2026)
- [Entropy-Aware Domain-Routed Mixture-of-Experts Speech-LLM Framework: A Case Study of Multi-Domain Child-Adult ASR](https://mlsanity.com/paper/2606.10454/) (2026)
- [G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition](https://mlsanity.com/paper/2603.10468/) (2026)
- [Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving](https://mlsanity.com/paper/2607.01733/) (2026)
- [Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognition](https://mlsanity.com/paper/2609.02901/) (2026)
- [The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge](https://mlsanity.com/paper/2609.11724/) (2026)
- [Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs](https://mlsanity.com/paper/2609.17981/) (2026)
- [Inference-Time Target Speaker Unlearning in LLM-Based Automatic Speech Recognition](https://mlsanity.com/paper/2609.30439/) (2026)

## Abstract (arXiv)

Joint automatic speech recognition (ASR) and speaker diarization aim to answer the question "who spoke what" in multi-speaker scenarios. In this paper, we present an end-to-end speech large language model (Speech-LLM) for Joint strEamable DIarization and aSr (JEDIS-LLM). The model is trained only on short audio under 20s but is capable of streamable inference on long-form audio without additional training. This is achieved by introducing a Speaker Prompt Cache (SPC) with an on-the-fly update mechanism during chunk-wise streaming inference, inspired by the autoregressive nature of LLMs. The SPC also allows the seamless use of pre-enrolled speaker profiles which is common in many scenarios like meeting transcription. To further enhance diarization capability, we incorporate word-level speaker supervision into the speech encoder during training. Experimental results demonstrate that our system outperforms strong baselines, including Sortformer and Meta-Cat in the local setting on audio up to 20s, and DiarizationLM on long-form audio, despite being fully end-to-end and streamable while DiarizationLM follows a cascaded offline pipeline. To the best of our knowledge, this is the first work enabling zero-shot streamable joint ASR and diarization on long audio using a Speech-LLM trained only on short audio, achieving state-of-the-art performance.