# Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR

arXiv: https://arxiv.org/abs/2609.18333 · HTML: https://mlsanity.com/paper/2609.18333/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context](https://mlsanity.com/paper/2309.08105/) (2023)
- [YODAS: Youtube-Oriented Dataset for Audio and Speech](https://mlsanity.com/paper/2406.00899/) (2024)
- [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025)
- [Pseudo-Labeling for Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2111.00161/) (2021)
- [Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation](https://mlsanity.com/paper/2510.06961/) (2025)
- [Group Relative Policy Optimization for Speech Recognition](https://mlsanity.com/paper/2509.01939/) (2025)
- [Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech](https://mlsanity.com/paper/2604.21045/) (2026)

## Abstract (arXiv)

Streaming automatic speech recognition (ASR) must be judged jointly on what it transcribes and on how quickly it commits each word. Delayed streams modeling (DSM) has become the dominant paradigm for streaming large audio-language models, exposing a structural delay τ that bounds the decoder's lookahead. We show that τ is a poor proxy for user-perceived latency, and that the alignment-based supervision of DSM leaves latency on the table: the same forced-aligned transcript is used at every τ, forcing the model to withhold words it could already commit. We introduce AWED, a word-level emission-delay metric defined relative to the acoustic end of each word, and post-train a DSM recognizer with GRPO under a reward that scores transcription accuracy and measured delay jointly. Trained at a single operating point (τ=6 frames), our model dominates both its supervised fine-tuning initialization and the Voxtral Realtime backbone across all evaluated lookahead budgets: it cuts WER by 30.8% relative at an 80,ms structural delay, and by 5.7% relative at 480,ms while lowering median AWED from 1.17,s to 1.04,s. Latency-rewarded post-training thus advances the accuracy--latency Pareto frontier of streaming ASR without architectural change.