Reading Less While Writing: A Closed-Form Bandwidth Dial for Streaming Multimodal Decoders
Not on the map. Why?
Abstract (arXiv)
A decoder that turns video or audio into text conventionally consumes the entire input before emitting a word. Offline this is merely more than the task requires; live it is impossible, since a caption cannot wait for a match to end. Streaming systems bolt on a fixed rule such as wait-k, which waits for the same number of input tokens before every word, regardless of the input's length or pace. We replace the fixed offset with ZENDAYA, a schedule governed by a single continuous parameter γ. It makes the visible source prefix a closed-form function of generation progress, scaled by the input's own predicted length, so an ordinary offline decoder and a real-time streaming decoder become two endpoints of one family rather than separate models. The same scalar fixes, in closed form, the mean fraction of source consumed per emitted word, barE(γ) ≈ 1/(1+γ), making it at once a latency dial and an interpretable budget. We prove a structural dependency theorem: under any schedule fixed in advance and non-decreasing, no emitted token can depend on input that has not yet arrived. The guarantee holds for trained and untrained weights alike, and extends to unbounded streams under arbitrary asynchronous arrival. The empirical result is counterintuitive: seeing less can produce better text, because a flood of source dilutes attention exactly when the model has the least of its own output to anchor on. Trained from scratch across two modalities and three public corpora (Charades-STA, ActivityNet Captions, LibriHeavy), a compact 29M-parameter decoder matches or beats the fixed schedule while reading less of the source, with the sharpest gains at the lowest latencies, where a fixed offset collapses. Streaming METEOR gains are statistically significant on all three corpora.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 5 Aug 2026.
Builds on
14 of 24 references.
References · 14
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2017Dense-Captioning Events in Videoscited by 793
- 2017TALL: Temporal Activity Localization via Language Querycited by 584
- 2020Multi-modal Dense Video Captioningcited by 113
- 2019Monotonic Multihead Attentioncited by 93
- 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
- 2024Streaming Dense Video Captioningcited by 82
- 2020Efficient Wait-k Models for Simultaneous Machine Translationcited by 61
- 2021SimulSLT: End-to-End Simultaneous Sign Language Translationcited by 27
- 2021Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognitioncited by 14
- 2022Variable Attention Masking for Configurable Transformer Transducer Speech Recognitioncited by 14
Built on by
Not cited yet.