# StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams

arXiv: https://arxiv.org/abs/2609.38612 · HTML: https://mlsanity.com/paper/2609.38612/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2011.02048/) (2020)
- [Real-Time Reinforcement Learning](https://mlsanity.com/paper/1911.04448/) (2019)
- [Robust Online Monitoring of Signal Temporal Logic](https://mlsanity.com/paper/1506.08234/) (2015)
- [Belief Revision: The Adaptability of Large Language Models Reasoning](https://mlsanity.com/paper/2406.19764/) (2024)
- [Multi-line AI-assisted Code Authoring](https://mlsanity.com/paper/2402.04141/) (2024)
- [Age of Information in Deep Learning-Driven Task-Oriented Communications](https://mlsanity.com/paper/2301.04298/) (2023)
- [Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs](https://mlsanity.com/paper/2505.19481/) (2025)
- [Your LLM Agents are Temporally Blind: The Misalignment Between Tool Use Decisions and Human Time Perception](https://mlsanity.com/paper/2510.23853/) (2025)
- [CoPE: A Small Language Model for Steerable and Scalable Content Labeling](https://mlsanity.com/paper/2512.18027/) (2025)
- [Early and Revocable Time Series Classification](https://mlsanity.com/paper/2109.10285/) (2021)
- [ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agents](https://mlsanity.com/paper/2605.24900/) (2026)
- [Impact of delayed response on Wearable Cognitive Assistance](https://mlsanity.com/paper/2011.02555/) (2020)
- [Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments](https://mlsanity.com/paper/2603.09337/) (2026)
- [StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance](https://mlsanity.com/paper/2606.14571/) (2026)
- [Never Stop Thinking: Continuous-Time Language Agents](https://mlsanity.com/paper/2609.17416/) (2026)

## Abstract (arXiv)

Language models increasingly make real-time decisions in applications that apply the latest answer until a newer one arrives. A late answer can prolong an outdated decision, such as a call recorder still running while a customer reads out card details, an error offline accuracy misses. We make three contributions. First, we release StreamDecisionBench (SDB), a dataset of eight streaming scenarios in four application families, with executable reference decisions derived from public rules. Second, we propose an evaluation protocol and a metric, in-force accuracy: the share of time the applied decision is correct across update intervals of 0.5-8 s. It reflects accuracy and latency jointly, attributing each error to judgment, latency or both. Third, we evaluate fifteen single-model settings, and this attribution separates speed-limited from judgment-limited models: slower, more accurate models lose 42-47% of the time to outdated answers, a fast model 35% to wrong ones. We therefore test hybrids in which a slow model corrects a fast one; with the right pairing and configuration, a hybrid outperforms every single model. However, even the best evaluated system keeps a correct decision in force only about two-thirds of the time, leaving a substantial gap for real-time use.