# KellyBench: A Benchmark for Long-Horizon Sequential Decision Making

arXiv: https://arxiv.org/abs/2604.27865 · HTML: https://mlsanity.com/paper/2604.27865/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 54 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 17 of 17)

- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Reasoning Models Don't Always Say What They Think](https://mlsanity.com/paper/2505.05410/) (2025)
- [ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization](https://mlsanity.com/paper/2509.13313/) (2025)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors](https://mlsanity.com/paper/2507.05246/) (2025)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [PostTrainBench: Can LLM Agents Automate LLM Post-Training?](https://mlsanity.com/paper/2603.08640/) (2026)
- [FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction](https://mlsanity.com/paper/2508.11987/) (2025)
- [StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?](https://mlsanity.com/paper/2510.02209/) (2025)
- [LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](https://mlsanity.com/paper/2510.17638/) (2025)
- [MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model](https://mlsanity.com/paper/2409.07486/) (2024)
- [AI-Trader: Benchmarking Autonomous Agents in Real-Time Financial Markets](https://mlsanity.com/paper/2512.10971/) (2025)
- [Bench to the Future: A Pastcasting Benchmark for Forecasting Agents](https://mlsanity.com/paper/2506.21558/) (2025)
- [Scaling Open-Ended Reasoning to Predict the Future](https://mlsanity.com/paper/2512.25070/) (2025)
- [TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?](https://mlsanity.com/paper/2603.00285/) (2026)
- [FinRL Contests: Benchmarking Data-driven Financial Reinforcement Learning Agents](https://mlsanity.com/paper/2504.02281/) (2025)

## Built on by (most cited first, 3 of 3)

- [FutureSim: Replaying World Events to Evaluate Adaptive Agents](https://mlsanity.com/paper/2605.15188/) (2026)
- [When Does Continual Learning Require Learning](https://mlsanity.com/paper/2607.07847/) (2026)
- [Benchmarking Open-Ended Multi-Agent Coordination in Language Agents](https://mlsanity.com/paper/2606.08340/) (2026)

## Abstract (arXiv)

Language models are saturating benchmarks for procedural tasks with narrow objectives. But they are increasingly being deployed in long-horizon, non-stationary environments with open-ended goals. In this paper we introduce KellyBench, an environment for evaluating sequential decision-making in sports betting markets. Agents are placed in a sequential simulation of the 2023-24 English Premier League season and tasked with maximising their long-term bankroll growth. They are given detailed historical data, including advanced statistics, lineups, and public odds. To succeed they must build machine learning models, identify edge in public markets, and adapt as the environment changes over time. We find that all frontier models evaluated lose money on average over the course of the season for five seeds. The best performing model achieves an average return of -8%, and many models experiencing ruin across seeds. To judge strategy sophistication, we use a human expert rubric to grade each model and find their approaches to be unsophisticated compared to human baselines; Claude Opus 4.6 achieves a rubric score of 26.5%, which means there is significant room for improvement. KellyBench is available as an open-access API endpoint at https://openreward.ai/GeneralReasoning/KellyBench.