# Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets

arXiv: https://arxiv.org/abs/2604.07355 · HTML: https://mlsanity.com/paper/2604.07355/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 17 references that are in the map.
- First posted on arXiv 28 Mar 2026.

## Builds on (most cited first, 4 of 4)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem](https://mlsanity.com/paper/1706.10059/) (2017)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)

## Built on by (most cited first, 10 of 10)

- [Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs](https://mlsanity.com/paper/2604.18576/) (2026)
- [FutureSim: Replaying World Events to Evaluate Adaptive Agents](https://mlsanity.com/paper/2605.15188/) (2026)
- [Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents](https://mlsanity.com/paper/2605.00420/) (2026)
- [Coordination as an Architectural Layer for LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2605.03310/) (2026)
- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [Nous: An Attempt to Extract and Inject the Cognition Behind Prediction-Market Behavior](https://mlsanity.com/paper/2606.13038/) (2026)
- [Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry](https://mlsanity.com/paper/2607.01661/) (2026)
- [Beyond Forecasting: The Belief-to-Trade Layer in Prediction-Market Agents](https://mlsanity.com/paper/2607.03015/) (2026)
- [StableEval Arena: A Cost-Aware Agentic Benchmark for Stablecoin Price Stability Prediction](https://mlsanity.com/paper/2609.18949/) (2026)
- [Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents](https://mlsanity.com/paper/2609.28876/) (2026)

## Abstract (arXiv)

We introduce Prediction Arena, a benchmark for evaluating AI models' predictive accuracy and decision-making by enabling them to trade autonomously on live prediction markets with real capital. Unlike synthetic benchmarks, Prediction Arena tests models in environments where trades execute on actual exchanges (Kalshi and Polymarket), providing objective ground truth that cannot be gamed or overfitted. Each model operates as an independent agent starting with 10,000, making autonomous decisions every 15-45 minutes. Over a 57-day longitudinal evaluation (January 12 to March 9, 2026), we track two cohorts: six frontier models in live trading (Cohort 1, full period) and four next-generation models in paper trading (Cohort 2, 3-day preliminary). For Cohort 1, final Kalshi returns range from -16.0% to -30.8%. Our analysis identifies a clear performance hierarchy: initial prediction accuracy and the ability to capitalize on correct predictions are the main drivers, while research volume shows no correlation with outcomes. A striking cross-platform contrast emerges from parallel Polymarket live trading: Cohort 1 models averaged only -1.1% on Polymarket vs. -22.6% on Kalshi, with grok-4-20-checkpoint achieving a 71.4% settlement win rate - the highest across any platform or cohort. gemini-3.1-pro-preview (Cohort 2), which executed zero trades on Kalshi, achieved +6.02% on Polymarket in 3 days - the best return of any model across either cohort - demonstrating that platform design has a profound effect on which models succeed. Beyond performance, we analyze computational efficiency (token usage, cycle time), settlement accuracy, exit patterns, and market preferences, providing a comprehensive view of how frontier models behave under real financial pressure.