# FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction

arXiv: https://arxiv.org/abs/2508.11987 · HTML: https://mlsanity.com/paper/2508.11987/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 16 Aug 2025.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 22 of 22)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning](https://mlsanity.com/paper/2504.13914/) (2025)
- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination](https://mlsanity.com/paper/2507.10532/) (2025)
- [SWE-bench Goes Live!](https://mlsanity.com/paper/2505.23419/) (2025)
- [AgentOrchestra: Orchestrating Multi-Agent Intelligence with the Tool-Environment-Agent(TEA) Protocol](https://mlsanity.com/paper/2506.12508/) (2025)
- [Pitfalls in Evaluating Language Model Forecasters](https://mlsanity.com/paper/2506.00723/) (2025)
- [Bench to the Future: A Pastcasting Benchmark for Forecasting Agents](https://mlsanity.com/paper/2506.21558/) (2025)
- [OpenEP: Open-Ended Future Event Prediction](https://mlsanity.com/paper/2408.06578/) (2024)
- [Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture](https://mlsanity.com/paper/2411.13768/) (2024)
- [Aime: Towards Fully-Autonomous Multi-Agent Framework](https://mlsanity.com/paper/2507.11988/) (2025)
- [Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction](https://mlsanity.com/paper/2501.05925/) (2025)

## Built on by (most cited first, 30 of 59)

- [LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](https://mlsanity.com/paper/2510.17638/) (2025)
- [Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams](https://mlsanity.com/paper/2606.01770/) (2026)
- [Scaling Open-Ended Reasoning to Predict the Future](https://mlsanity.com/paper/2512.25070/) (2025)
- [Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning](https://mlsanity.com/paper/2512.19920/) (2025)
- [MiroFlow: Towards High-Performance and Robust Open-Source Agent Framework for General Deep Research Tasks](https://mlsanity.com/paper/2602.22808/) (2026)
- [Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond](https://mlsanity.com/paper/2604.22748/) (2026)
- [OneMillion-Bench: How Far are Language Agents from Human Experts?](https://mlsanity.com/paper/2603.07980/) (2026)
- [PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data](https://mlsanity.com/paper/2604.14199/) (2026)
- [Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs](https://mlsanity.com/paper/2604.18576/) (2026)
- [FutureX-Pro: Extending Future Prediction to High-Value Vertical Domains](https://mlsanity.com/paper/2601.12259/) (2026)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce](https://mlsanity.com/paper/2512.08868/) (2025)
- [FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs](https://mlsanity.com/paper/2601.13836/) (2026)
- [ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Education](https://mlsanity.com/paper/2512.05671/) (2025)
- [CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrency](https://mlsanity.com/paper/2512.00417/) (2025)
- [BenchBench: Benchmarking Automated Benchmark Generation](https://mlsanity.com/paper/2603.20807/) (2026)
- [FinDeepForecast: A Live Multi-Agent System for Benchmarking Deep Research Agents in Financial Forecasting](https://mlsanity.com/paper/2601.05039/) (2026)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents](https://mlsanity.com/paper/2602.14257/) (2026)
- [KellyBench: A Benchmark for Long-Horizon Sequential Decision Making](https://mlsanity.com/paper/2604.27865/) (2026)
- [Scientific reasoning does not reliably translate into scientific forecasting in frontier AI](https://mlsanity.com/paper/2605.22681/) (2026)
- [Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting](https://mlsanity.com/paper/2605.27668/) (2026)
- [AQuA: Recursively Self-Improving Quantitative Trading Research Agents](https://mlsanity.com/paper/2608.12841/) (2026)
- [Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking](https://mlsanity.com/paper/2511.18394/) (2025)
- [JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks](https://mlsanity.com/paper/2602.06486/) (2026)
- [Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecasting](https://mlsanity.com/paper/2605.27904/) (2026)
- [ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment](https://mlsanity.com/paper/2606.00644/) (2026)
- [Agentic Time Machine as an Infrastructure for Future-Event Forecasting](https://mlsanity.com/paper/2606.21013/) (2026)
- [Diversity is the Strength of the AI Crowd](https://mlsanity.com/paper/2606.29661/) (2026)
- [PriceSeer: Evaluating Large Language Models in Real-Time Stock Prediction](https://mlsanity.com/paper/2601.06088/) (2025)

## Abstract (arXiv)

Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce FutureX, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.