mlsanityData to 14 Oct 2026

FutureSim: Replaying World Events to Evaluate Adaptive Agents

arXiv 2605.15188 ↗First posted 14 May 2026cs.LG cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently measure this capability for realistic use-cases, we propose building grounded simulations that replay real-world events in the order they occurred. We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arriving and questions resolving over the simulated period. We evaluate frontier agents in their native harness, testing their ability to predict world events over a three-month period from January to March 2026. FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all. Through careful ablations, we show how FutureSim offers a realistic setting to study emerging research directions like long-horizon test-time adaptation, search, memory, and reasoning about uncertainty. Overall, we hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

37 of 52 references.

References · 37

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2018World Modelscited by 1,513
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  7. 2024Genie: Generative Interactive Environmentscited by 747
  8. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  9. 2020Performative Predictioncited by 327
  10. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  11. 2018Towards Robust Evaluations of Continual Learningcited by 148
  12. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
25 more
  1. 2017Emergent Complexity via Multi-Agent Competitioncited by 120
  2. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  3. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 110
  4. 2024Approaching Human-Level Forecasting with Language Modelscited by 107
  5. 2025Recursive Language Modelscited by 104
  6. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 90
  7. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 88
  8. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 80
  9. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 77
  10. 2026AutoHarness: improving LLM agents by automatically synthesizing a code harnesscited by 76
  11. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  12. 2024Synthetic continued pretrainingcited by 60
  13. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 59
  14. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 54
  15. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 40
  16. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 40
  17. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 39
  18. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  19. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 33
  20. 2026Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgetscited by 30
  21. 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
  22. 2026textttYC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Executioncited by 11
  23. 2026Prediction Arena: Benchmarking AI Models on Real-World Prediction Marketscited by 10
  24. 2026Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefscited by 10
  25. 2026KellyBench: A Benchmark for Long-Horizon Sequential Decision Makingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economiescited by 8
  2. 2026Scientific reasoning does not reliably translate into scientific forecasting in frontier AIcited by 3
  3. 2026Agentic Time Machine as an Infrastructure for Future-Event Forecastingcited by 2
  4. 2026What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulnesscited by 2
  5. 2026Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasterscited by 1
  6. 2026Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agentscited by 0
  7. 2026ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Realitycited by 0