mlsanityData to 7 Oct 2026

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

arXiv 2609.28876 ↗First posted 24 Sep 2026cs.AI cs.LGcited by 0

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 38 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

28 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning10 Jun
  2. 2026Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs20 Apr
  3. 2026FutureX-Pro: Extending Future Prediction to High-Value Vertical Domains18 Jan
  4. 2025FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction16 Aug
  5. 2025Bench to the Future: A Pastcasting Benchmark for Forecasting Agents11 Jun
  6. 2024MIRAI: Evaluating LLM Agents for Event Forecasting1 Jul

Other references · 22

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  6. 2024The Faiss librarycited by 370
  7. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  8. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  9. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  10. 2022Forecasting Future World Events with Neural Networkscited by 63
  11. 2020ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Datacited by 43
  12. 2025MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineeringcited by 38
10 more
  1. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 36
  2. 2024Consistency Checks for Language Model Forecasterscited by 27
  3. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 19
  4. 2026PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Datacited by 11
  5. 2026Prediction Arena: Benchmarking AI Models on Real-World Prediction Marketscited by 10
  6. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  7. 2025Do Language Models Update their Forecasts with New Information?cited by 2
  8. 2026Reinforcement Learning for LLM-based Event Forecastingcited by 2
  9. 2026Evaluating Strategic Reasoning in Forecasting Agentscited by 1
  10. 2026Can LLMs Take the Pulse of the Economy? A Real-Time Evaluation of LLM Nowcasts on Macroeconomic Indicatorscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Your Own Research: Learning to Forecast by Learning to Search
  2. 2026LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications
  3. 2026Agentic Time Machine as an Infrastructure for Future-Event Forecasting
  4. 2026From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting
  5. 2026TimeSeek: Temporal Reliability of Agentic Forecasters
  6. 2026Bridging the Last Mile of Time Series Forecasting with LLM Agents
  7. 2025The Forecast Critic: Leveraging Large Language Models for Poor Forecast Identification
  8. 2026TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems