mlsanityData to 7 Oct 2026

LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena

arXiv 2510.17638 ↗First posted 20 Oct 2025cs.AI cs.CL cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Forecasting is not only a fundamental intellectual pursuit but also is of significant importance to societal systems such as finance and economics. With the rapid advances of large language models (LLMs) trained on Internet-scale data, it raises the promise of employing LLMs to forecast real-world future events, an emerging paradigm we call "LLM-as-a-Prophet". This paper systematically investigates such predictive intelligence of LLMs. To this end, we build Prophet Arena, a general evaluation benchmark that continuously collects live forecasting tasks and decomposes each task into distinct pipeline stages, in order to support our controlled and large-scale experimentation. Our comprehensive evaluation reveals that many LLMs already exhibit impressive forecasting capabilities, reflected in, e.g., their small calibration errors, consistent prediction confidence and promising market returns. However, we also uncover key bottlenecks towards achieving superior predictive intelligence via LLM-as-a-Prophet, such as LLMs' inaccurate event recalls, misunderstanding of data sources and slower information aggregation compared to markets when resolution nears.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 27 of its 81 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

27 of 81 references.

References · 27

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  9. 2025Kimi K2: Open Agentic Intelligencecited by 523
  10. 2020Time Series Forecasting With Deep Learning: A Surveycited by 457
  11. 2022Conformal prediction beyond exchangeabilitycited by 348
  12. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
15 more
  1. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  2. 2023Calibrated Language Models Must Hallucinatecited by 114
  3. 2025Confidence Improves Self-Consistency in LLMscited by 111
  4. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  5. 2022Forecasting Future World Events with Neural Networkscited by 63
  6. 2025FutureX: An Advanced Live Benchmark for LLM Agents in Future Predictioncited by 55
  7. 2020ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Datacited by 43
  8. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 40
  9. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 40
  10. 2024Consistency Checks for Language Model Forecasterscited by 27
  11. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
  12. 2024Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understandingcited by 23
  13. 2025Bench to the Future: A Pastcasting Benchmark for Forecasting Agentscited by 20
  14. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  15. 2024OpenEP: Open-Ended Future Event Predictioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 36.

36 papers.

Sort:

By area:

  1. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
  2. 2026PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Datacited by 11
  3. 2026Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefscited by 9
  4. 2026Live-Evo: Online Evolution of Agentic Memory from Continuous Feedbackcited by 8
  5. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  6. 2026KellyBench: A Benchmark for Long-Horizon Sequential Decision Makingcited by 3
  7. 2026Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecastingcited by 3
  8. 2026WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoningcited by 2
  9. 2026Agentic Time Machine as an Infrastructure for Future-Event Forecastingcited by 2
  10. 2026Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Modelscited by 2
  11. 2026TruthTensor: Evaluating LLMs through Human Imitation on Prediction Market under Drift and Holistic Reasoningcited by 1
  12. 2026TimeSeek: Temporal Reliability of Agentic Forecasterscited by 1
24 more
  1. 2026Evaluating Strategic Reasoning in Forecasting Agentscited by 1
  2. 2026Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasterscited by 1
  3. 2026LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sportscited by 1
  4. 2026LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecastingcited by 1
  5. 2026Adaptive Contracts for Cost-Effective AI Delegationcited by 0
  6. 2026LEAF: A Living Benchmark for Event-Augmented Forecastingcited by 0
  7. 2026PolyGnosis 2.0: Enhancing LLM Reasoning via Agentic Harness Engineering for Polymarket and OSINT Insight Extractioncited by 0
  8. 2026ForecastCompass: Guiding Agentic Forecasting with Adaptive Factor Memorycited by 0
  9. 2026GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimizationcited by 0
  10. 2026From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocolcited by 0
  11. 2026Beyond Forecasting: The Belief-to-Trade Layer in Prediction-Market Agentscited by 0
  12. 2026Decentralized Aggregation of LLM Predictions via Wagering Mechanismscited by 0
  13. 2026When do prophets profit in prediction markets?cited by 0
  14. 2026Robust Human-AI Complementarity under Uncertaintycited by 0
  15. 2026Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasterscited by 0
  16. 2026Distilling Temporal Search and Reasoning: Evolving LLMs for Future Prediction via Harness-Assisted Efficient Data Synthesiscited by 0
  17. 2026ForeDreamer: A Self-Evolving Dual-Agent Memory Architecture for Future Event Predictioncited by 0
  18. 2026LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applicationscited by 0
  19. 2026Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessmentcited by 0
  20. 2026From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecastingcited by 0
  21. 2026Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agentscited by 0
  22. 2026Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecastscited by 0
  23. 2026CruxBench: A Benchmark of Information Discoverycited by 0
  24. 2026ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Realitycited by 0