mlsanityData to 14 Oct 2026

Scientific reasoning does not reliably translate into scientific forecasting in frontier AI

arXiv 2605.22681 ↗First posted 21 May 2026cs.AIcited by 3

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

AI systems are increasingly used to support forward-looking scientific judgment, but it remains unclear whether they can form reliable expectations about future scientific advances. Here we show that strong scientific reasoning does not reliably translate into accurate forecasting of future scientific advances. To study this question, we introduce CUSP, a temporally grounded evaluation suite for event-level scientific forecasting across eight scientific disciplines. Across six frontier AI models, we observe a striking asymmetry in forecasting performance together with systematic error patterns. Models often identify plausible mechanisms underlying future scientific advances, yet perform near chance on feasibility assessment, generate solution strategies that only weakly align with realized advances, and systematically predict scientific advances later than they become publicly observable. Providing additional pre-cutoff scientific knowledge improves performance but does not eliminate these forecasting limitations. These findings suggest that current AI systems possess substantial retrospective scientific competence but limited forward-looking predictive capability. Scientific forecasting should therefore be evaluated as a complementary dimension of AI scientific capability when deploying AI systems for research prioritization and scientific decision-making.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 59 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

17 of 59 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  5. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  6. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 422
  7. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  8. 2025Kosmos: An AI Scientist for Autonomous Discoverycited by 92
  9. 2025ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decompositioncited by 66
  10. 2025FutureX: An Advanced Live Benchmark for LLM Agents in Future Predictioncited by 59
  11. 2026InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fieldscited by 23
  12. 2024International Scientific Report on the Safety of Advanced AI (Interim Report)cited by 19
5 more
  1. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 19
  2. 2025PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimationcited by 8
  3. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  4. 2025AI for Scientific Discovery is a Social Problemcited by 3
  5. 2026PreScience: A Dataset and Benchmark for Scientific Forecastingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Tasks
  2. 2026Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation
  3. 2026Contemporary AI lacks the imagination to diverge or negate in science
  4. 2026AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
  5. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
  6. 2026SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?
  7. 2024Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges
  8. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents