# Can Large Language Models Forecast What Researchers Study Next?

arXiv: https://arxiv.org/abs/2609.00747 · HTML: https://mlsanity.com/paper/2609.00747/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents](https://mlsanity.com/paper/2602.02474/) (2026)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025)
- [Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution](https://mlsanity.com/paper/2512.10696/) (2025)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas](https://mlsanity.com/paper/2506.20803/) (2025)
- [CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation](https://mlsanity.com/paper/2503.22708/) (2025)
- [ResearchTown: Simulator of Human Research Community](https://mlsanity.com/paper/2412.17767/) (2024)
- [HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation](https://mlsanity.com/paper/2504.11524/) (2025)
- [Quantifying Long-Term Scientific Impact](https://mlsanity.com/paper/1306.3293/) (2013)
- [IdeaBench: Benchmarking Large Language Models for Research Idea Generation](https://mlsanity.com/paper/2411.02429/) (2024)
- [Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition](https://mlsanity.com/paper/2412.13612/) (2024)
- [The Ramon Llull's Thinking Machine for Automated Ideation](https://mlsanity.com/paper/2508.19200/) (2025)
- [PreScience: A Dataset and Benchmark for Scientific Forecasting](https://mlsanity.com/paper/2602.20459/) (2026)
- [Scientific reasoning does not reliably translate into scientific forecasting in frontier AI](https://mlsanity.com/paper/2605.22681/) (2026)

## Built on by (most cited first, 2 of 2)

- [Science Utopia? Closed-Loop LLM Simulation of Academic Research Ecosystems](https://mlsanity.com/paper/2610.01257/) (2026)
- [Priced Guidance: Can Language Models Generate Future Research Ideas?](https://mlsanity.com/paper/2610.04976/) (2026)

## Abstract (arXiv)

Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they anticipate subsequent work. We introduce IdeaForecastBench to evaluate research idea forecasting. Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers. The benchmark comprises 624 rolling episodes across 52 topics, with a fixed retrieve-then-judge protocol and separately reported results from two judges. We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF). Under the primary GPT-4.1-mini judge, Summary improves on Direct in Hit@5 and Precision@5 across all four backbones. Qwen2.5 scores above GPT-4.1, whereas Qwen3.5 scores below it. An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage. Threshold and judge diagnostics further clarify the limits of interpreting realization as precise anticipation. IdeaForecastBench provides a common task for studying which research ideas a community subsequently pursues and how reliably this outcome can be measured.