# Evidence-Based Scientific Question Discovery: A Framework with Historical Backtesting

arXiv: https://arxiv.org/abs/2608.09968 · HTML: https://mlsanity.com/paper/2608.09968/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Builds on (most cited first, 10 of 10)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts](https://mlsanity.com/paper/2205.01833/) (2022)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [Construction of the Literature Graph in Semantic Scholar](https://mlsanity.com/paper/1805.02262/) (2018)

## Abstract (arXiv)

Current AI systems are optimized for answering questions; the scientific enterprise is bottlenecked earlier, at discovering the questions worth investigating. We present a framework that turns a traceable, reproducible, scope controlled research corpus into ranked, falsifiable research questions: evidence is represented as provenance carrying claims; cross paper tensions are detected, typed, and human adjudicated; surviving signals are refined into questions and ranked by a two stage protocol separating scientific priority from execution priority. We instantiate the framework on exoplanet atmospheres, a domain that uniquely combines literature, structured catalogs, and space telescope archives. In a historical backtest, all questions generated from evidence available before 2021 were substantively engaged by the 2021 to 2026 literature the sys?tem never saw: two were answered, including one whose premise the community later explicitly refuted and the top ranked question is independently posed and still open. These results sug?gest that systematic question discovery from evidence tensions surfaces the questions working scientists subsequently invest in.