mlsanityData to 7 Oct 2026

Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

arXiv 2608.16645 ↗First posted 17 Aug 2026cs.AI cs.CL cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

21 of 31 references.

References · 21

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  4. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  6. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  7. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 375
  8. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  9. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  10. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agentscited by 94
  11. 2024MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypothesescited by 68
  12. 2025ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decompositioncited by 57
9 more
  1. 2025How much do language models memorize?cited by 53
  2. 2025The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideascited by 49
  3. 2025AI Idea Bench 2025: AI Research Idea Generation Benchmarkcited by 33
  4. 2026Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideascited by 8
  5. 2026HindSight: Evaluating LLM-Generated Research Ideas via Future Impactcited by 8
  6. 2026Measuring the Gap Between Human and LLM Research Ideascited by 7
  7. 2025Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Networkcited by 5
  8. 2015On the ranking of a Swiss system chess team tournamentcited by 4
  9. 2026ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomescited by 2

Built on by

Not cited yet.