# AI Idea Bench 2025: AI Research Idea Generation Benchmark

arXiv: https://arxiv.org/abs/2504.14191 · HTML: https://mlsanity.com/paper/2504.14191/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2025.

## Built on by (most cited first, 30 of 40)

- [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025)
- [Creativity in LLM-based Multi-Agent Systems: A Survey](https://mlsanity.com/paper/2505.21116/) (2025)
- [ResearchGym: Evaluating Language Model Agents on Real-World AI Research](https://mlsanity.com/paper/2602.15112/) (2026)
- [SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents](https://mlsanity.com/paper/2505.23559/) (2025)
- [CellForge: Agentic Design of Virtual Cell Models](https://mlsanity.com/paper/2508.02276/) (2025)
- [A Survey of AI Scientists](https://mlsanity.com/paper/2510.23045/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility](https://mlsanity.com/paper/2605.16616/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [On the Limits of LLM-as-Judge for Scientific Novelty Assessment](https://mlsanity.com/paper/2606.12071/) (2026)
- [EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning](https://mlsanity.com/paper/2603.21728/) (2026)
- [Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists](https://mlsanity.com/paper/2604.28158/) (2026)
- [InnoGym: Benchmarking the Innovation Potential of AI Agents](https://mlsanity.com/paper/2512.01822/) (2025)
- [PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review](https://mlsanity.com/paper/2601.19916/) (2026)
- [AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents](https://mlsanity.com/paper/2602.14257/) (2026)
- [Evolving Idea Graphs with Learnable Edits-and-Commits for Multi-Agent Scientific Ideation](https://mlsanity.com/paper/2605.04922/) (2026)
- [ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes](https://mlsanity.com/paper/2607.04439/) (2026)
- [Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation](https://mlsanity.com/paper/2608.01666/) (2026)
- [LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation](https://mlsanity.com/paper/2608.13136/) (2026)
- [What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation](https://mlsanity.com/paper/2608.22948/) (2026)
- [SciDER: Scientific Data-centric End-to-end Researcher](https://mlsanity.com/paper/2603.01421/) (2026)
- [AgentPanel: Toward a New Paradigm for Human--AI Collaboration in Exploring Scientific Questions](https://mlsanity.com/paper/2608.03283/) (2026)
- [Tree-of-Ideas: Automated Research Ideation via Cross-Trajectory Reasoning over Scholarly Evolution](https://mlsanity.com/paper/2608.10740/) (2026)
- [Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies](https://mlsanity.com/paper/2608.16645/) (2026)
- [Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment](https://mlsanity.com/paper/2608.29696/) (2026)
- [DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation](https://mlsanity.com/paper/2609.22104/) (2026)
- [MindFlow: Mind Supernet Powered Thinking Flows for Research Idea Innovation](https://mlsanity.com/paper/2610.11966/) (2026)
- [Universe of Thoughts: Enabling Creative Reasoning with Large Language Models](https://mlsanity.com/paper/2511.20471/) (2025)
- [Graph Your Way to Inspiration: Integrating Co-Author Graphs with Retrieval-Augmented Generation for Large Language Model Based Scientific Idea Generation](https://mlsanity.com/paper/2602.22215/) (2025)

## Abstract (arXiv)

Large-scale Language Models (LLMs) have revolutionized human-AI interaction and achieved significant success in the generation of novel ideas. However, current assessments of idea generation overlook crucial factors such as knowledge leakage in LLMs, the absence of open-ended benchmarks with grounded truth, and the limited scope of feasibility analysis constrained by prompt design. These limitations hinder the potential of uncovering groundbreaking research ideas. In this paper, we present AI Idea Bench 2025, a framework designed to quantitatively evaluate and compare the ideas generated by LLMs within the domain of AI research from diverse perspectives. The framework comprises a comprehensive dataset of 3,495 AI papers and their associated inspired works, along with a robust evaluation methodology. This evaluation system gauges idea quality in two dimensions: alignment with the ground-truth content of the original papers and judgment based on general reference material. AI Idea Bench 2025's benchmarking system stands to be an invaluable resource for assessing and comparing idea-generation techniques, thereby facilitating the automation of scientific discovery.