# Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders

arXiv: https://arxiv.org/abs/2405.17044 · HTML: https://mlsanity.com/paper/2405.17044/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 89 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts](https://mlsanity.com/paper/2205.01833/) (2022)
- [Generative artificial intelligence enhances creativity but reduces the diversity of novel content](https://mlsanity.com/paper/2312.00506/) (2023)
- [The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4](https://mlsanity.com/paper/2311.07361/) (2023)
- [Simplicial Closure and higher-order link prediction](https://mlsanity.com/paper/1802.06916/) (2018)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [On scientific understanding with artificial intelligence](https://mlsanity.com/paper/2204.01467/) (2022)
- [Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2303.13948/) (2023)
- [Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy](https://mlsanity.com/paper/2402.04247/) (2024)
- [Predicting the Future of AI with AI: High-quality link prediction in an exponentially growing knowledge network](https://mlsanity.com/paper/2210.00881/) (2022)
- [Accelerating science with human-aware artificial intelligence](https://mlsanity.com/paper/2306.01495/) (2023)
- [Predicting Research Trends with Semantic and Neural Networks with an application in Quantum Physics](https://mlsanity.com/paper/1906.06843/) (2019)
- [Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study](https://mlsanity.com/paper/2106.09700/) (2021)
- [A Computational Inflection for Scientific Discovery](https://mlsanity.com/paper/2205.02007/) (2022)
- [Forecasting high-impact research topics via machine learning on evolving knowledge graphs](https://mlsanity.com/paper/2402.08640/) (2024)
- [AGATHA: Automatic Graph-mining And Transformer based Hypothesis generation Approach](https://mlsanity.com/paper/2002.05635/) (2020)
- [The CSO Classifier: Ontology-Driven Detection of Research Topics in Scholarly Articles](https://mlsanity.com/paper/2104.00948/) (2021)
- [Towards a Benchmark for Scientific Understanding in Humans and Machines](https://mlsanity.com/paper/2304.10327/) (2023)
- [Principles of scientific research team formation and evolution](https://mlsanity.com/paper/1403.2787/) (2014)
- [Extracting human interpretable structure-property relationships in chemistry using XAI and large language models](https://mlsanity.com/paper/2311.04047/) (2023)

## Built on by (most cited first, 22 of 22)

- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context](https://mlsanity.com/paper/2412.17596/) (2024)
- [PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors](https://mlsanity.com/paper/2507.15550/) (2025)
- [Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Models](https://mlsanity.com/paper/2501.05382/) (2025)
- [Predicting New Research Directions in Materials Science using Large Language Models and Concept Graphs](https://mlsanity.com/paper/2506.16824/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings](https://mlsanity.com/paper/2503.19257/) (2025)
- [Towards autonomous quantum physics research using LLM agents with access to intelligent tools](https://mlsanity.com/paper/2511.11752/) (2025)
- [EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning](https://mlsanity.com/paper/2603.21728/) (2026)
- [Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists](https://mlsanity.com/paper/2604.28158/) (2026)
- [Transformational Creativity in Science: A Graphical Theory](https://mlsanity.com/paper/2504.18687/) (2025)
- [ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes](https://mlsanity.com/paper/2607.04439/) (2026)
- [How do Humans and Language Models Reason About Creativity? A Comparative Analysis](https://mlsanity.com/paper/2502.03253/) (2025)
- [Active Inference AI Systems for Scientific Discovery](https://mlsanity.com/paper/2506.21329/) (2025)
- [Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems](https://mlsanity.com/paper/2603.26718/) (2026)
- [What's In Your Field? Mapping Scientific Research with Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2503.09894/) (2025)
- [AINL-Eval 2025 Shared Task: Detection of AI-Generated Scientific Abstracts in Russian](https://mlsanity.com/paper/2508.09622/) (2025)
- [Galton's Law of Mediocrity: Why Large Language Models Regress to the Mean and Fail at Creativity in Advertising](https://mlsanity.com/paper/2509.25767/) (2025)
- [GRAFT: Graph-Distilled Generative Retrieval for Facet-Aware Scientific Literature Exploration](https://mlsanity.com/paper/2608.22381/) (2026)

## Abstract (arXiv)

The rapid growth of scientific literature makes it increasingly challenging for researchers to identify novel and impactful ideas, especially across disciplines. Modern artificial intelligence (AI) systems offer new opportunities for scientific ideation, but how compelling are AI-generated ideas, and how can their quality be improved? Here, we introduce SciMuse, which generates personalized research ideas using a knowledge graph of 58 million papers and a large language model (LLM). A central focus of this work is to understand how interesting these ideas are. Therefore, we conducted a large-scale evaluation in which more than 100 research group leaders -- spanning the natural sciences to the humanities -- rated over 4,400 personalized ideas according to their level of interest. Overall, expert ratings were modest (mean 2.40 on a 5-point scale, most common rating 1), while 24.9% of ideas were rated 4 or 5. We find that supplying concept pairs selected using the knowledge graph does not improve expert-rated interest over a titles-only GPT baseline. High-citation-predicted pairs even showed a weak tendency (1.94σ) toward lower interest than random pairs. Nevertheless, graph features can be used to control properties of ideas, and, using this unique evaluation dataset, we show that idea interest can be predicted with both a supervised neural network based on graph features and a zero-shot ranking approach based on an LLM. Our work provides an AI methodology for generating scientific ideas and a large-scale interdisciplinary expert evaluation, paving the way to study and improve difficult-to-measure metrics such as expert-perceived scientific interestingness.