mlsanityData to 14 Oct 2026

Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents

arXiv 2410.13185 ↗First posted 17 Oct 2024cs.AI cs.CLcited by 97

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Effective research ideation is a critical step for scientific research. However, the exponential increase in scientific literature makes it challenging for researchers to stay current with recent advances and identify meaningful research directions. Recent developments in large language models~(LLMs) suggest a promising avenue for automating the generation of novel research ideas. However, existing methods for idea generation either trivially prompt LLMs or directly expose LLMs to extensive literature without indicating useful information. Inspired by the research process of human researchers, we propose a Chain-of-Ideas~(CoI) agent, an LLM-based agent that organizes relevant literature in a chain structure to effectively mirror the progressive development in a research domain. This organization facilitates LLMs to capture the current advancements in research, thereby enhancing their ideation capabilities. Furthermore, we propose Idea Arena, an evaluation protocol that can comprehensively evaluate idea generation methods from different perspectives, aligning closely with the preferences of human researchers. Experimental results indicate that the CoI agent consistently outperforms other methods and shows comparable quality as humans in research idea generation. Moreover, our CoI agent is budget-friendly, with a minimum cost of 0.50 to generate a candidate idea and its corresponding experimental design.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 41 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

36 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning9 Sep
  2. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers6 Sep
  3. 2024MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents26 Aug
  4. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation5 Oct

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2024Qwen2 Technical Reportcited by 2,120
  12. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
20 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  3. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  4. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  5. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  6. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  7. 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 226
  8. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 210
  9. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 179
  10. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  11. 2023Emergent autonomous scientific research capabilities of large language modelscited by 132
  12. 2023Agents: An Open-source Framework for Autonomous Language Agentscited by 116
  13. 2024Symbolic Learning Enables Self-Evolving Agentscited by 106
  14. 2023Large Language Models for Automated Open-domain Scientific Hypotheses Discoverycited by 100
  15. 2023Large Language Models are Zero Shot Hypothesis Proposerscited by 45
  16. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 42
  17. 2024Can Large Language Models Unlock Novel Scientific Research Ideas?cited by 24
  18. 2021Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attentioncited by 20
  19. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
  20. 2024Prompting Diverse Ideas: Increasing AI Idea Variancecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 46; 2026: 51.

97 papers.

Sort:

By area:

  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  3. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  4. 2025AI-Researcher: Autonomous Scientific Innovationcited by 100
  5. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  6. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  7. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  8. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 70
  9. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 64
  10. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  11. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  12. 2025Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?cited by 32
38 more
  1. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 32
  2. 2025AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineagecited by 32
  3. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  4. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 28
  5. 2025Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedbackcited by 25
  6. 2025DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Taskscited by 24
  7. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Researchcited by 22
  8. 2025AI Scientists Fail Without Strong Implementation Capabilitycited by 21
  9. 2025IRIS: Interactive Research Ideation System for Accelerating Scientific Discoverycited by 19
  10. 2025Literature-Grounded Novelty Assessment of Scientific Ideascited by 19
  11. 2025All That Glitters is Not Novel: Plagiarism in AI Generated Researchcited by 18
  12. 2025How Far Are AI Scientists from Changing the World?cited by 15
  13. 2026Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attemptscited by 13
  14. 2025Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedbackcited by 12
  15. 2025MIR: Methodology Inspiration Retrieval for Scientific Research Problemscited by 11
  16. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  17. 2025HypER: Literature-grounded Hypothesis Generation and Distillation with Provenancecited by 10
  18. 2026Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideascited by 10
  19. 2025Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Modelscited by 9
  20. 2025LLM-REVal: Can We Trust LLM Reviewers Yet?cited by 9
  21. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  22. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  23. 2025GenoMAS: A Multi-Agent Framework for Scientific Discovery via Code-Driven Gene Expression Analysiscited by 8
  24. 2025PosterForest: Hierarchical Multi-Agent Collaboration for Scientific Poster Generationcited by 8
  25. 2026InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problemcited by 8
  26. 2026Learning to Predict Future-Aligned Research Proposals with Language Modelscited by 8
  27. 2026GIANTS: Generative Insight Anticipation from Scientific Literaturecited by 8
  28. 2025XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaborationcited by 7
  29. 2025GoAI: Enhancing AI Students' Learning Paths and Idea Generation via Graph of AI Ideascited by 6
  30. 2026Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientistscited by 6
  31. 2026Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproductioncited by 6
  32. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 5
  33. 2025SurveyGen-I: Consistent Scientific Survey Generation with Evolving Plans and Memory-Guided Writingcited by 5
  34. 2025MotivGraph-SoIQ: Integrating Motivational Knowledge Graphs and Socratic Dialogue for Enhanced LLM Ideationcited by 5
  35. 2026HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Executioncited by 5
  36. 2025Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Optionscited by 3
  37. 2025Augmenting Research Ideation with Data: An Empirical Investigation in Social Sciencecited by 3
  38. 2025Agent Ideate: A Framework for Product Idea Generation from Patents Using Agentic AIcited by 3

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Design of Multi-Agent LLM Dialogues for Research Ideation
  2. 2026Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
  3. 2025Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network
  4. 2025Idea2Plan: Exploring AI-Powered Research Planning
  5. 2026IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas
  6. 2026Enhancing Research Idea Generation through Combinatorial Innovation and Multi-Agent Iterative Search Strategies
  7. 2025A Vision for Auto Research with LLM Agents
  8. 2025The Budget AI Researcher and the Power of RAG Chains