mlsanityData to 14 Oct 2026

Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator

arXiv 2507.11810 ↗First posted 16 Jul 2025cs.DL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used in scientific research and discovery, supporting tasks ranging from literature retrieval and synthesis to hypothesis generation, autonomous experimentation, and research evaluation. Existing surveys often conflate scientific research with scientific discovery and typically organize systems by domain, task, or autonomy level alone. In this survey, we propose a four-role framework for understanding LLMs in scientific innovation: Assistant, Collaborator, Scientist, and Evaluator. The framework integrates three complementary dimensions: autonomy level, cognitive function, and scientific innovation, to distinguish research-oriented support from frontier-oriented discovery. We review representative methods, benchmarks, and evaluation practices for each role, examining their capabilities, limitations, and human oversight requirements. Across the literature, Assistant systems are comparatively mature in retrieval and synthesis but remain unreliable in open-ended applications; Collaborator systems expand the space of candidate hypotheses yet struggle with novelty-grounding trade-offs; Scientist systems increasingly automate research workflows but face reliability and safety bottlenecks; and Evaluator systems support review and verification while remaining weak in novelty assessment. We argue that progress in AI for science depends not only on model capability, but also on evaluation, oversight, accountability, and institutional integration.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 120 of its 207 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

120 of 207 references.

References · 120

  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  3. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  4. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  5. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  6. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  7. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  8. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 422
  9. 2022Multimodal Learning with Transformers: A Surveycited by 400
  10. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  11. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  12. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 235
38 more
  1. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 211
  2. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 188
  3. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 183
  4. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 179
  5. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  6. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  7. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
  8. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  9. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  10. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  11. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  12. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 133
  13. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
  14. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  15. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  16. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  17. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  18. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 109
  19. 2023Large Language Models for Automated Open-domain Scientific Hypotheses Discoverycited by 100
  20. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  21. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
  22. 2025AI-Researcher: Autonomous Scientific Innovationcited by 100
  23. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  24. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  25. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 93
  26. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  27. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  28. 2024LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discoverycited by 86
  29. 2025LLM4SR: A Survey on Large Language Models for Scientific Researchcited by 83
  30. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  31. 2024MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypothesescited by 72
  32. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  33. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
  34. 2024The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Ratescited by 69
  35. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  36. 2024Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent Systemcited by 68
  37. 2024MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agentscited by 67
  38. 2025PaSa: An LLM Agent for Comprehensive Academic Paper Searchcited by 64

50 most cited of 120 All 120 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.