# Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation

arXiv: https://arxiv.org/abs/2502.05151 · HTML: https://mlsanity.com/paper/2502.05151/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 158 of its 335 references that are in the map.
- First posted on arXiv 7 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 158)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement](https://mlsanity.com/paper/2402.14658/) (2024)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)

## Built on by (most cited first, 30 of 42)

- [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025)
- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Step-by-Step Fact Verification System for Medical Claims with Explainable Reasoning](https://mlsanity.com/paper/2502.14765/) (2025)
- [CellForge: Agentic Design of Virtual Cell Models](https://mlsanity.com/paper/2508.02276/) (2025)
- [PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing](https://mlsanity.com/paper/2604.05018/) (2026)
- [LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answering](https://mlsanity.com/paper/2505.05423/) (2025)
- [LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models](https://mlsanity.com/paper/2505.19240/) (2025)
- [Exploring Design of Multi-Agent LLM Dialogues for Research Ideation](https://mlsanity.com/paper/2507.08350/) (2025)
- [Real Money, Fake Models: Deceptive Model Claims in Shadow APIs](https://mlsanity.com/paper/2603.01919/) (2026)
- [TikZilla: Scaling Text-to-TikZ with High-Quality Data and Reinforcement Learning](https://mlsanity.com/paper/2603.03072/) (2026)
- [Large Language Models Penetration in Scholarly Writing and Peer Review](https://mlsanity.com/paper/2502.11193/) (2025)
- [Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey](https://mlsanity.com/paper/2505.03418/) (2025)
- [From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Models](https://mlsanity.com/paper/2505.21935/) (2025)
- [Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning](https://mlsanity.com/paper/2510.19733/) (2025)
- [PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies](https://mlsanity.com/paper/2510.16505/) (2025)
- [ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond](https://mlsanity.com/paper/2608.14354/) (2026)
- [Agents of Discovery](https://mlsanity.com/paper/2509.08535/) (2025)
- [CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research](https://mlsanity.com/paper/2510.11985/) (2025)
- [On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?](https://mlsanity.com/paper/2510.20810/) (2025)
- [QuantumBench: A Benchmark for Quantum Problem Solving](https://mlsanity.com/paper/2511.00092/) (2025)
- [AutoPR: Let's Automate Your Academic Promotion!](https://mlsanity.com/paper/2510.09558/) (2025)
- [ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows](https://mlsanity.com/paper/2510.20279/) (2025)
- [What Understanding Means in AI-Laden Astronomy](https://mlsanity.com/paper/2601.10038/) (2026)
- [Enhancing Study-Level Inference from Clinical Trial Papers via Reinforcement Learning-Based Numeric Reasoning](https://mlsanity.com/paper/2505.22928/) (2025)
- [Generative AI and the future of scientometrics: current topics and future questions](https://mlsanity.com/paper/2507.00783/) (2025)
- [A unified multimodal understanding and generation model for cross-disciplinary scientific research](https://mlsanity.com/paper/2601.01363/) (2026)
- [FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research](https://mlsanity.com/paper/2602.14285/) (2026)
- [Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers](https://mlsanity.com/paper/2603.25638/) (2026)
- [Scientific discovery as meta-optimization: a combinatorial optimization case study](https://mlsanity.com/paper/2606.26728/) (2026)

## Abstract (arXiv)

With the advent of large multimodal language models, science is now at a threshold of an AI-based technological transformation. An emerging ecosystem of models and tools aims to support researchers throughout the scientific lifecycle, including (1) searching for relevant literature, (2) generating research ideas and conducting experiments, (3) producing text-based content, (4) creating multimodal artifacts such as figures and diagrams, and (5) evaluating scientific work, as in peer review. In this survey, we provide a curated overview of literature representative of the core techniques, evaluation practices, and emerging trends in AI-assisted scientific discovery. Across the five tasks outlined above, we discuss datasets, methods, results, evaluation strategies, limitations, and ethical concerns, including risks to research integrity through the misuse of generative models. We aim for this survey to serve both as an accessible, structured orientation for newcomers to the field, as well as a catalyst for new AI-based initiatives and their integration into future ``AI4Science'' systems.