# Language agents achieve superhuman synthesis of scientific knowledge

arXiv: https://arxiv.org/abs/2409.13740 · HTML: https://mlsanity.com/paper/2409.13740/

## Key facts

- Cited by 137 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 10 Sep 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [Generation-Augmented Retrieval for Open-domain Question Answering](https://mlsanity.com/paper/2009.08553/) (2020)
- [MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning](https://mlsanity.com/paper/2205.00445/) (2022)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [The Semantic Scholar Open Data Platform](https://mlsanity.com/paper/2301.10140/) (2023)
- [Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking](https://mlsanity.com/paper/2310.13243/) (2023)
- [SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation](https://mlsanity.com/paper/2405.09939/) (2024)
- [The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches](https://mlsanity.com/paper/2406.03339/) (2024)
- [The Semantic Reader Project: Augmenting Scholarly Documents through AI-Powered Interactive Reading Interfaces](https://mlsanity.com/paper/2303.14334/) (2023)

## Built on by (most cited first, 30 of 137)

- [Kosmos: An AI Scientist for Autonomous Discovery](https://mlsanity.com/paper/2511.02824/) (2025)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [Robin: A multi-agent system for automating scientific discovery](https://mlsanity.com/paper/2505.13400/) (2025)
- [LLM Agents for Education: Advances and Applications](https://mlsanity.com/paper/2503.11733/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks](https://mlsanity.com/paper/2502.08586/) (2025)
- [ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics](https://mlsanity.com/paper/2509.00496/) (2025)
- [TxGemma: Efficient and Agentic LLMs for Therapeutics](https://mlsanity.com/paper/2504.06196/) (2025)
- [CASCADE: Cumulative Agentic Skill Creation through Autonomous Development and Evolution](https://mlsanity.com/paper/2512.23880/) (2025)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise](https://mlsanity.com/paper/2507.00310/) (2025)
- [Probing the limitations of multimodal language models for chemistry and materials research](https://mlsanity.com/paper/2411.16955/) (2024)
- [MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research](https://mlsanity.com/paper/2503.13399/) (2025)
- [SciSciGPT: Advancing Human-AI Collaboration in the Science of Science](https://mlsanity.com/paper/2504.05559/) (2025)
- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [Ai2 Scholar QA: Organized Literature Synthesis with Attribution](https://mlsanity.com/paper/2504.10861/) (2025)
- [MDCrow: Automating Molecular Dynamics Workflows with Large Language Models](https://mlsanity.com/paper/2502.09565/) (2025)
- [Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark](https://mlsanity.com/paper/2509.26574/) (2025)
- [IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery](https://mlsanity.com/paper/2504.16728/) (2025)
- [OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking](https://mlsanity.com/paper/2501.09751/) (2025)
- [RExBench: Can coding agents autonomously implement AI research extensions?](https://mlsanity.com/paper/2506.22598/) (2025)
- [SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification](https://mlsanity.com/paper/2506.15569/) (2025)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature](https://mlsanity.com/paper/2510.10909/) (2025)
- [CellForge: Agentic Design of Virtual Cell Models](https://mlsanity.com/paper/2508.02276/) (2025)
- [ScholarEval: Research Idea Evaluation Grounded in Literature](https://mlsanity.com/paper/2510.16234/) (2025)
- [A Survey of AI Scientists](https://mlsanity.com/paper/2510.23045/) (2025)
- [AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery](https://mlsanity.com/paper/2604.25256/) (2026)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities](https://mlsanity.com/paper/2412.18161/) (2024)

## Abstract (arXiv)

Language models are known to hallucinate incorrect information, and it is unclear if they are sufficiently accurate and reliable for use in scientific research. We developed a rigorous human-AI comparison methodology to evaluate language model agents on real-world literature search tasks covering information retrieval, summarization, and contradiction detection tasks. We show that PaperQA2, a frontier language model agent optimized for improved factuality, matches or exceeds subject matter expert performance on three realistic literature research tasks without any restrictions on humans (i.e., full access to internet, search tools, and time). PaperQA2 writes cited, Wikipedia-style summaries of scientific topics that are significantly more accurate than existing, human-written Wikipedia articles. We also introduce a hard benchmark for scientific literature research called LitQA2 that guided design of PaperQA2, leading to it exceeding human performance. Finally, we apply PaperQA2 to identify contradictions within the scientific literature, an important scientific task that is challenging for humans. PaperQA2 identifies 2.34 +/- 1.99 contradictions per paper in a random subset of biology papers, of which 70% are validated by human experts. These results demonstrate that language model agents are now capable of exceeding domain experts across meaningful tasks on scientific literature.