mlsanityData to 14 Oct 2026

Language agents achieve superhuman synthesis of scientific knowledge

arXiv 2409.13740 ↗First posted 10 Sep 2024cs.CL cs.AI cs.IRcited by 137

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Language models are known to hallucinate incorrect information, and it is unclear if they are sufficiently accurate and reliable for use in scientific research. We developed a rigorous human-AI comparison methodology to evaluate language model agents on real-world literature search tasks covering information retrieval, summarization, and contradiction detection tasks. We show that PaperQA2, a frontier language model agent optimized for improved factuality, matches or exceeds subject matter expert performance on three realistic literature research tasks without any restrictions on humans (i.e., full access to internet, search tools, and time). PaperQA2 writes cited, Wikipedia-style summaries of scientific topics that are significantly more accurate than existing, human-written Wikipedia articles. We also introduce a hard benchmark for scientific literature research called LitQA2 that guided design of PaperQA2, leading to it exceeding human performance. Finally, we apply PaperQA2 to identify contradictions within the scientific literature, an important scientific task that is challenging for humans. PaperQA2 identifies 2.34 +/- 1.99 contradictions per paper in a random subset of biology papers, of which 70% are validated by human experts. These results demonstrate that language model agents are now capable of exceeding domain experts across meaningful tasks on scientific literature.

Key facts. Cited by 137 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 50 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

26 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Research8 Dec

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  8. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  9. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  10. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  11. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 269
  12. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 242
13 more
  1. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 213
  2. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  3. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  4. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
  5. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  6. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 154
  7. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  8. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  9. 2023The Semantic Scholar Open Data Platformcited by 118
  10. 2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
  11. 2024SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluationcited by 24
  12. 2024The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approachescited by 18
  13. 2023The Semantic Reader Project: Augmenting Scholarly Documents through AI-Powered Interactive Reading Interfacescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 59; 2026: 74.

137 papers.

Sort:

By area:

  1. 2025Kosmos: An AI Scientist for Autonomous Discoverycited by 92
  2. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 89
  3. 2025Robin: A multi-agent system for automating scientific discoverycited by 62
  4. 2025LLM Agents for Education: Advances and Applicationscited by 59
  5. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  6. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  7. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  8. 2025TxGemma: Efficient and Agentic LLMs for Therapeuticscited by 41
  9. 2025CASCADE: Cumulative Agentic Skill Creation through Autonomous Development and Evolutioncited by 34
  10. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  11. 2025AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprisecited by 33
  12. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 32
38 more
  1. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 32
  2. 2025SciSciGPT: Advancing Human-AI Collaboration in the Science of Sciencecited by 27
  3. 2024Aviary: training language agents on challenging scientific taskscited by 26
  4. 2025Ai2 Scholar QA: Organized Literature Synthesis with Attributioncited by 25
  5. 2025MDCrow: Automating Molecular Dynamics Workflows with Large Language Modelscited by 22
  6. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 21
  7. 2025IRIS: Interactive Research Ideation System for Accelerating Scientific Discoverycited by 19
  8. 2025OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinkingcited by 18
  9. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 18
  10. 2025SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verificationcited by 17
  11. 2025How Far Are AI Scientists from Changing the World?cited by 15
  12. 2025PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literaturecited by 15
  13. 2025CellForge: Agentic Design of Virtual Cell Modelscited by 13
  14. 2025ScholarEval: Research Idea Evaluation Grounded in Literaturecited by 12
  15. 2025A Survey of AI Scientistscited by 11
  16. 2026AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discoverycited by 11
  17. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  18. 2024VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilitiescited by 10
  19. 2025How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?cited by 10
  20. 2026AI Agents Can Already Autonomously Perform Experimental High Energy Physicscited by 10
  21. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 10
  22. 2026NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?cited by 10
  23. 2025LLMs Outperform Experts on Challenging Biology Benchmarkscited by 9
  24. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 9
  25. 2025Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Modelscited by 9
  26. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  27. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  28. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  29. 2025TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agentscited by 8
  30. 2026SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directionscited by 8
  31. 2024Creativity in AI: Progresses and Challengescited by 7
  32. 2026EvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scalecited by 7
  33. 2025Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generationcited by 6
  34. 2025Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transitioncited by 6
  35. 2025DynaMate: An Autonomous Agent for Protein-Ligand Molecular Dynamics Simulationscited by 6
  36. 2026PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVRcited by 6
  37. 2025Wikipedia in the Era of LLMs: Evolution and Riskscited by 4
  38. 2025Operationalizing Serendipity: Multi-Agent AI Workflows for Enhanced Materials Characterization with Theory-in-the-Loopcited by 4

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
  2. 2026The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents
  3. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
  4. 2026Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning
  5. 2026TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs
  6. 2026SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agents
  7. 2024ResearchArena: Benchmarking Large Language Models' Ability to Collect and Organize Information as Research Agents
  8. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Research