mlsanityData to 7 Oct 2026

GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information

arXiv 2304.09667 ↗First posted 19 Apr 2023cs.CL cs.AI q-bio.GNcited by 115

Not on the map. Why?

Cited by 45 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While large language models (LLMs) have been successfully applied to various tasks, they still face challenges with hallucinations. Augmenting LLMs with domain-specific tools such as database utilities can facilitate easier and more precise access to specialized knowledge. In this paper, we present GeneGPT, a novel method for teaching LLMs to use the Web APIs of the National Center for Biotechnology Information (NCBI) for answering genomics questions. Specifically, we prompt Codex to solve the GeneTuring tests with NCBI Web APIs by in-context learning and an augmented decoding algorithm that can detect and execute API calls. Experimental results show that GeneGPT achieves state-of-the-art performance on eight tasks in the GeneTuring benchmark with an average score of 0.83, largely surpassing retrieval-augmented LLMs such as the new Bing (0.44), biomedical LLMs such as BioMedLM (0.08) and BioGPT (0.04), as well as GPT-3 (0.16) and ChatGPT (0.12). Our further analyses suggest that: (1) API demonstrations have good cross-task generalizability and are more useful than documentations for in-context learning; (2) GeneGPT can generalize to longer chains of API calls and answer multi-hop questions in GeneHop, a novel dataset introduced in this work; (3) Different types of errors are enriched in different tasks, providing valuable insights for future improvements.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 20 of its 41 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

20 of 41 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
8 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2022PAL: Program-aided Language Modelscited by 635
  3. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  4. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  5. 2023Augmented Language Models: a Surveycited by 398
  6. 2022TALM: Tool Augmented Language Modelscited by 197
  7. 2022Can large language models reason about medical questions?cited by 194
  8. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 49; 2025: 21; 2026: 23.

115 papers.

Sort:

By area:

  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  3. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  4. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  5. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  6. 2024Tool Learning with Large Language Models: A Surveycited by 221
  7. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  8. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  9. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  10. 2024Continual Learning for Large Language Models: A Surveycited by 144
  11. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  12. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
38 more
  1. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  2. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  3. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  4. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  5. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
  6. 2024I-Design: Personalized LLM Interior Designercited by 87
  7. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  8. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  9. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  10. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 75
  11. 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 71
  12. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 69
  13. 2023Matching Patients to Clinical Trials with Large Language Modelscited by 68
  14. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculumcited by 64
  15. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 64
  16. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmarkcited by 62
  17. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agentscited by 56
  18. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  19. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  20. 2024BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experimentscited by 49
  21. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 47
  22. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoningcited by 42
  23. 2025SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integrationcited by 40
  24. 2024Reducing Tool Hallucination via Reliability Alignmentcited by 37
  25. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  26. 2024Efficient Tool Use with Chain-of-Abstraction Reasoningcited by 33
  27. 2024Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosiscited by 23
  28. 2024Planning and Editing What You Retrieve for Enhanced Tool Learningcited by 21
  29. 2024Closing the gap between open-source and commercial large language models for medical evidence summarizationcited by 21
  30. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  31. 2025Alignment for Efficient Tool Calling of Large Language Modelscited by 17
  32. 2023PubMed and Beyond: Biomedical Literature Search in the Age of Artificial Intelligencecited by 16
  33. 2024ArgMed-Agents: Explainable Clinical Decision Reasoning with LLM Disscusion via Argumentation Schemescited by 13
  34. 2024Towards LifeSpan Cognitive Systemscited by 13
  35. 2023Gene Set Summarization using Large Language Modelscited by 12
  36. 2023Rational Decision-Making Agent with Internalized Utility Judgmentcited by 11
  37. 2024Budget-Constrained Tool Learning with Planningcited by 11
  38. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11

50 most cited of 115 All 115 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification
  2. 2024How do you know that? Teaching Generative Language Models to Reference Answers to Biomedical Questions
  3. 2025Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting
  4. 2024Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models
  5. 2024A Survey for Large Language Models in Biomedicine
  6. 2024Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical Machine Reading Comprehension
  7. 2024BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicine
  8. 2023Benchmarking large language models for biomedical natural language processing applications and recommendations