mlsanityData to 7 Oct 2026

Gorilla: Large Language Model Connected with Massive APIs

arXiv 2305.15334 ↗First posted 24 May 2023cs.CL cs.AIcited by 1,302

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call. We release Gorilla, a finetuned LLaMA-based model that surpasses the performance of GPT-4 on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time document changes, enabling flexible user updates or version changes. It also substantially mitigates the issue of hallucination, commonly encountered when prompting LLMs directly. To evaluate the model's ability, we introduce APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and TensorHub APIs. The successful integration of the retrieval system with Gorilla demonstrates the potential for LLMs to use tools more accurately, keep up with frequently updated documentation, and consequently increase the reliability and applicability of their outputs. Gorilla's code, model, data, and demo are available at https://gorilla.cs.berkeley.edu

Key facts. Cited by 1,302 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 43 of its 53 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

43 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  2. 2022PAL: Program-aided Language Models18 Nov
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
28 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  13. 2023StarCoder: may the source be with you!cited by 790
  14. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  15. 2023Language Models can Solve Computer Taskscited by 371
  16. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  17. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  18. 2020Unsupervised Translation of Programming Languagescited by 262
  19. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  20. 2021Internet-Augmented Dialogue Generationcited by 217
  21. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  22. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  23. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  24. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  25. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  26. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  27. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  28. 2019Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehensioncited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 96; 2024: 256; 2025: 324; 2026: 626.

1,302 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  8. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  11. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  12. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
38 more
  1. 2024Large Language Models: A Surveycited by 506
  2. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  3. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  4. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  5. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  6. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  7. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  8. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  9. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  10. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  11. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  12. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 329
  13. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  14. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  15. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  16. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  17. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  18. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  19. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  20. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  21. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  22. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  23. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  24. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  25. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  26. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  27. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  28. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repaircited by 225
  29. 2024Tool Learning with Large Language Models: A Surveycited by 221
  30. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  31. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  32. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  33. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  34. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  35. 2025Survey on Evaluation of LLM-based Agentscited by 190
  36. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  37. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  38. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182

50 most cited of 1,302 All 1,302 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
  2. 2024Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation
  3. 2024An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks
  4. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
  5. 2024Large Language Models (LLMs): Deployment, Tokenomics and Sustainability
  6. 2024Octopus: On-device language model for function calling of software APIs
  7. 2023GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond
  8. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact