mlsanityData to 14 Oct 2026

ToolQA: A Dataset for LLM Question Answering with External Tools

arXiv 2306.13304 ↗First posted 23 Jun 2023cs.CL cs.AIcited by 274

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive performance in various NLP tasks, but they still suffer from challenges such as hallucination and weak numerical reasoning. To overcome these challenges, external tools can be used to enhance LLMs' question-answering abilities. However, current evaluation methods do not distinguish between questions that can be answered using LLMs' internal knowledge and those that require external information through tool use. To address this issue, we introduce a new dataset called ToolQA, which is designed to faithfully evaluate LLMs' ability to use external tools for question answering. Our development of ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions. Importantly, we strive to minimize the overlap between our benchmark data and LLMs' pre-training data, enabling a more precise evaluation of LLMs' tool-use reasoning abilities. We conducted an in-depth diagnosis of existing tool-use LLMs to highlight their strengths, weaknesses, and potential improvements. Our findings set a new benchmark for evaluating LLMs and suggest new directions for future advancements. Our data and code are freely available to the broader scientific community on GitHub.

Key facts. Cited by 274 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 63 of its 103 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

63 of 103 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  2. 2023ART: Automatic multi-step reasoning and tool-use for large language models16 Mar
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb
  4. 2022TALM: Tool Augmented Language Models24 May

Other references · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  13. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  14. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  15. 2023Teaching Large Language Models to Self-Debugcited by 988
  16. 2021Ethical and social risks of harm from Language Modelscited by 966
  17. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  18. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  20. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  21. 2022PAL: Program-aided Language Modelscited by 652
  22. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  23. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  24. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  25. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  26. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  27. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  28. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  29. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  30. 2022RealTime QA: What's the Answer Right Now?cited by 244
  31. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  32. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 196
  33. 2021A Dataset for Answering Time-Sensitive Questionscited by 154
  34. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  35. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  36. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  37. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  38. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 111; 2025: 106; 2026: 41.

274 papers.

Sort:

By area:

  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  2. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  5. 2024Large Language Models: A Surveycited by 525
  6. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  7. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  8. 2024Tool Learning with Large Language Models: A Surveycited by 223
  9. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  10. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  11. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
  12. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
38 more
  1. 2024Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphscited by 119
  2. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agentscited by 115
  3. 2024Large Language Models Can Learn Temporal Reasoningcited by 111
  4. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
  5. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  6. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  7. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  8. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 82
  9. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
  10. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 72
  11. 2025ACEBench: Who Wins the Match Point in Tool Usage?cited by 71
  12. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 70
  13. 2024MindSearch: Mimicking Human Minds Elicits Deep AI Searchercited by 66
  14. 2024LLM With Tools: A Surveycited by 61
  15. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 60
  16. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 59
  17. 2025LLM Agents for Education: Advances and Applicationscited by 59
  18. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  19. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 57
  20. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agentscited by 56
  21. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  22. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 54
  23. 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
  24. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Settingcited by 53
  25. 2024Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrievalcited by 52
  26. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 51
  27. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 50
  28. 2024GraphArena: Evaluating and Exploring Large Language Models on Graph Computationcited by 49
  29. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Modelscited by 48
  30. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 47
  31. 2024Current state of LLM Risks and AI Guardrailscited by 46
  32. 2023ToolTalk: Evaluating Tool-Usage in a Conversational Settingcited by 45
  33. 2026Skill Retrieval Augmentation for Agentic AIcited by 44
  34. 2024Learning to Ask: When LLM Agents Meet Unclear Instructioncited by 42
  35. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 42
  36. 2025When2Call: When (not) to Call Toolscited by 42
  37. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 41
  38. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMscited by 39

50 most cited of 274 All 274 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools
  2. 2023NLPBench: Evaluating Large Language Models on Solving NLP Problems
  3. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text
  4. 2025Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering
  5. 2025Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering
  6. 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
  7. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMs
  8. 2024Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions