mlsanityData to 7 Oct 2026

Reducing Tool Hallucination via Reliability Alignment

arXiv 2412.04141 ↗First posted 5 Dec 2024cs.CLcited by 37

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Large Language Models (LLMs) have expanded their capabilities beyond language generation to interact with external tools, enabling automation and real-world applications. However, tool hallucinations, where models either select inappropriate tools or misuse them, pose significant challenges, leading to erroneous task execution, increased computational costs, and reduced system reliability. To systematically address this issue, we define and categorize tool hallucinations into two main types, tool selection hallucination and tool usage hallucination. To evaluate and mitigate these issues, we introduce RelyToolBench, which integrates specialized test cases and novel metrics to assess hallucination-aware task success and efficiency. Finally, we propose Relign, a reliability alignment framework that expands the tool-use action space to include indecisive actions, allowing LLMs to defer tool use, seek clarification, or adjust tool selection dynamically. Through extensive experiments, we demonstrate that Relign significantly reduces tool hallucinations, improves task reliability, and enhances the efficiency of LLM tool interactions.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 30 of its 41 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

30 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models12 Mar
  2. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  3. 2023Large Language Models as Tool Makers26 May
  4. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  5. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
12 more
  1. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  2. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  3. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  4. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  5. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
  6. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  7. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  8. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  9. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  10. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  11. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
  12. 2024On Mitigating Code LLM Hallucinations with API Documentationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 22.

37 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  3. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  4. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 19
  5. 2025Alignment for Efficient Tool Calling of Large Language Modelscited by 17
  6. 2026From Storage to Steering: Memory Control Flow Attacks on LLM Agentscited by 17
  7. 2025Thinking with Visual Abstract: Enhancing Multimodal Reasoning via Visual Abstractioncited by 15
  8. 2026A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)cited by 15
  9. 2025Enhancing LLM Reliability via Explicit Knowledge Boundary Modelingcited by 13
  10. 2025The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucinationcited by 12
  11. 2025Routine: A Structural Planning Framework for LLM Agent System in Enterprisecited by 9
  12. 2026CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challengescited by 9
25 more
  1. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agentscited by 8
  2. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 8
  3. 2025Agentic Services Computingcited by 6
  4. 2025Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologiescited by 5
  5. 2025Delusions of Large Language Modelscited by 4
  6. 2026Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectoriescited by 4
  7. 2026Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environmentscited by 4
  8. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agentscited by 3
  9. 2025More Vulnerable than You Think: On the Stability of Tool-Integrated LLM Agentscited by 2
  10. 2026RUBAS: Rubric-Based Reinforcement Learning for Agent Safetycited by 2
  11. 2026IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignmentcited by 2
  12. 2026Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Makingcited by 2
  13. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  14. 2026DiffoR: A Unified Continuous Generative Framework for Universal Ordinal Regressioncited by 1
  15. 2026Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reportscited by 1
  16. 2026A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Usecited by 1
  17. 2025Coinvisor: An RL-Enhanced Chatbot Agent for Interactive Cryptocurrency Investment Analysiscited by 0
  18. 2025From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Modelscited by 0
  19. 2026RAS: a Reliability Oriented Metric for Automatic Speech Recognitioncited by 0
  20. 2026No Action Without a NOD: A Heterogeneous Multi-Agent Architecture for Reliable Service Agentscited by 0
  21. 2026XFlow: An Executable Protocol Programming System for Reliable Multi-Agent Workflowscited by 0
  22. 2026To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learningcited by 0
  23. 2026PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noisecited by 0
  24. 2026Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responsescited by 0
  25. 2026Invocation-Level Reliability of Tool-Using Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Mitigating Hallucination Escape in Tool-Using LLM Agents
  2. 2026Internal Representations as Indicators of Hallucinations in Agent Tool Selection
  3. 2026Closed-World Resolution Against Tool Hallucination in LLM Agents
  4. 2026ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents
  5. 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
  6. 2026Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
  7. 2025Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems
  8. 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents