mlsanityData to 14 Oct 2026

Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration

arXiv 2402.00367 ↗First posted 1 Feb 2024cs.CLcited by 180

Not on the map. Why?

Cited by 25 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite efforts to expand the knowledge of large language models (LLMs), knowledge gaps -- missing or outdated information in LLMs -- might always persist given the evolving nature of knowledge. In this work, we study approaches to identify LLM knowledge gaps and abstain from answering questions when knowledge gaps are present. We first adapt existing approaches to model calibration or adaptation through fine-tuning/prompting and analyze their ability to abstain from generating low-confidence outputs. Motivated by their failures in self-reflection and over-reliance on held-out sets, we propose two novel approaches that are based on model collaboration, i.e., LLMs probing other LLMs for knowledge gaps, either cooperatively or competitively. Extensive experiments with three LLMs on four QA tasks featuring diverse knowledge domains demonstrate that both cooperative and competitive approaches to unveiling LLM knowledge gaps achieve up to 19.3% improvements on abstain accuracy against the strongest baseline. Further analysis reveals that our proposed mechanisms could help identify failure cases in retrieval augmentation and pinpoint knowledge gaps in multi-hop reasoning.

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). Builds on 60 of its 93 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

60 of 93 references.

References · 60

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2017On Calibration of Modern Neural Networkscited by 4,960
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  12. 2019Language Models as Knowledge Bases?cited by 1,936
38 more
  1. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  13. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  14. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  15. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  16. 2021Editing Factual Knowledge in Language Modelscited by 553
  17. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  18. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  19. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  20. 2023Language Models can Solve Computer Taskscited by 372
  21. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  22. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  23. 2020Calibration of Pre-trained Transformerscited by 315
  24. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  25. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 256
  26. 2022RealTime QA: What's the Answer Right Now?cited by 244
  27. 2020Selective Question Answering under Domain Shiftcited by 238
  28. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  29. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  30. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  31. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 98
  32. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  33. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  34. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  35. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  36. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  37. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 83
  38. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 48; 2025: 74; 2026: 58.

180 papers.

Sort:

By area:

  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  3. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  4. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  5. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  6. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  7. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  8. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  9. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 43
  10. 2024DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detectioncited by 41
  11. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 41
  12. 2024MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoningcited by 41
38 more
  1. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  2. 2025GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modelingcited by 36
  3. 2024Reverse Thinking Makes LLMs Stronger Reasonerscited by 29
  4. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  5. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  6. 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answercited by 27
  7. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  8. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 24
  9. 2024Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMscited by 23
  10. 2024A Survey on the Honesty of Large Language Modelscited by 23
  11. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 23
  12. 2025Multiple LLM Agents Debate for Equitable Cultural Alignmentcited by 21
  13. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  14. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  15. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  16. 2026Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montagecited by 19
  17. 2024Long-Form Answers to Visual Questions from Blind and Low Vision Peoplecited by 18
  18. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  19. 2025Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Modelscited by 18
  20. 2024Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMscited by 17
  21. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  22. 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMscited by 15
  23. 2025Shifting from Ranking to Set Selection for Retrieval Augmented Generationcited by 15
  24. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 15
  25. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  26. 2024DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metricscited by 14
  27. 2024Distinguishing Ignorance from Error in LLM Hallucinationscited by 14
  28. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 14
  29. 2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
  30. 2025Guiding Reasoning in Small Language Models with LLM Assistancecited by 14
  31. 2025Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAGcited by 14
  32. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  33. 2026Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RLcited by 14
  34. 2024Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Modelscited by 13
  35. 2024Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?cited by 13
  36. 2025Detecting (Un)answerability in Large Language Models with Linear Directionscited by 13
  37. 2024LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statementscited by 12
  38. 2024Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Frameworkcited by 12

50 most cited of 180 All 180 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs
  2. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text
  3. 2026How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledge
  4. 2024Hierarchical Deconstruction of LLM Reasoning: A Graph-Based Framework for Analyzing Knowledge Utilization
  5. 2026Exploring Knowledge Conflicts for Faithful LLM Reasoning: Benchmark and Method
  6. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges
  7. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge
  8. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA