mlsanityData to 14 Oct 2026

FactGuard: Leveraging Multi-Agent Systems to Generate Answerable and Unanswerable Questions for Enhanced Long-Context LLM Extraction

arXiv 2504.05607 ↗First posted 8 Apr 2025cs.CL cs.AIcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Despite the success of Large Language Models (LLMs) in long-context tasks, their reliability in recognizing unanswerable questions remains a critical challenge. In information-seeking scenarios, models often suffer from overconfidence and hallucinations when evidence is missing or misleading. In this paper, we propose FACTGUARD, a systematic multi-task workflow for generating evidence-grounded answerable and unanswerable questions. We introduce FACTGUARD-BENCH, a benchmark comprising 25,220 examples with context lengths up to 128K, specifically targeting Lack of Evidence and Misleading Evidence scenarios. Evaluations of eleven representative open-source and proprietary LLMs reveal a significant reliability gap: models struggle to reject unanswerable questions as context scales, with the leading model achieving only 71.62% accuracy. By training with FACTGUARD-BENCH, we improve overall accuracy to 81.17% and significantly enhance models' reasoned rejection capabilities. Our work provides a scalable path toward building more faithful and trustworthy long-context LLMs. Our source is publicly available at https://github.com/biaoliu-kiritsugu/FactGuard.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 30 of its 53 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

30 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents20 Nov
  2. 2023Agents: An Open-source Framework for Autonomous Language Agents14 Sep
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug
  4. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
14 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,049
  2. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  3. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  4. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  5. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  6. 2022(QA)2: Question Answering with Questionable Assumptionscited by 51
  7. 2017Relation Extraction : A Surveycited by 44
  8. 2020A Survey on Machine Reading Comprehension Systemscited by 41
  9. 2018Read + Verify: Machine Reading Comprehension with Unanswerable Questionscited by 31
  10. 2024Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanationscited by 30
  11. 2023TigerBot: An Open Multilingual Multitask LLMcited by 15
  12. 2024Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaborationcited by 10
  13. 2023SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examplescited by 8
  14. 2024RetinaQA: A Robust Knowledge Base Question Answering Model for both Answerable and Unanswerable Questionscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration
  2. 2026A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
  3. 2026Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models
  4. 2025Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Models
  5. 2025Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaboration
  6. 2025Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration
  7. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
  8. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth