# FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Models

arXiv: https://arxiv.org/abs/2502.17924 · HTML: https://mlsanity.com/paper/2502.17924/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 41 of its 65 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 41)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020)
- [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information](https://mlsanity.com/paper/2106.05707/) (2021)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)

## Built on by (most cited first, 19 of 19)

- [Agent-as-a-Judge](https://mlsanity.com/paper/2601.05111/) (2026)
- [AgentMaster: A Multi-Agent Conversational Framework Using A2A and MCP Protocols for Multimodal Information Retrieval and Analysis](https://mlsanity.com/paper/2507.21105/) (2025)
- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025)
- [ProbeLLM: Automating Principled Diagnosis of LLM Failures](https://mlsanity.com/paper/2602.12966/) (2026)
- [AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness](https://mlsanity.com/paper/2507.01702/) (2025)
- [Debating Truth: Debate-driven Claim Verification with Multiple Large Language Model Agents](https://mlsanity.com/paper/2507.19090/) (2025)
- [DR-Arena: an Automated Evaluation Framework for Deep Research Agents](https://mlsanity.com/paper/2601.10504/) (2026)
- [T^2Agent A Tool-augmented Multimodal Misinformation Detection Agent with Monte Carlo Tree Search](https://mlsanity.com/paper/2505.19768/) (2025)
- [Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking](https://mlsanity.com/paper/2601.02669/) (2026)
- [Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents](https://mlsanity.com/paper/2508.17393/) (2025)
- [MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique](https://mlsanity.com/paper/2511.09067/) (2025)
- [SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring](https://mlsanity.com/paper/2607.18665/) (2026)
- [HiFACTMix: A Code-Mixed Benchmark and Graph-Aware Model for EvidenceBased Political Claim Verification in Hinglish](https://mlsanity.com/paper/2508.10001/) (2025)
- [MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models](https://mlsanity.com/paper/2510.27196/) (2025)
- [Story2Proposal: A Scaffold for Structured Scientific Paper Writing](https://mlsanity.com/paper/2603.27065/) (2026)
- [Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity](https://mlsanity.com/paper/2604.04692/) (2026)
- [TRUST Agents: A Collaborative Multi-Agent Framework for Fake News Detection, Explainable Verification, and Logic-Aware Claim Reasoning](https://mlsanity.com/paper/2604.12184/) (2026)
- [CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild](https://mlsanity.com/paper/2605.30241/) (2026)
- [MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentials](https://mlsanity.com/paper/2609.08399/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have significantly advanced the fact-checking studies. However, existing automated fact-checking evaluation methods rely on static datasets and classification metrics, which fail to automatically evaluate the justification production and uncover the nuanced limitations of LLMs in fact-checking. In this work, we introduce FACT-AUDIT, an agent-driven framework that adaptively and dynamically assesses LLMs' fact-checking capabilities. Leveraging importance sampling principles and multi-agent collaboration, FACT-AUDIT generates adaptive and scalable datasets, performs iterative model-centric evaluations, and updates assessments based on model-specific responses. By incorporating justification production alongside verdict prediction, this framework provides a comprehensive and evolving audit of LLMs' factual reasoning capabilities, to investigate their trustworthiness. Extensive experiments demonstrate that FACT-AUDIT effectively differentiates among state-of-the-art LLMs, providing valuable insights into model strengths and limitations in model-centric fact-checking analysis.