# UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking

arXiv: https://arxiv.org/abs/2505.15063 · HTML: https://mlsanity.com/paper/2505.15063/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 48 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 19 of 19)

- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [X-FACT: A New Benchmark Dataset for Multilingual Fact Checking](https://mlsanity.com/paper/2106.09248/) (2021)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [FIRE: Fact-checking with Iterative Retrieval and Verification](https://mlsanity.com/paper/2411.00784/) (2024)
- [OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs](https://mlsanity.com/paper/2408.11832/) (2024)
- [Loki: An Open-Source Tool for Fact Verification](https://mlsanity.com/paper/2410.01794/) (2024)
- [OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs](https://mlsanity.com/paper/2405.05583/) (2024)
- [UQA: Corpus for Urdu Question Answering](https://mlsanity.com/paper/2405.01458/) (2024)
- [Cross-lingual COVID-19 Fake News Detection](https://mlsanity.com/paper/2110.06495/) (2021)
- [UQuAD1.0: Development of an Urdu Question Answering Training Data for Machine Reading Comprehension](https://mlsanity.com/paper/2111.01543/) (2021)
- [Overview of the Shared Task on Fake News Detection in Urdu at FIRE 2021](https://mlsanity.com/paper/2207.05133/) (2022)
- [Detecting Social Media Manipulation in Low-Resource Languages](https://mlsanity.com/paper/2011.05367/) (2020)

## Built on by (most cited first, 4 of 4)

- [Agent-as-a-Judge](https://mlsanity.com/paper/2601.05111/) (2026)
- [DunbaaBERT: From Sacrifice to Semantics](https://mlsanity.com/paper/2605.26935/) (2026)
- [UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding](https://mlsanity.com/paper/2606.07167/) (2026)
- [Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability](https://mlsanity.com/paper/2607.10402/) (2026)

## Abstract (arXiv)

The rapid adoption of Large Language Models (LLMs) has raised important concerns about the factual reliability of their outputs, particularly in low-resource languages such as Urdu. Existing automated fact-checking systems are predominantly developed for English, leaving a significant gap for the more than 200 million Urdu speakers worldwide. In this work, we present UrduFactBench and UrduFactQA, two novel hand-annotated benchmarks designed to enable fact-checking and factual consistency evaluation in Urdu. While UrduFactBench focuses on claim verification, UrduFactQA targets the factuality of LLMs in question answering. These resources, the first of their kind for Urdu, were developed through a multi-stage annotation process involving native Urdu speakers. To complement these benchmarks, we introduce UrduFactCheck, a modular fact-checking framework that incorporates both monolingual and translation-based evidence retrieval strategies to mitigate the scarcity of high-quality Urdu evidence. Leveraging these resources, we conduct an extensive evaluation of twelve LLMs and demonstrate that translation-augmented pipelines consistently enhance performance compared to monolingual ones. Our findings reveal persistent challenges for open-source LLMs in Urdu and underscore the importance of developing targeted resources. All code and data are publicly available at https://github.com/mbzuai-nlp/UrduFactCheck.