# BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali

arXiv: https://arxiv.org/abs/2605.31483 · HTML: https://mlsanity.com/paper/2605.31483/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 26 of 26)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://mlsanity.com/paper/2101.00204/) (2021)
- [Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?](https://mlsanity.com/paper/2502.12215/) (2025)
- [HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation](https://mlsanity.com/paper/2406.07070/) (2024)
- [FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs](https://mlsanity.com/paper/2410.13210/) (2024)
- [BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP](https://mlsanity.com/paper/2309.13173/) (2023)
- [Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation](https://mlsanity.com/paper/2506.17088/) (2025)
- [TigerLLM - A Family of Bangla Large Language Models](https://mlsanity.com/paper/2503.10995/) (2025)
- [HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations](https://mlsanity.com/paper/2503.07833/) (2025)
- [AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs](https://mlsanity.com/paper/2509.04656/) (2025)
- [BnSentMix: A Diverse Bengali-English Code-Mixed Dataset for Sentiment Analysis](https://mlsanity.com/paper/2408.08964/) (2024)
- [Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?](https://mlsanity.com/paper/2507.23248/) (2025)
- [Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning](https://mlsanity.com/paper/2505.21354/) (2025)
- [BnMMLU: Measuring Massive Multitask Language Understanding in Bengali](https://mlsanity.com/paper/2505.18951/) (2025)
- [MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning](https://mlsanity.com/paper/2510.14305/) (2025)
- [BanglaLlama: LLaMA for Bangla Language](https://mlsanity.com/paper/2410.21200/) (2024)
- [How Much Do LLMs Hallucinate across Languages? On Realistic Multilingual Estimation of LLM Hallucination](https://mlsanity.com/paper/2502.12769/) (2025)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Halluverse-M^3: A multitask multilingual benchmark for hallucination in LLMs](https://mlsanity.com/paper/2602.06920/) (2026)
- [Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation](https://mlsanity.com/paper/2602.14564/) (2026)
- [BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization](https://mlsanity.com/paper/2602.16843/) (2026)
- [MixSarc: A Bangla-English Code-Mixed Corpus for Implicit Meaning Identification](https://mlsanity.com/paper/2602.21608/) (2026)

## Built on by (most cited first, 1 of 1)

- [When a Name Is Not a Name: A Benchmark Dataset and Distilled Reasoning for Culturally Entangled Bangla Homographs in Low-Resource LLMs](https://mlsanity.com/paper/2607.17828/) (2026)

## Abstract (arXiv)

Despite Bengali being the sixth most spoken language in the world, no prior work has systematically evaluated hallucination in large language models (LLMs) for Bengali. We introduce BenHalluEval, a fine-grained hallucination evaluation framework for Bengali covering four tasks: Generative Question Answering (GQA), Bangla-English Code-Mixed QA, Summarization, and Reasoning. We construct 12,000 hallucinated candidates using GPT-5.4 across twelve task-specific hallucination types, drawn from three existing Bengali datasets, and evaluate seven LLMs spanning reasoning-oriented, multilingual, and Bengali-centric categories under a dual-track protocol that independently measures false-positive rate on ground-truth instances (Track A) and hallucination detection rate on hallucinated candidates (Track B). To jointly penalise both failure modes and prevent inflated scores from uniform response bias, we propose BenHalluScore, a dual-track calibration metric that ranges from 7.72% to 55.42% across models and tasks, revealing substantial variation in hallucination calibration. Chain-of-thought prompting, applied as a mitigation strategy, shifts response distributions without consistently improving hallucination discrimination. BenHalluEval establishes the first dedicated hallucination benchmark for Bengali and highlights the inadequacy of single-track and prompting-only evaluation approaches for low-resource language settings. The dataset and code are available at https://anonymous.4open.science/r/BanglaHalluEval-EB77.