# Evaluating Hallucinations in Chinese Large Language Models

arXiv: https://arxiv.org/abs/2310.03368 · HTML: https://mlsanity.com/paper/2310.03368/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [Detecting Hallucinated Content in Conditional Neural Sequence Generation](https://mlsanity.com/paper/2011.02593/) (2020)
- [PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://mlsanity.com/paper/2104.12369/) (2021)
- [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021)

## Built on by (most cited first, 30 of 33)

- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [HalluLens: LLM Hallucination Benchmark](https://mlsanity.com/paper/2504.17550/) (2025)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [Safeguarding Large Language Models: A Survey](https://mlsanity.com/paper/2406.02622/) (2024)
- [SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models](https://mlsanity.com/paper/2502.13059/) (2025)
- [Can AI Assistants Know What They Don't Know?](https://mlsanity.com/paper/2401.13275/) (2024)
- [Large Language Models Hallucination: A Comprehensive Survey](https://mlsanity.com/paper/2510.06265/) (2025)
- [HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation](https://mlsanity.com/paper/2406.07070/) (2024)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation](https://mlsanity.com/paper/2311.15296/) (2023)
- [UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction](https://mlsanity.com/paper/2402.06861/) (2024)
- [ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models](https://mlsanity.com/paper/2407.04693/) (2024)
- [ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models](https://mlsanity.com/paper/2403.05266/) (2024)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [A Survey of Automatic Hallucination Evaluation on Natural Language Generation](https://mlsanity.com/paper/2404.12041/) (2024)
- [DefAn: Definitive Answer Dataset for LLMs Hallucination Evaluation](https://mlsanity.com/paper/2406.09155/) (2024)
- [Exploring the LLM Journey from Cognition to Expression with Linear Representations](https://mlsanity.com/paper/2405.16964/) (2024)
- [Methodology of Adapting Large English Language Models for Specific Cultural Contexts](https://mlsanity.com/paper/2406.18192/) (2024)
- [DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios](https://mlsanity.com/paper/2505.11340/) (2025)
- [MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs](https://mlsanity.com/paper/2510.22967/) (2025)
- [PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations](https://mlsanity.com/paper/2604.16909/) (2026)
- [C3Bench: A Comprehensive Classical Chinese Understanding Benchmark for Large Language Models](https://mlsanity.com/paper/2405.17732/) (2024)
- [Retrieval-Augmented Generation by Evidence Retroactivity in LLMs](https://mlsanity.com/paper/2501.05475/) (2025)
- [FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models](https://mlsanity.com/paper/2502.18573/) (2025)
- [HalluScore: Large Language Model Hallucination Question Answering Benchmark](https://mlsanity.com/paper/2605.17007/) (2026)
- [OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching](https://mlsanity.com/paper/2503.21813/) (2025)
- [Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors](https://mlsanity.com/paper/2607.00447/) (2026)
- [Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction](https://mlsanity.com/paper/2609.25176/) (2026)

## Abstract (arXiv)

In this paper, we establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. HalluQA contains 450 meticulously designed adversarial questions, spanning multiple domains, and takes into account Chinese historical culture, customs, and social phenomena. During the construction of HalluQA, we consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated. We conduct extensive experiments on 24 large language models, including ERNIE-Bot, Baichuan2, ChatGLM, Qwen, SparkDesk and etc. Out of the 24 models, 18 achieved non-hallucination rates lower than 50%. This indicates that HalluQA is highly challenging. We analyze the primary types of hallucinations in different types of models and their causes. Additionally, we discuss which types of hallucinations should be prioritized for different types of models.