mlsanityData to 7 Oct 2026

Evaluating Hallucinations in Chinese Large Language Models

arXiv 2310.03368 ↗First posted 5 Oct 2023cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

In this paper, we establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. HalluQA contains 450 meticulously designed adversarial questions, spanning multiple domains, and takes into account Chinese historical culture, customs, and social phenomena. During the construction of HalluQA, we consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated. We conduct extensive experiments on 24 large language models, including ERNIE-Bot, Baichuan2, ChatGLM, Qwen, SparkDesk and etc. Out of the 24 models, 18 achieved non-hallucination rates lower than 50%. This indicates that HalluQA is highly challenging. We analyze the primary types of hallucinations in different types of models and their causes. Additionally, we discuss which types of hallucinations should be prioritized for different types of models.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

33 of 46 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
21 more
  1. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  3. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  6. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  8. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  11. 2023GPTScore: Evaluate as You Desirecited by 357
  12. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  13. 2022UL2: Unifying Language Learning Paradigmscited by 292
  14. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  15. 2023Do Large Language Models Know What They Don't Know?cited by 238
  16. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  17. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  18. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  19. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  20. 2023Evaluating Open-QA Evaluationcited by 35
  21. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 14; 2025: 12; 2026: 4.

33 papers.

Sort:

By area:

  1. 2024Measuring short-form factuality in large language modelscited by 345
  2. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  3. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  4. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  5. 2025HalluLens: LLM Hallucination Benchmarkcited by 88
  6. 2024Safeguarding Large Language Models: A Surveycited by 73
  7. 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 61
  8. 2024Can AI Assistants Know What They Don't Know?cited by 47
  9. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  10. 2024HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluationcited by 26
  11. 2023UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generationcited by 22
  12. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
21 more
  1. 2024UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Constructioncited by 15
  2. 2024ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Modelscited by 14
  3. 2024ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Modelscited by 9
  4. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  5. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  6. 2024DefAn: Definitive Answer Dataset for LLMs Hallucination Evaluationcited by 6
  7. 2024Exploring the LLM Journey from Cognition to Expression with Linear Representationscited by 5
  8. 2024Methodology of Adapting Large English Language Models for Specific Cultural Contextscited by 4
  9. 2025DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarioscited by 4
  10. 2025MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMscited by 4
  11. 2025Retrieval-Augmented Generation by Evidence Retroactivity in LLMscited by 3
  12. 2026PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinationscited by 3
  13. 2024C3Bench: A Comprehensive Classical Chinese Understanding Benchmark for Large Language Modelscited by 2
  14. 2025FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Modelscited by 2
  15. 2026HalluScore: Large Language Model Hallucination Question Answering Benchmarkcited by 2
  16. 2025OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matchingcited by 1
  17. 2026Understanding Why Language Models Hallucinate: Testing Reasoning Against Priorscited by 1
  18. 2026Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interactioncited by 1
  19. 2025Iterative Tree Analysis for Medical Criticscited by 0
  20. 2025OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Modelscited by 0
  21. 2025Acquiring Common Chinese Emotional Events Using Large Language Modelcited by 0