mlsanityData to 7 Oct 2026

HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs

arXiv 2402.16211 ↗First posted 25 Feb 2024cs.CLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Hallucinations pose a significant challenge to the reliability and alignment of Large Language Models (LLMs), limiting their widespread acceptance beyond chatbot applications. Despite ongoing efforts, hallucinations remain a prevalent challenge in LLMs. The detection of hallucinations itself is also a formidable task, frequently requiring manual labeling or constrained evaluations. This paper introduces an automated scalable framework that combines benchmarking LLMs' hallucination tendencies with efficient hallucination detection. We leverage LLMs to generate challenging tasks related to hypothetical phenomena, subsequently employing them as agents for efficient hallucination detection. The framework is domain-agnostic, allowing the use of any language model for benchmark creation or evaluation in any domain. We introduce the publicly available HypoTermQA Benchmarking Dataset, on which state-of-the-art models' performance ranged between 3% and 11%, and evaluator agents demonstrated a 6% error rate in hallucination prediction. The proposed framework provides opportunities to test and improve LLMs. Additionally, it has the potential to generate benchmarking datasets tailored to specific domains, such as law, health, and finance.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 41 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

28 of 41 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  12. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
16 more
  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  3. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  4. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  5. 2022Prompting GPT-3 To Be Reliablecited by 258
  6. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  7. 2023Do Large Language Models Know What They Don't Know?cited by 238
  8. 2023Language Modeling Is Compressioncited by 225
  9. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  10. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  11. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  12. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 130
  13. 2023DelucionQA: Detecting Hallucinations in Domain-specific Question Answeringcited by 43
  14. 2023A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detectioncited by 42
  15. 2023UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generationcited by 22
  16. 2023Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  2. 2024From Single to Multi: How LLMs Hallucinate in Multi-Document Summarizationcited by 17
  3. 2025Characterizing the Robustness of Black-Box LLM Planners Under Perturbed Observations with Adaptive Stress Testingcited by 1
  4. 2026PhantomBench: Benchmarking the Non-existential Threat of Language Modelscited by 1
  5. 2025OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Modelscited by 0
  6. 2026Inducing Epistemological Humility in Large Language Models: A Targeted SFT Approach to Reducing Hallucinationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
  2. 2025Theoretical Foundations and Mitigation of Hallucination in Large Language Models
  3. 2023Chainpoll: A high efficacy method for LLM hallucination detection
  4. 2025SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models
  5. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models
  6. 2026Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
  7. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models
  8. 2024THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models