# Lynx: An Open Source Hallucination Evaluation Model

arXiv: https://arxiv.org/abs/2407.08488 · HTML: https://mlsanity.com/paper/2407.08488/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination detection in RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Placed in

- Retrieval-augmented generation / [Hallucination detection in RAG](https://mlsanity.com/rag/hallucination-detection-in-rag/)

## Builds on (most cited first, 28 of 28)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://mlsanity.com/paper/2204.07931/) (2022)
- [Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization](https://mlsanity.com/paper/2109.09784/) (2021)
- [ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks](https://mlsanity.com/paper/1803.01937/) (2018)
- [Neural Natural Language Inference Models Enhanced with External Knowledge](https://mlsanity.com/paper/1711.04289/) (2017)
- [A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology](https://mlsanity.com/paper/2308.04709/) (2023)
- [Embedding Hallucination for Few-Shot Language Fine-tuning](https://mlsanity.com/paper/2205.01307/) (2022)

## Built on by (most cited first, 30 of 57)

- [MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models](https://mlsanity.com/paper/2502.14302/) (2025)
- [LMUnit: Fine-grained Evaluation with Natural Language Unit Tests](https://mlsanity.com/paper/2412.13091/) (2024)
- [Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings](https://mlsanity.com/paper/2503.15620/) (2025)
- [Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges](https://mlsanity.com/paper/2601.20913/) (2026)
- [Reconsidering LLM Uncertainty Estimation Methods in the Wild](https://mlsanity.com/paper/2506.01114/) (2025)
- [Differential Transformer](https://mlsanity.com/paper/2410.05258/) (2024)
- [Mitigating Manipulation and Enhancing Persuasion: A Reflective Multi-Agent Approach for Legal Argument Generation](https://mlsanity.com/paper/2506.02992/) (2025)
- [A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation](https://mlsanity.com/paper/2412.15298/) (2024)
- [Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets](https://mlsanity.com/paper/2504.20119/) (2025)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [Passage Segmentation of Documents for Extractive Question Answering](https://mlsanity.com/paper/2501.09940/) (2025)
- [Exposing the Ghost in the Transformer: Abnormal Detection for Large Language Models via Hidden State Forensics](https://mlsanity.com/paper/2504.00446/) (2025)
- [FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG](https://mlsanity.com/paper/2601.05866/) (2026)
- [Hybrid Retrieval for Hallucination Mitigation in Large Language Models: A Comparative Analysis](https://mlsanity.com/paper/2504.05324/) (2025)
- [Can LLMs Detect Their Confabulations? Estimating Reliability in Uncertainty-Aware Language Models](https://mlsanity.com/paper/2508.08139/) (2025)
- [Learning to Reason for Hallucination Span Detection](https://mlsanity.com/paper/2510.02173/) (2025)
- [Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output](https://mlsanity.com/paper/2411.01022/) (2024)
- [TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs](https://mlsanity.com/paper/2507.08203/) (2025)
- [Representation-based Broad Hallucination Detectors Fail to Generalize Out of Distribution](https://mlsanity.com/paper/2509.19372/) (2025)
- [FinGround: Detecting and Grounding Financial Hallucinations via Atomic Claim Verification](https://mlsanity.com/paper/2604.23588/) (2026)
- [HalluMix: A Task-Agnostic, Multi-Domain Benchmark for Real-World Hallucination Detection](https://mlsanity.com/paper/2505.00506/) (2025)
- [Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection](https://mlsanity.com/paper/2505.17558/) (2025)
- [Reasoning's Razor: Reasoning Improves Accuracy but Can Hurt Recall at Critical Operating Points in Safety and Hallucination Detection](https://mlsanity.com/paper/2510.21049/) (2025)
- [Large Language Models as Search Engines: Societal Challenges](https://mlsanity.com/paper/2512.08946/) (2025)
- [Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction](https://mlsanity.com/paper/2603.18014/) (2026)
- [PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts](https://mlsanity.com/paper/2605.17028/) (2026)
- [Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents](https://mlsanity.com/paper/2607.00895/) (2026)
- [SCAN: Structured Capability Assessment and Navigation for LLMs](https://mlsanity.com/paper/2505.06698/) (2025)
- [MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations](https://mlsanity.com/paper/2505.14101/) (2025)
- [HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.00880/) (2025)

## Abstract (arXiv)

Retrieval Augmented Generation (RAG) techniques aim to mitigate hallucinations in Large Language Models (LLMs). However, LLMs can still produce information that is unsupported or contradictory to the retrieved contexts. We introduce LYNX, a SOTA hallucination detection LLM that is capable of advanced reasoning on challenging real-world hallucination scenarios. To evaluate LYNX, we present HaluBench, a comprehensive hallucination evaluation benchmark, consisting of 15k samples sourced from various real-world domains. Our experiment results show that LYNX outperforms GPT-4o, Claude-3-Sonnet, and closed and open-source LLM-as-a-judge models on HaluBench. We release LYNX, HaluBench and our evaluation code for public access.