# HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews

arXiv: https://arxiv.org/abs/2609.03580 · HTML: https://mlsanity.com/paper/2609.03580/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://mlsanity.com/paper/2104.08704/) (2021)
- [Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization](https://mlsanity.com/paper/2109.09784/) (2021)
- [The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates](https://mlsanity.com/paper/2405.02150/) (2024)
- [Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review](https://mlsanity.com/paper/2412.01708/) (2024)
- [Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025](https://mlsanity.com/paper/2504.09737/) (2025)
- [Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters](https://mlsanity.com/paper/2204.07447/) (2022)
- [FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs](https://mlsanity.com/paper/2410.13210/) (2024)
- [Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review](https://mlsanity.com/paper/2410.03019/) (2024)
- [SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations](https://mlsanity.com/paper/2510.04398/) (2025)

## Abstract (arXiv)

The growing scale of academic peer review has motivated the use of Large Language Models (LLMs) as review assistants, yet LLMs can generate fluent but unsupported claims that undermine review reliability. Existing hallucination benchmarks are not designed for peer review, where verification requires grounding claims in long, technical papers. We introduce HalluPeer, a benchmark for detecting hallucinations in scientific peer reviews, providing aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization. Our pipeline induces a peer-review-specific hallucination taxonomy, identifies review contexts, and injects hallucinations with automated filtering. Experiments on 12K papers and 38K reviews show that existing detectors struggle to separate hallucinations from legitimate critique, while evaluation on authentic reviews demonstrates that HalluPeer-defined hallucination patterns occur in real peer reviews, highlighting the critical need for source-aware verification. Our project page can be found in https://github.com/Lin-TzuLing/HalluPeer.git