# LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization

arXiv: https://arxiv.org/abs/2608.18082 · HTML: https://mlsanity.com/paper/2608.18082/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 56 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [Giraffe: Adventures in Expanding Context Lengths in LLMs](https://mlsanity.com/paper/2308.10882/) (2023)
- [HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations](https://mlsanity.com/paper/2503.07833/) (2025)
- [CLIPPER: Compression enables long-context synthetic data generation](https://mlsanity.com/paper/2502.14854/) (2025)
- [360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training](https://mlsanity.com/paper/2505.22296/) (2025)
- [Uncertainty Resolution in Misinformation Detection](https://mlsanity.com/paper/2401.01197/) (2024)

## Abstract (arXiv)

Although context windows have expanded significantly in recent years, hallucinations in long-context summarization remain a challenge. Long novels are better suited than news or papers for researching these hallucinations, due to their intrinsic information and detailed descriptions of events and dialogues. However, current research lacks a multi-scale benchmark for hallucination detection in long-context novel summarization and does not fully explore how hallucinations change as the context grows longer. In this study, we propose LongNovel, a multi-scale long-context bilingual (Chinese and English) novel benchmark for hallucination detection. This benchmark is constructed from 29 Chinese novels (ranging from 16k to 100k tokens) and chapter-level data from the BookSum dataset. We design 8 hallucination types and employ a combination of Multi-Model Arbitration and Entity-Referenced Hallucination Generation to ensure both data authenticity and a balanced distribution of hallucination categories. Furthermore, we manually revise the content in the test set to guarantee data reliability. Extensive experimental results demonstrate that LongNovel is a challenging benchmark. We release LongNovel for future research. https://github.com/BDML-lab/LongNovel