# NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

arXiv: https://arxiv.org/abs/2403.12766 · HTML: https://mlsanity.com/paper/2403.12766/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading](https://mlsanity.com/paper/2310.05029/) (2023)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [Retrieval meets Long Context Large Language Models](https://mlsanity.com/paper/2310.03025/) (2023)
- [LongAlign: A Recipe for Long Context Alignment of Large Language Models](https://mlsanity.com/paper/2401.18058/) (2024)
- [Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache](https://mlsanity.com/paper/2401.02669/) (2024)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)

## Built on by (most cited first, 30 of 32)

- [CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension](https://mlsanity.com/paper/2510.05520/) (2025)
- [Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs](https://mlsanity.com/paper/2410.14641/) (2024)
- [NarraBench: A Comprehensive Framework for Narrative Benchmarking](https://mlsanity.com/paper/2510.09869/) (2025)
- [Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs](https://mlsanity.com/paper/2502.12462/) (2025)
- [Focus Directions Make Your Language Models Pay More Attention to Relevant Contexts](https://mlsanity.com/paper/2503.23306/) (2025)
- [AbsenceBench: Language Models Can't Tell What's Missing](https://mlsanity.com/paper/2506.11440/) (2025)
- [The Pensieve Paradigm: Stateful Language Models Mastering Their Own Context](https://mlsanity.com/paper/2602.12108/) (2026)
- [NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts](https://mlsanity.com/paper/2506.02000/) (2025)
- [Towards Robust Evaluation of STEM Education: Leveraging MLLMs in Project-Based Learning](https://mlsanity.com/paper/2505.17050/) (2025)
- [Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration](https://mlsanity.com/paper/2505.20625/) (2025)
- [A Controllable Examination for Long-Context Language Models](https://mlsanity.com/paper/2506.02921/) (2025)
- [PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts](https://mlsanity.com/paper/2508.09848/) (2025)
- [LMEB: Long-horizon Memory Embedding Benchmark](https://mlsanity.com/paper/2603.12572/) (2026)
- [LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA](https://mlsanity.com/paper/2510.13494/) (2025)
- [AMA: Adaptive Memory via Multi-Agent Collaboration](https://mlsanity.com/paper/2601.20352/) (2026)
- [Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation](https://mlsanity.com/paper/2510.07414/) (2025)
- [Logic Haystacks: Probing LLMs Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding)](https://mlsanity.com/paper/2502.17169/) (2025)
- [STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Stories](https://mlsanity.com/paper/2601.08510/) (2026)
- [Is Progressive Disclosure All You Need for Long-Context Agents?](https://mlsanity.com/paper/2607.17598/) (2026)
- [DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models](https://mlsanity.com/paper/2602.22175/) (2026)
- [The AI Fiction Paradox](https://mlsanity.com/paper/2603.13545/) (2026)
- [ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RL](https://mlsanity.com/paper/2608.28476/) (2026)
- [Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learning](https://mlsanity.com/paper/2505.20195/) (2025)
- [Multimodal Mathematical Reasoning Embedded in Aerial Vehicle Imagery: Benchmarking, Analysis, and Exploration](https://mlsanity.com/paper/2509.10059/) (2025)
- [Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contexts](https://mlsanity.com/paper/2510.22956/) (2025)
- [Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation](https://mlsanity.com/paper/2604.18169/) (2026)
- [Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitations](https://mlsanity.com/paper/2606.00881/) (2026)
- [ELITE: Embedding-Less retrieval with Iterative Text Exploration](https://mlsanity.com/paper/2505.11908/) (2025)
- [SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novels](https://mlsanity.com/paper/2601.09723/) (2025)
- [Storyline Trees: Hierarchical Representations for Long-Form Narratives](https://mlsanity.com/paper/2606.20900/) (2026)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have pushed the boundaries of natural language processing, especially in long-context understanding. However, the evaluation of these models' long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, we introduce NovelQA, a benchmark tailored for evaluating LLMs with complex, extended narratives. Constructed from English novels, NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper details the design and construction of NovelQA, focusing on its comprehensive manual annotation process and the variety of question types aimed at evaluating nuanced comprehension. Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses. Notably, the models struggle with multi-hop reasoning, detail-oriented questions, and handling extremely long inputs, with average lengths exceeding 200,000 tokens. Results highlight the need for substantial advancements in LLMs to enhance their long-context comprehension and contribute effectively to computational literary analysis.