mlsanityData to 14 Oct 2026

NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

arXiv 2403.12766 ↗First posted 18 Mar 2024cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have pushed the boundaries of natural language processing, especially in long-context understanding. However, the evaluation of these models' long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, we introduce NovelQA, a benchmark tailored for evaluating LLMs with complex, extended narratives. Constructed from English novels, NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper details the design and construction of NovelQA, focusing on its comprehensive manual annotation process and the variety of question types aimed at evaluating nuanced comprehension. Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses. Notably, the models struggle with multi-hop reasoning, detail-oriented questions, and handling extremely long inputs, with average lengths exceeding 200,000 tokens. Results highlight the need for substantial advancements in LLMs to enhance their long-context comprehension and contribute effectively to computational literary analysis.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

39 of 58 references.

References · 39

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  11. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  12. 2024OLMo: Accelerating the Science of Language Modelscited by 613
27 more
  1. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  4. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  5. 2023Adapting Language Models to Compress Contextscited by 356
  6. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  7. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  8. 2022Memorizing Transformerscited by 226
  9. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  10. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  11. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  12. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  13. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  14. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  15. 2023Retrieval meets Long Context Large Language Modelscited by 115
  16. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  17. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  18. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  19. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  20. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  21. 2023A Survey on Long Text Modeling with Transformerscited by 47
  22. 2024With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generationcited by 37
  23. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  24. 2023Evaluating Open-QA Evaluationcited by 35
  25. 2023GeoGalactica: A Scientific Large Language Model in Geosciencecited by 33
  26. 2023Recurrent Attention Networks for Long-text Modelingcited by 12
  27. 2023NarrativeXL: A Large-scale Dataset For Long-Term Memory Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 13.

32 papers.

Sort:

By area:

  1. 2025CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehensioncited by 23
  2. 2024Distance between Relevant Information Pieces Causes Bias in Long-Context LLMscited by 16
  3. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  4. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMscited by 9
  5. 2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
  6. 2025AbsenceBench: Language Models Can't Tell What's Missingcited by 9
  7. 2026The Pensieve Paradigm: Stateful Language Models Mastering Their Own Contextcited by 9
  8. 2025NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contextscited by 8
  9. 2025Towards Robust Evaluation of STEM Education: Leveraging MLLMs in Project-Based Learningcited by 7
  10. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  11. 2025A Controllable Examination for Long-Context Language Modelscited by 7
  12. 2025PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contextscited by 7
20 more
  1. 2026LMEB: Long-horizon Memory Embedding Benchmarkcited by 7
  2. 2025LiteraryQA: Towards Effective Evaluation of Long-document Narrative QAcited by 6
  3. 2026AMA: Adaptive Memory via Multi-Agent Collaborationcited by 6
  4. 2025Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluationcited by 4
  5. 2025Logic Haystacks: Probing LLMs Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding)cited by 3
  6. 2026STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Storiescited by 3
  7. 2026Is Progressive Disclosure All You Need for Long-Context Agents?cited by 3
  8. 2026DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Modelscited by 2
  9. 2026The AI Fiction Paradoxcited by 2
  10. 2026ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RLcited by 2
  11. 2025Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learningcited by 1
  12. 2025Multimodal Mathematical Reasoning Embedded in Aerial Vehicle Imagery: Benchmarking, Analysis, and Explorationcited by 1
  13. 2025Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contextscited by 1
  14. 2026Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translationcited by 1
  15. 2026Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitationscited by 1
  16. 2025ELITE: Embedding-Less retrieval with Iterative Text Explorationcited by 0
  17. 2025SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novelscited by 0
  18. 2026Storyline Trees: Hierarchical Representations for Long-Form Narrativescited by 0
  19. 2026WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoningcited by 0
  20. 2026Just-In-Time Agent Memory with Runtime Agentic Researchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels
  2. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge
  3. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
  4. 2024FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models
  5. 2023Marathon: A Race Through the Realm of Long Context with Large Language Models
  6. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  7. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  8. 2023ToolQA: A Dataset for LLM Question Answering with External Tools