mlsanityData to 7 Oct 2026

One Thousand and One Pairs: A "novel" challenge for long-context language models

arXiv 2406.16264 ↗First posted 24 Jun 2024cs.CL cs.AIcited by 80

Not on the map. Why?

Abstract (arXiv)

Synthetic long-context LLM benchmarks (e.g., "needle-in-the-haystack") test only surface-level retrieval capabilities, but how well can long-context LLMs retrieve, synthesize, and reason over information across book-length inputs? We address this question by creating NoCha, a dataset of 1,001 minimally different pairs of true and false claims about 67 recently-published English fictional books, written by human readers of those books. In contrast to existing long-context benchmarks, our annotators confirm that the largest share of pairs in NoCha require global reasoning over the entire book to verify. Our experiments show that while human readers easily perform this task, it is enormously challenging for all ten long-context LLMs that we evaluate: no open-weight model performs above random chance (despite their strong performance on synthetic benchmarks), while GPT-4o achieves the highest accuracy at 55.8%. Further analysis reveals that (1) on average, models perform much better on pairs that require only sentence-level retrieval vs. global reasoning; (2) model-generated explanations for their decisions are often inaccurate even for correctly-labeled claims; and (3) models perform substantially worse on speculative fiction books that contain extensive world-building. The methodology proposed in NoCha allows for the evolution of the benchmark dataset and the easy analysis of future models.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 31 of its 66 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

31 of 66 references.

References · 31

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  8. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  11. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  12. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
19 more
  1. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  4. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  5. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  7. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  8. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  9. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  10. 2024Long-form factuality in large language modelscited by 146
  11. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  12. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  13. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  14. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  15. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  16. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  17. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  18. 2022MuLD: The Multitask Long Document Benchmarkcited by 9
  19. 2021Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QAcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 45; 2026: 16.

80 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  2. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  3. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  4. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  5. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  6. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  7. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  8. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  9. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 95
  10. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 85
  11. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  12. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 49
38 more
  1. 2025People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated textcited by 39
  2. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  3. 2025BEARCUBS: A benchmark for computer-using web agentscited by 29
  4. 2024Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Conceptscited by 28
  5. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  6. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  7. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  8. 2024DetectiveQA: Evaluating Long-Context Reasoning on Detective Novelscited by 21
  9. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 18
  10. 2026LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleaguescited by 17
  11. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  12. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  13. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  14. 2025Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactionscited by 12
  15. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
  16. 2024RefreshKV: Updating Small KV Cache During Long-form Generationcited by 11
  17. 2024On Many-Shot In-Context Learning for Long-Context Evaluationcited by 10
  18. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  19. 2025Code Like Humans: A Multi-Agent Solution for Medical Codingcited by 10
  20. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMscited by 9
  21. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 9
  22. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  23. 2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
  24. 2025Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novelscited by 9
  25. 2025Positional Biases Shift as Inputs Approach Context Window Limitscited by 9
  26. 2025NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contextscited by 8
  27. 2025SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehensioncited by 8
  28. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 8
  29. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuningcited by 7
  30. 2025A Controllable Examination for Long-Context Language Modelscited by 7
  31. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  32. 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?cited by 5
  33. 2025Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understandingcited by 5
  34. 2025Movie Facts and Fibs (MF2): A Benchmark for Long Movie Understandingcited by 4
  35. 2025HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modelingcited by 4
  36. 2026Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancingcited by 4
  37. 2024Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Taskscited by 3
  38. 2025Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approachescited by 3

50 most cited of 80 All 80 (JSON)