mlsanityData to 14 Oct 2026

Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels

arXiv 2505.14925 ↗First posted 20 May 2025cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Although the context length of large language models (LLMs) has increased to millions of tokens, evaluating their effectiveness beyond needle-in-a-haystack approaches has proven difficult. We argue that novels provide a case study of subtle, complicated structure and long-range semantic dependencies often over 128k tokens in length. Inspired by work on computational novel analysis, we release the Too Long, Didn't Model (TLDM) benchmark, which tests a model's ability to report plot summary, storyworld configuration, and elapsed narrative time. We find that none of seven tested frontier LLMs retain stable understanding beyond 64k tokens. Our results suggest language model developers must look beyond "lost in the middle" benchmarks when evaluating model performance in complex long-context scenarios. To aid in further development we release the TLDM benchmark together with reference code and data.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 16 of its 34 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

16 of 34 references.

References · 16

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  5. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  7. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  8. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  9. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  10. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  11. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  12. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 62
4 more
  1. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 53
  2. 2025(How) Do Language Models Track State?cited by 34
  3. 2025Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detectioncited by 24
  4. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  2. 2026STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Storiescited by 3
  3. 2026The AI Fiction Paradoxcited by 2
  4. 2026Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Lengthcited by 2
  5. 2026AI as a Tool for Simulation-Based Experiments in Literary Studiescited by 2
  6. 2026Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translationcited by 1
  7. 2026Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAGcited by 1
  8. 2026CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Basescited by 1
  9. 2026GPT-5 vs Other LLMs in Long Short-Context Performancecited by 0