mlsanityData to 14 Oct 2026

Context Is Not Comprehension

arXiv 2506.04907 ↗First posted 5 Jun 2025cs.CL cs.AI cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

The dominant way of judging Large Language Models (LLMs) has been to ask how well they can recall explicit facts from very long inputs. While today's best models achieve near perfect recall, this masks a harder skill: performing multi-step reasoning and tracking intermediate state that never appears verbatim. We introduce Verbose ListOps (VLO), a benchmark that embeds deterministic ListOps computations inside narrative camouflage and, crucially, allows step-level evaluation of every intermediate result. Experiments show that models which solve raw ListOps with approximately 100% accuracy collapse on VLO after only 10,000 tokens. By exposing where a model's reasoning chain first diverges, VLO moves assessment beyond sheer context length and toward genuine comprehension. VLO's generation pipeline is task-agnostic: it can weave any deterministically verifiable reasoning schema -- arithmetic, symbolic, abductive, inductive or defeasible -- into narrative form. This makes VLO a reusable test-bed for the next wave of reasoning-centric model designs, not merely those with step-explicit scaffolds.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

30 of 52 references.

References · 30

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  8. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  9. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  10. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  12. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
18 more
  1. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  2. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  3. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  4. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  5. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  6. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  7. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  8. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  9. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 89
  10. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 50
  11. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  12. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  13. 2025Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Modelscited by 14
  14. 2025Medical large language models are easily distractedcited by 9
  15. 2024Are LLMs classical or nonmonotonic reasoners? Lessons from genericscited by 7
  16. 2025Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generatorcited by 5
  17. 2024PECC: Problem Extraction and Coding Challengescited by 4
  18. 2025Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystackscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EVAR: Evidence-Validated Hypothesis Admission for Budget-Aware Narrative Reasoning
  2. 2024Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives
  3. 2025Language Models Might Not Understand You: Evaluating Theory of Mind via Story Prompting
  4. 2026MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs
  5. 2025ComoRAG: A Cognitive-Inspired Memory-Organized RAG for Stateful Long Narrative Reasoning
  6. 2025RELIC: Evaluating Complex Reasoning via the Recognition of Languages In-Context
  7. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
  8. 2024MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks