mlsanityData to 7 Oct 2026

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

arXiv 2409.12640 ↗First posted 19 Sep 2024cs.CL cs.LGcited by 85

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We introduce Michelangelo: a minimal, synthetic, and unleaked long-context reasoning evaluation for large language models which is also easy to automatically score. This evaluation is derived via a novel, unifying framework for evaluations over arbitrarily long contexts which measure the model's ability to do more than retrieve a single piece of information from its context. The central idea of the Latent Structure Queries framework (LSQ) is to construct tasks which require a model to ``chisel away'' the irrelevant information in the context, revealing a latent structure in the context. To verify a model's understanding of this latent structure, we query the model for details of the structure. Using LSQ, we produce three diagnostic long-context evaluations across code and natural-language domains intended to provide a stronger signal of long-context language model capabilities. We perform evaluations on several state-of-the-art models and demonstrate both that a) the proposed evaluations are high-signal and b) that there is significant room for improvement in synthesizing long-context information.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 16 of its 23 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

16 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack14 Jun
  2. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models19 Feb

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  5. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  6. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  7. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  8. 2018How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarkscited by 86
  9. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  10. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  11. 2016Revisiting Summarization Evaluation for Scientific Articlescited by 31
  12. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27
2 more
  1. 2024Exploring and Benchmarking the Planning Capabilities of Large Language Modelscited by 19
  2. 2024Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 34; 2026: 45.

85 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  3. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  4. 2026Gemma 4 Technical Reportcited by 187
  5. 2026MiMo-V2-Flash Technical Reportcited by 163
  6. 2025BIG-Bench Extra Hardcited by 123
  7. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  8. 2025Recursive Language Modelscited by 94
  9. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  10. 2025Glyph: Scaling Context Windows via Visual-Text Compressioncited by 49
  11. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 33
  12. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
38 more
  1. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  2. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 27
  3. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
  4. 2024On the Way to LLM Personalization: Learning to Remember User Conversationscited by 21
  5. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluationcited by 18
  6. 2025Unable to Forget: Proactive Interference Reveals Working Memory Limits in LLMs Beyond Context Lengthcited by 18
  7. 2025A Survey on LLM Mid-Trainingcited by 18
  8. 2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 17
  9. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
  10. 2025Episodic Memories Generation and Evaluation Benchmark for Large Language Modelscited by 14
  11. 2025Shifting Long-Context LLMs Research from Input to Outputcited by 13
  12. 2025T5Gemma 2: Seeing, Reading, and Understanding Longercited by 12
  13. 2026LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growthcited by 12
  14. 2025TokenButler: Token Importance is Predictablecited by 11
  15. 2025VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?cited by 11
  16. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  17. 2025AbsenceBench: Language Models Can't Tell What's Missingcited by 8
  18. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  19. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 8
  20. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 7
  21. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  22. 2025A Controllable Examination for Long-Context Language Modelscited by 7
  23. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6
  24. 2026OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactionscited by 6
  25. 2026SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generationcited by 6
  26. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  27. 2026Sabi'a-4 Technical Reportcited by 5
  28. 2026LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoningcited by 5
  29. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 4
  30. 2026Nexus : An Agentic Framework for Time Series Forecastingcited by 4
  31. 2026GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignmentcited by 4
  32. 2026FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attentioncited by 4
  33. 2026Sakana Fugu Technical Reportcited by 4
  34. 2025RELIC: Evaluating Complex Reasoning via the Recognition of Languages In-Contextcited by 3
  35. 2025CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Densitycited by 3
  36. 2026Learning When to Attend: Conditional Memory Access for Long-Context LLMscited by 3
  37. 2024Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAGcited by 2
  38. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 2

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
  2. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  3. 2026LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
  4. 2024Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps
  5. 2025Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning
  6. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
  7. 2025Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation
  8. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering