# Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

arXiv: https://arxiv.org/abs/2409.12640 · HTML: https://mlsanity.com/paper/2409.12640/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://mlsanity.com/paper/1808.04926/) (2018)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [FABLES: Evaluating faithfulness and content selection in book-length summarization](https://mlsanity.com/paper/2404.01261/) (2024)
- [Revisiting Summarization Evaluation for Scientific Articles](https://mlsanity.com/paper/1604.00400/) (2016)
- [Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP](https://mlsanity.com/paper/2407.00402/) (2024)
- [Exploring and Benchmarking the Planning Capabilities of Large Language Models](https://mlsanity.com/paper/2406.13094/) (2024)
- [Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation](https://mlsanity.com/paper/2406.00179/) (2024)

## Built on by (most cited first, 30 of 88)

- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [Gemma 4 Technical Report](https://mlsanity.com/paper/2607.02770/) (2026)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://mlsanity.com/paper/2502.05167/) (2025)
- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [Recursive Language Models](https://mlsanity.com/paper/2512.24601/) (2025)
- [SCBench: A KV Cache-Centric Analysis of Long-Context Methods](https://mlsanity.com/paper/2412.10319/) (2024)
- [Glyph: Scaling Context Windows via Visual-Text Compression](https://mlsanity.com/paper/2510.17800/) (2025)
- [Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities](https://mlsanity.com/paper/2511.02817/) (2025)
- [Large Language Models Can Self-Improve in Long-context Reasoning](https://mlsanity.com/paper/2411.08147/) (2024)
- [LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation](https://mlsanity.com/paper/2501.05414/) (2025)
- [SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models](https://mlsanity.com/paper/2509.23863/) (2025)
- [When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research](https://mlsanity.com/paper/2505.11855/) (2025)
- [On the Way to LLM Personalization: Learning to Remember User Conversations](https://mlsanity.com/paper/2411.13405/) (2024)
- [A Survey on LLM Mid-Training](https://mlsanity.com/paper/2510.23081/) (2025)
- [Unable to Forget: Proactive Interference Reveals Working Memory Limits in LLMs Beyond Context Length](https://mlsanity.com/paper/2506.08184/) (2025)
- [Evaluating LLM Metrics Through Real-World Capabilities](https://mlsanity.com/paper/2505.08253/) (2025)
- [GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation](https://mlsanity.com/paper/2506.07671/) (2025)
- [Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management](https://mlsanity.com/paper/2508.04664/) (2025)
- [T5Gemma 2: Seeing, Reading, and Understanding Longer](https://mlsanity.com/paper/2512.14856/) (2025)
- [Episodic Memories Generation and Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2501.13121/) (2025)
- [Shifting Long-Context LLMs Research from Input to Output](https://mlsanity.com/paper/2503.04723/) (2025)
- [LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth](https://mlsanity.com/paper/2602.07962/) (2026)
- [TokenButler: Token Importance is Predictable](https://mlsanity.com/paper/2503.07518/) (2025)
- [VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?](https://mlsanity.com/paper/2512.15649/) (2025)
- [WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale](https://mlsanity.com/paper/2502.16684/) (2025)
- [AbsenceBench: Language Models Can't Tell What's Missing](https://mlsanity.com/paper/2506.11440/) (2025)
- [AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts](https://mlsanity.com/paper/2601.20730/) (2026)

## Abstract (arXiv)

We introduce Michelangelo: a minimal, synthetic, and unleaked long-context reasoning evaluation for large language models which is also easy to automatically score. This evaluation is derived via a novel, unifying framework for evaluations over arbitrarily long contexts which measure the model's ability to do more than retrieve a single piece of information from its context. The central idea of the Latent Structure Queries framework (LSQ) is to construct tasks which require a model to ``chisel away'' the irrelevant information in the context, revealing a latent structure in the context. To verify a model's understanding of this latent structure, we query the model for details of the structure. Using LSQ, we produce three diagnostic long-context evaluations across code and natural-language domains intended to provide a stronger signal of long-context language model capabilities. We perform evaluations on several state-of-the-art models and demonstrate both that a) the proposed evaluations are high-signal and b) that there is significant room for improvement in synthesizing long-context information.