# Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities

arXiv: https://arxiv.org/abs/2511.02817 · HTML: https://mlsanity.com/paper/2511.02817/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 63 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Builds on (most cited first, 30 of 40)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://mlsanity.com/paper/2502.05167/) (2025)
- [HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly](https://mlsanity.com/paper/2410.02694/) (2024)
- [Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries](https://mlsanity.com/paper/2409.12640/) (2024)
- [LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion](https://mlsanity.com/paper/2501.15089/) (2025)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence](https://mlsanity.com/paper/2210.07109/) (2022)
- [NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities](https://mlsanity.com/paper/2407.11963/) (2024)
- [Over-Reasoning and Redundant Calculation of Large Language Models](https://mlsanity.com/paper/2401.11467/) (2024)
- [Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP](https://mlsanity.com/paper/2407.00402/) (2024)
- [CALYPSO: LLMs as Dungeon Masters' Assistants](https://mlsanity.com/paper/2308.07540/) (2023)
- [MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents](https://mlsanity.com/paper/2508.11133/) (2025)
- [Database Reasoning Over Text](https://mlsanity.com/paper/2106.01074/) (2021)
- [FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information](https://mlsanity.com/paper/2305.01528/) (2023)
- [This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models](https://mlsanity.com/paper/2310.15941/) (2023)
- [Detecting Label Errors by using Pre-Trained Language Models](https://mlsanity.com/paper/2205.12702/) (2022)

## Built on by (most cited first, 30 of 40)

- [Recursive Language Models](https://mlsanity.com/paper/2512.24601/) (2025)
- [τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge](https://mlsanity.com/paper/2603.04370/) (2026)
- [Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction](https://mlsanity.com/paper/2512.14865/) (2025)
- [LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth](https://mlsanity.com/paper/2602.07962/) (2026)
- [Prime Agent: A Self-Improving RLM Harness](https://mlsanity.com/paper/2608.23552/) (2026)
- [Coding Agents are Effective Long-Context Processors](https://mlsanity.com/paper/2603.20432/) (2026)
- [Decentralized Multi-Agent Systems with Shared Context](https://mlsanity.com/paper/2606.10662/) (2026)
- [GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment](https://mlsanity.com/paper/2605.19577/) (2026)
- [ReCA: Multi-Shot Long Video Extrapolation via Recursive Context Allocation](https://mlsanity.com/paper/2605.26525/) (2026)
- [Recursive Agent Optimization](https://mlsanity.com/paper/2605.06639/) (2026)
- [LCM: Lossless Context Management](https://mlsanity.com/paper/2605.04050/) (2026)
- [PEEK: Context Map as an Orientation Cache for Long-Context LLM Agents](https://mlsanity.com/paper/2605.19932/) (2026)
- [Think, But Don't Overthink: Reproducing Recursive Language Models](https://mlsanity.com/paper/2603.02615/) (2026)
- [Recursive Language Models Meet Uncertainty: The Surprising Effectiveness of Self-Reflective Program Search for Long Context](https://mlsanity.com/paper/2603.15653/) (2026)
- [Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length](https://mlsanity.com/paper/2603.22608/) (2026)
- [SLIDERS: Systematic Reviews via Automated Evidence Synthesis and Reconciliation](https://mlsanity.com/paper/2604.22294/) (2026)
- [Deep Reasoning in General Purpose Agents via Structured Meta-Cognition](https://mlsanity.com/paper/2605.11388/) (2026)
- [ATLAS: All-round Testing of Long-context Abilities across Scales](https://mlsanity.com/paper/2605.28079/) (2026)
- [SWE-Pruner Pro: The Coder LLM Already Knows What to Prune](https://mlsanity.com/paper/2607.18213/) (2026)
- [LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing](https://mlsanity.com/paper/2608.01662/) (2026)
- [Probing How Scalable Table Data Enhances General Long-Context Reasoning](https://mlsanity.com/paper/2603.21719/) (2026)
- [Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents](https://mlsanity.com/paper/2605.21347/) (2026)
- [Mergeable Model-Side Aggregation States for Long-Context Language Models](https://mlsanity.com/paper/2607.26448/) (2026)
- [EnSI-RAG: Entity-Structure-Indexed Retrieval-Augmented Generation for Long-Document Question Answering](https://mlsanity.com/paper/2608.21252/) (2026)
- [Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning](https://mlsanity.com/paper/2609.38147/) (2026)
- [Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition](https://mlsanity.com/paper/2602.19001/) (2026)
- [π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2604.05114/) (2026)
- [Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media](https://mlsanity.com/paper/2605.21338/) (2026)
- [Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks](https://mlsanity.com/paper/2608.03297/) (2026)
- [Chained Recursive Language Models for Multi-Iteration Reasoning](https://mlsanity.com/paper/2608.05124/) (2026)

## Abstract (arXiv)

As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. While several carefully designed long-context evaluations have recently been released, these evaluations tend to rely on retrieval from one or more sections of the context, which allows nearly all of the context tokens to be disregarded as noise. This represents only one type of task that might be performed with long context. We introduce Oolong, a benchmark of long-context reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions. Oolong is separated into two task sets: Oolong-synth, a set of naturalistic synthetic tasks, where we can easily ablate components of the reasoning problem; and Oolong-real, a downstream setting which requires reasoning over real-world conversational data. Oolong requires models to reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations. Even frontier models struggle on Oolong, with GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text.