# LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth

arXiv: https://arxiv.org/abs/2602.07962 · HTML: https://mlsanity.com/paper/2602.07962/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 16 of 16)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://mlsanity.com/paper/2508.06600/) (2025)
- [Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries](https://mlsanity.com/paper/2409.12640/) (2024)
- [The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution](https://mlsanity.com/paper/2510.25726/) (2025)
- [The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents](https://mlsanity.com/paper/2511.03690/) (2025)
- [Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities](https://mlsanity.com/paper/2511.02817/) (2025)
- [MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use](https://mlsanity.com/paper/2509.24002/) (2025)
- [GEM: A Gym for Agentic LLMs](https://mlsanity.com/paper/2510.01051/) (2025)

## Built on by (most cited first, 14 of 14)

- [Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation](https://mlsanity.com/paper/2607.08938/) (2026)
- [Pseudo-Formalization for Automatic Proof Verification](https://mlsanity.com/paper/2605.20531/) (2026)
- [Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.29790/) (2026)
- [Context as an Environment: Programmatic Context Management for Long-Horizon Agents](https://mlsanity.com/paper/2608.21690/) (2026)
- [LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception](https://mlsanity.com/paper/2606.30005/) (2026)
- [View-oriented Conversation Compiler for Agent Trace Analysis](https://mlsanity.com/paper/2603.29678/) (2026)
- [HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following](https://mlsanity.com/paper/2607.25398/) (2026)
- [TRACER: Per-Tool Context Retention for LLM Agents via Consequence-Attributed Reinforcement Learning](https://mlsanity.com/paper/2608.29363/) (2026)
- [ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management](https://mlsanity.com/paper/2607.20764/) (2026)
- [Parsing the Stream: A Live Trace Model for Long-Horizon Agents and Their Observers](https://mlsanity.com/paper/2609.01466/) (2026)
- [Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation](https://mlsanity.com/paper/2609.26693/) (2026)
- [Beyond Memory: Harnessing Long-Horizon Agents with Explicit Belief States](https://mlsanity.com/paper/2610.01415/) (2026)
- [ProgAgent: Learning Tool Orchestration through Programmatic Training](https://mlsanity.com/paper/2610.14774/) (2026)
- [SkillHorizon: Constructing Long-Horizon Agent Tasks through Skill Composition](https://mlsanity.com/paper/2610.14966/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly capable of carrying out long-running, real-world tasks. However, as the amount of context grows, their reliability often deteriorates, a phenomenon known as "context rot". Existing long-context benchmarks primarily focus on single-step settings that evaluate a model's ability to retrieve information from a long snippet. In realistic scenarios, however, LLMs often need to act as agents that explore environments, follow instructions and plans, extract useful information, and predict correct actions under a dynamically growing context. To assess language agents in such settings, we introduce LOCA-bench (a benchmark for LOng-Context Agents). Given a task prompt, LOCA-bench leverages automated and scalable control of environment states to regulate the agent's context length. This design enables LOCA-bench to extend the context length potentially to infinity in a controlled way while keeping the underlying task semantics fixed. LOCA-bench evaluates language agents as a combination of models and scaffolds, including various context management strategies. While agent performance generally degrades as the environment states grow more complex, advanced context management techniques can substantially improve the overall success rate. We open-source LOCA-bench to provide a platform for evaluating models and scaffolds in long-context, agentic scenarios: https://github.com/hkust-nlp/LOCA-bench