# ContextBench: A Benchmark for Context Retrieval in Coding Agents arXiv: https://arxiv.org/abs/2602.05892 ยท HTML: https://mlsanity.com/paper/2602.05892/ ## Key facts - Cited by 25 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 25 of its 33 references that are in the map. - First posted on arXiv 5 Feb 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 25 of 25) - [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025) - [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025) - [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026) - [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024) - [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023) - [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025) - [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025) - [Where LLM Agents Fail and How They can Learn From Failures](https://mlsanity.com/paper/2509.25370/) (2025) - [SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?](https://mlsanity.com/paper/2502.12115/) (2025) - [Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?](https://mlsanity.com/paper/2511.13646/) (2025) - [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025) - [SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents](https://mlsanity.com/paper/2504.08703/) (2025) - [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025) - [SWE-bench Goes Live!](https://mlsanity.com/paper/2505.23419/) (2025) - [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024) - [SWE-Exp: Experience-Driven Software Issue Resolution](https://mlsanity.com/paper/2507.23361/) (2025) - [Doc2Feat-Bench: Evaluating Documentation-Driven Feature Addition](https://mlsanity.com/paper/2507.18130/) (2025) - [An Empirical Study on Failures in Automated Issue Solving](https://mlsanity.com/paper/2509.13941/) (2025) - [Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios](https://mlsanity.com/paper/2503.12374/) (2025) - [Prometheus: Towards Long-Horizon Codebase Navigation for Repository-Level Problem Solving](https://mlsanity.com/paper/2507.19942/) (2025) - [Lingxi: Repository-Level Issue Resolution Framework Enhanced by Procedural Knowledge Guided Scaling](https://mlsanity.com/paper/2510.11838/) (2025) - [CodeClash: Benchmarking Goal-Oriented Software Engineering](https://mlsanity.com/paper/2511.00839/) (2025) - [Detecting Silent Failures in Multi-Agentic AI Trajectories](https://mlsanity.com/paper/2511.04032/) (2025) ## Built on by (most cited first, 25 of 25) - [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026) - [SWE-Explore: Benchmarking How Coding Agents Explore Repositories](https://mlsanity.com/paper/2606.07297/) (2026) - [From Question Answering to Task Completion: A Survey on Agent System and Harness Design](https://mlsanity.com/paper/2606.20683/) (2026) - [CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding](https://mlsanity.com/paper/2606.11864/) (2026) - [DeonticBench: A Benchmark for Reasoning over Rules](https://mlsanity.com/paper/2604.04443/) (2026) - [Quantifying Overclaiming Propensity in Frontier LLM Agents](https://mlsanity.com/paper/2609.20812/) (2026) - [Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Tasks](https://mlsanity.com/paper/2606.02875/) (2026) - [What Context Does a Coding Agent Actually Need to Act?](https://mlsanity.com/paper/2607.09691/) (2026) - [The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior](https://mlsanity.com/paper/2608.11386/) (2026) - [Scaling Laws for Agent Harnesses via Effective Feedback Compute](https://mlsanity.com/paper/2605.29682/) (2026) - [OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling](https://mlsanity.com/paper/2608.05141/) (2026) - [Decision-Aware Memory Cards: Counterfactual-Inspired Context Selection and Compression for Tool-Using LLM Agents](https://mlsanity.com/paper/2606.08151/) (2026) - [The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages](https://mlsanity.com/paper/2607.22807/) (2026) - [Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents](https://mlsanity.com/paper/2608.16295/) (2026) - [Theory of Code Space: Do Code Agents Understand Software Architecture?](https://mlsanity.com/paper/2603.00601/) (2026) - [How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?](https://mlsanity.com/paper/2606.26346/) (2026) - [CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents](https://mlsanity.com/paper/2607.25431/) (2026) - [The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context](https://mlsanity.com/paper/2608.14838/) (2026) - [Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale](https://mlsanity.com/paper/2609.15939/) (2026) - [The Missing Complement: State-Conditioned Minimal Sufficient Evidence for Coding Agents](https://mlsanity.com/paper/2609.20050/) (2026) - [Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer](https://mlsanity.com/paper/2609.31587/) (2026) - [Evaluating Name-Only Directory Routing for One-Shot Code Search](https://mlsanity.com/paper/2609.35918/) (2026) - [StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agents](https://mlsanity.com/paper/2609.36319/) (2026) - [SWEPatchTest: Evaluating Software Tests Beyond Reproduction](https://mlsanity.com/paper/2610.18004/) (2026) - [What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports](https://mlsanity.com/paper/2610.20003/) (2026) ## Abstract (arXiv) LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving. We introduce ContextBench, a process-oriented evaluation of context retrieval in coding agents. ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. We further implement an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution. Using ContextBench, we evaluate four frontier LLMs and five coding agents. Our results show that sophisticated agent scaffolding yields only marginal gains in context retrieval ("The Bitter Lesson" of coding agents), LLMs consistently favor recall over precision, and substantial gaps exist between explored and utilized context. ContextBench augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks.