# LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering

arXiv: https://arxiv.org/abs/2509.09614 · HTML: https://mlsanity.com/paper/2509.09614/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2025.

## Built on by (most cited first, 12 of 12)

- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering](https://mlsanity.com/paper/2511.13998/) (2025)
- [SWE Context Bench: A Benchmark for Context Learning in Coding](https://mlsanity.com/paper/2602.08316/) (2026)
- [AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?](https://mlsanity.com/paper/2510.07740/) (2025)
- [Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.29790/) (2026)
- [RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists](https://mlsanity.com/paper/2608.04783/) (2026)
- [Bridging Reasoning to Learning: Unmasking Illusions using Complexity Out of Distribution Generalization](https://mlsanity.com/paper/2510.06274/) (2025)
- [ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation](https://mlsanity.com/paper/2603.25770/) (2026)
- [StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns](https://mlsanity.com/paper/2606.19613/) (2026)
- [Position: Vector Prompt Interfaces Should Be Exposed to Enable Customization of Large Language Models](https://mlsanity.com/paper/2603.04292/) (2026)
- [Teaching LLMs Program Semantics via Symbolic Execution Traces](https://mlsanity.com/paper/2605.06184/) (2026)
- [Evolve Together, Fall into Place: How Social Scaffolding Shapes Self-Evolving Multi-Agent Systems](https://mlsanity.com/paper/2610.19799/) (2026)

## Abstract (arXiv)

The emergence of long-context language models with context windows extending to millions of tokens has created new opportunities for sophisticated code understanding and software development evaluation. We propose LoCoBench, a comprehensive benchmark specifically designed to evaluate long-context LLMs in realistic, complex software development scenarios. Unlike existing code evaluation benchmarks that focus on single-function completion or short-context tasks, LoCoBench addresses the critical evaluation gap for long-context capabilities that require understanding entire codebases, reasoning across multiple files, and maintaining architectural consistency across large-scale software systems. Our benchmark provides 8,000 evaluation scenarios systematically generated across 10 programming languages, with context lengths spanning 10K to 1M tokens, a 100x variation that enables precise assessment of long-context performance degradation in realistic software development settings. LoCoBench introduces 8 task categories that capture essential long-context capabilities: architectural understanding, cross-file refactoring, multi-session development, bug investigation, feature implementation, code comprehension, integration testing, and security analysis. Through a 5-phase pipeline, we create diverse, high-quality scenarios that challenge LLMs to reason about complex codebases at unprecedented scale. We introduce a comprehensive evaluation framework with 17 metrics across 4 dimensions, including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our evaluation of state-of-the-art long-context models reveals substantial performance gaps, demonstrating that long-context understanding in complex software development represents a significant unsolved challenge that demands more attention. LoCoBench is released at: https://github.com/SalesforceAIResearch/LoCoBench.