# LongCodeBench: Evaluating Coding LLMs at 1M Context Windows

arXiv: https://arxiv.org/abs/2505.07897 · HTML: https://mlsanity.com/paper/2505.07897/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 13 of 13)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [Long Code Arena: a Set of Benchmarks for Long-Context Code Models](https://mlsanity.com/paper/2406.11612/) (2024)
- [LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs](https://mlsanity.com/paper/2409.02076/) (2024)
- [Measuring The Impact Of Programming Language Distribution](https://mlsanity.com/paper/2302.01973/) (2023)

## Built on by (most cited first, 30 of 37)

- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [LongCodeZip: Compress Long Context for Code Language Models](https://mlsanity.com/paper/2510.00446/) (2025)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding](https://mlsanity.com/paper/2602.01785/) (2026)
- [SWE Context Bench: A Benchmark for Context Learning in Coding](https://mlsanity.com/paper/2602.08316/) (2026)
- [SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding](https://mlsanity.com/paper/2603.16124/) (2026)
- [Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compression](https://mlsanity.com/paper/2602.00746/) (2026)
- [KV Admission: Learning What to Write for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2512.17452/) (2025)
- [Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering](https://mlsanity.com/paper/2603.26567/) (2026)
- [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](https://mlsanity.com/paper/2605.17170/) (2026)
- [Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model](https://mlsanity.com/paper/2608.13867/) (2026)
- [Automating Database-Native Function Code Synthesis with LLMs](https://mlsanity.com/paper/2604.06231/) (2026)
- [Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning](https://mlsanity.com/paper/2605.15315/) (2026)
- [ATLAS: All-round Testing of Long-context Abilities across Scales](https://mlsanity.com/paper/2605.28079/) (2026)
- [SWE-Pruner Pro: The Coder LLM Already Knows What to Prune](https://mlsanity.com/paper/2607.18213/) (2026)
- [LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing](https://mlsanity.com/paper/2608.01662/) (2026)
- [Self-Evolving Coding Agents](https://mlsanity.com/paper/2608.03392/) (2026)
- [OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling](https://mlsanity.com/paper/2608.05141/) (2026)
- [Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection](https://mlsanity.com/paper/2602.03216/) (2026)
- [The Limits of Long-Context Reasoning in Automated Bug Fixing](https://mlsanity.com/paper/2602.16069/) (2026)
- [Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering](https://mlsanity.com/paper/2602.17183/) (2026)
- [FastCode: Fast and Cost-Efficient Code Understanding and Reasoning](https://mlsanity.com/paper/2603.01012/) (2026)
- [ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation](https://mlsanity.com/paper/2603.25770/) (2026)
- [A Study on the Impact of Fault localization Granularity for Repository-Scale Code Repair Tasks](https://mlsanity.com/paper/2604.00167/) (2026)
- [CodeComp: Structural KV Cache Compression for Agentic Coding](https://mlsanity.com/paper/2604.10235/) (2026)
- [Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study](https://mlsanity.com/paper/2608.01507/) (2026)
- [Flash-Fusion: Enabling Expressive, Low-Latency Queries on IoT Sensor Streams with LLMs](https://mlsanity.com/paper/2511.11885/) (2025)
- [Beyond Accuracy: Characterizing Code Comprehension Capabilities in (Large) Language Models](https://mlsanity.com/paper/2601.12951/) (2026)
- [PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models](https://mlsanity.com/paper/2606.09697/) (2026)
- [Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code](https://mlsanity.com/paper/2606.23690/) (2026)

## Abstract (arXiv)

Context lengths for models have grown rapidly, from thousands to millions of tokens in just a few years. The extreme context sizes of modern long-context models have made it difficult to construct realistic long-context benchmarks -- not only due to the cost of collecting million-context tasks but also in identifying realistic scenarios that require significant contexts. We identify code comprehension and repair as a natural testbed and challenge task for long-context models and introduce LongCodeBench (LCB), a benchmark to test LLM coding abilities in long-context scenarios. Our benchmark tests both the comprehension and repair capabilities of LCLMs in realistic and important settings by drawing from real-world GitHub issues and constructing QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. We carefully stratify the complexity of our benchmark, enabling us to evaluate models across different scales -- ranging from Qwen2.5 14B Instruct to Google's flagship Gemini model. We find that long-context remains a weakness for all models, with performance drops such as from 29% to 3% for Claude 3.5 Sonnet, or from 70.2% to 40% for Qwen2.5. The LCB dataset is available publicly at https://huggingface.co/datasets/Steefano/LCB and the codebase to replicate the work on this paper at https://github.com/Zteefano/long-code-bench.