mlsanityData to 14 Oct 2026

LongCodeBench: Evaluating Coding LLMs at 1M Context Windows

arXiv 2505.07897 ↗First posted 12 May 2025cs.CL cs.AIcited by 37

Not on the map. Why?

Abstract (arXiv)

Context lengths for models have grown rapidly, from thousands to millions of tokens in just a few years. The extreme context sizes of modern long-context models have made it difficult to construct realistic long-context benchmarks -- not only due to the cost of collecting million-context tasks but also in identifying realistic scenarios that require significant contexts. We identify code comprehension and repair as a natural testbed and challenge task for long-context models and introduce LongCodeBench (LCB), a benchmark to test LLM coding abilities in long-context scenarios. Our benchmark tests both the comprehension and repair capabilities of LCLMs in realistic and important settings by drawing from real-world GitHub issues and constructing QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. We carefully stratify the complexity of our benchmark, enabling us to evaluate models across different scales -- ranging from Qwen2.5 14B Instruct to Google's flagship Gemini model. We find that long-context remains a weakness for all models, with performance drops such as from 29% to 3% for Claude 3.5 Sonnet, or from 70.2% to 40% for Qwen2.5. The LCB dataset is available publicly at https://huggingface.co/datasets/Steefano/LCB and the codebase to replicate the work on this paper at https://github.com/Zteefano/long-code-bench.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 13 of its 21 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

13 of 21 references.

References · 13

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  7. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  10. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  11. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 50
  12. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
1 more
  1. 2023Measuring The Impact Of Programming Language Distributioncited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 33.

37 papers.

Sort:

By area:

  1. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 44
  2. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 39
  3. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  4. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  5. 2026SWE Context Bench: A Benchmark for Context Learning in Codingcited by 9
  6. 2026SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understandingcited by 9
  7. 2026Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compressioncited by 5
  8. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  9. 2026Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answeringcited by 3
  10. 2026TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Taskscited by 3
  11. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 3
  12. 2026Automating Database-Native Function Code Synthesis with LLMscited by 2
25 more
  1. 2026Context Pruning for Coding Agents via Multi-Rubric Latent Reasoningcited by 2
  2. 2026ATLAS: All-round Testing of Long-context Abilities across Scalescited by 2
  3. 2026SWE-Pruner Pro: The Coder LLM Already Knows What to Prunecited by 2
  4. 2026LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexingcited by 2
  5. 2026Self-Evolving Coding Agentscited by 2
  6. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
  7. 2026Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selectioncited by 1
  8. 2026The Limits of Long-Context Reasoning in Automated Bug Fixingcited by 1
  9. 2026Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answeringcited by 1
  10. 2026FastCode: Fast and Cost-Efficient Code Understanding and Reasoningcited by 1
  11. 2026ReCUBE: Evaluating Repository-Level Context Utilization in Code Generationcited by 1
  12. 2026A Study on the Impact of Fault localization Granularity for Repository-Scale Code Repair Taskscited by 1
  13. 2026CodeComp: Structural KV Cache Compression for Agentic Codingcited by 1
  14. 2026Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Studycited by 1
  15. 2025Flash-Fusion: Enabling Expressive, Low-Latency Queries on IoT Sensor Streams with LLMscited by 0
  16. 2026Beyond Accuracy: Characterizing Code Comprehension Capabilities in (Large) Language Modelscited by 0
  17. 2026PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Modelscited by 0
  18. 2026Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Codecited by 0
  19. 2026The Spec Growth Engine: Spec-Anchored, Code-Coupled, Drift-Enforced Architecture for AI-Assisted Software Developmentcited by 0
  20. 2026CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understandingcited by 0
  21. 2026BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Trainingcited by 0
  22. 2026Just-In-Time Agent Memory with Runtime Agentic Researchcited by 0
  23. 2026LongSpark: Efficient speculative decoding with a fixed-cost parallel draftercited by 0
  24. 2026Complexity-Aware Evaluation of LLM Comprehensioncited by 0
  25. 2026CommunityKV: Efficient Long-Context Decoding via Graph Partitioningcited by 0