mlsanityData to 14 Oct 2026

Long Code Arena: a Set of Benchmarks for Long-Context Code Models

arXiv 2406.11612 ↗First posted 17 Jun 2024cs.LG cs.AI cs.IRcited by 50

Not on the map. Why?

Abstract (arXiv)

Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes have increased by orders of magnitude over the last few years. However, there is a shortage of benchmarks for code processing that go beyond a single file of context, while the most popular ones are limited to a single method. With this work, we aim to close this gap by introducing Long Code Arena, a suite of six benchmarks for code processing tasks that require project-wide context. These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization. For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions based on popular LLMs to showcase the usage of the dataset and to simplify adoption by other researchers. We publish the benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace Hub for all the datasets, and link to the GitHub repository with baselines: https://huggingface.co/spaces/JetBrains-Research/long-code-arena.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 33 of its 64 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

33 of 64 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2023Mistral 7Bcited by 3,164
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2024Mixtral of Expertscited by 1,823
21 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  3. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  4. 2021Measuring Coding Challenge Competence With APPScited by 902
  5. 2023StarCoder: may the source be with you!cited by 799
  6. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  10. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  11. 2023Active Retrieval Augmented Generationcited by 516
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  13. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  14. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  15. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  16. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  17. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  18. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  19. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 70
  20. 2023Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecturecited by 63
  21. 2024CommitBench: A Benchmark for Commit Message Generationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 28; 2026: 17.

50 papers.

Sort:

By area:

  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  2. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  3. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 39
  4. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  5. 2025Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approachescited by 23
  6. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  7. 2025LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineeringcited by 16
  8. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  9. 2026OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Codingcited by 15
  10. 2025SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositoriescited by 14
  11. 2025A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Codecited by 14
  12. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMscited by 13
38 more
  1. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  2. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  3. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodingcited by 10
  4. 2024RepairBench: Leaderboard of Frontier Models for Program Repaircited by 9
  5. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 8
  6. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  7. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  8. 2026Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systemscited by 6
  9. 2025YABLoCo: Yet Another Benchmark for Long Context Code Generationcited by 5
  10. 2025Intuition to Evidence: Measuring AI's True Impact on Developer Productivitycited by 5
  11. 2026Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compressioncited by 5
  12. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  13. 2025How can we assess human-agent interactions? Case studies in software agent designcited by 4
  14. 2026ORCA-bench: How Ready Are Language Model Agents for Oncall?cited by 4
  15. 2025RSQ: Learning from Important Tokens Leads to Better Quantized LLMscited by 3
  16. 2025CoRet: Improved Retriever for Code Editingcited by 3
  17. 2025Practical Code RAG at Scale: Task-Aware Retrieval Design Choices under Compute Budgetscited by 3
  18. 2026URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Modelscited by 3
  19. 2025CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Modelscited by 2
  20. 2025On Pretraining for Project-Level Code Completioncited by 1
  21. 2026CodeComp: Structural KV Cache Compression for Agentic Codingcited by 1
  22. 2024Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settingscited by 0
  23. 2024WarriorCoder: Learning from Expert Battles to Augment Code Large Language Modelscited by 0
  24. 2025MutaGReP: Execution-Free Repository-Grounded Plan Search for Code-Usecited by 0
  25. 2025TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEscited by 0
  26. 2025Challenge on Optimization of Context Collection for Code Completioncited by 0
  27. 2025Diff-XYZ: A Benchmark for Evaluating Diff Understandingcited by 0
  28. 2025LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understandingcited by 0
  29. 2025Reformulate, Retrieve, Localize: Agents for Repository-Level Bug Localizationcited by 0
  30. 2026TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenancecited by 0
  31. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demandscited by 0
  32. 2026FlashRT: Towards Computationally and Memory Efficient Red-Teaming for Prompt Injection and Knowledge Corruptioncited by 0
  33. 2026Improving BM25 Code Retrieval Under Fixed Generic Tokenization: Adaptive q-Log Odds as a Drop-In BM25 Fixcited by 0
  34. 2026LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Modelscited by 0
  35. 2026Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBenchcited by 0
  36. 2026Retrieval-Oriented Code Representations in Agentic Bug Localizationcited by 0
  37. 2026Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformerscited by 0
  38. 2026FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decodingcited by 0