mlsanityData to 7 Oct 2026

RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems

arXiv 2306.03091 ↗First posted 5 Jun 2023cs.CL cs.AI cs.SEcited by 346

Not on the map. Why?

Cited by 99 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have greatly advanced code auto-completion systems, with a potential for substantial productivity enhancements for developers. However, current benchmarks mainly focus on single-file tasks, leaving an assessment gap for more complex, real-world, multi-file programming scenarios. To fill this gap, we introduce RepoBench, a new benchmark specifically designed for evaluating repository-level code auto-completion systems. RepoBench supports both Python and Java and consists of three interconnected evaluation tasks: RepoBench-R (Retrieval), RepoBench-C (Code Completion), and RepoBench-P (Pipeline). Each task respectively measures the system's ability to retrieve the most relevant code snippets from other files as cross-file context, predict the next line of code with cross-file and in-file context, and handle complex tasks that require a combination of both retrieval and next-line prediction. RepoBench aims to facilitate a more complete comparison of performance and encouraging continuous improvement in auto-completion systems. RepoBench is publicly available at https://github.com/Leolty/repobench.

Key facts. Cited by 346 papers in the map (data through 7 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

33 of 49 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
21 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  4. 2023StarCoder: may the source be with you!cited by 790
  5. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  6. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  7. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  8. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  9. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  10. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  12. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  13. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  14. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  15. 2023SantaCoder: don't reach for the stars!cited by 169
  16. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  17. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  18. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  19. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 64
  20. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  21. 2023Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Studycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 86; 2025: 112; 2026: 139.

346 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024A Survey on Large Language Models for Code Generationcited by 646
  4. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  6. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  7. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  8. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  9. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  10. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  11. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  12. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
38 more
  1. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  3. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  4. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  5. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  7. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  8. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  9. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  10. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  11. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  12. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  13. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  14. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  15. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  16. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  17. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  18. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  19. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 63
  20. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  21. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  22. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
  23. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  24. 2024Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Explorationcited by 58
  25. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  26. 2025FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementationcited by 57
  27. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 56
  28. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  29. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  30. 2024Self-evolving Agents with reflective and memory-augmented abilitiescited by 52
  31. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  32. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  33. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48
  34. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  35. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedbackcited by 45
  36. 2024AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodologycited by 45
  37. 2025LLM Agents Making Agent Toolscited by 45
  38. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43

50 most cited of 346 All 346 (JSON)