mlsanityData to 7 Oct 2026

CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion

arXiv 2310.11248 ↗First posted 17 Oct 2023cs.LG cs.CL cs.SEcited by 181

Not on the map. Why?

Cited by 36 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion tasks within a single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly. To fill in this gap, we propose CrossCodeEval, a diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CrossCodeEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose a straightforward yet efficient static-analysis-based approach to pinpoint the use of cross-file context within the current file. Extensive experiments on state-of-the-art code language models like CodeGen and StarCoder demonstrate that CrossCodeEval is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements, the pinnacle of performance remains notably unattained even with the highest-performing model, indicating that CrossCodeEval is also capable of assessing model's capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CrossCodeEval can also be used to measure the capability of code retrievers.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

32 of 44 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  8. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
20 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  4. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  5. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  6. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  7. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  8. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  9. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  10. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  11. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  12. 2023SantaCoder: don't reach for the stars!cited by 169
  13. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  14. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  15. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  16. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  17. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  18. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 64
  19. 2021Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchycited by 14
  20. 2023Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 55; 2025: 61; 2026: 62.

181 papers.

Sort:

By area:

  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  3. 2024A Survey on Large Language Models for Code Generationcited by 646
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  5. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  6. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  7. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  8. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  9. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  10. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  11. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  12. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 89
38 more
  1. 2024RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graphcited by 88
  2. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  3. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  4. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  5. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
  6. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  7. 2024Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Explorationcited by 58
  8. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
  9. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  10. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  11. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  12. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  13. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46
  14. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedbackcited by 45
  15. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
  16. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 40
  17. 2024Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completioncited by 40
  18. 2024GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Modelcited by 40
  19. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  20. 2024RepoQA: Evaluating Long Context Code Understandingcited by 35
  21. 2024SWE-bench-java: A GitHub Issue Resolving Benchmark for Javacited by 34
  22. 2024EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluationscited by 33
  23. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  24. 2024Test-Driven Development for Code Generationcited by 32
  25. 2026SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolvingcited by 30
  26. 2024Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Modelscited by 29
  27. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfacescited by 29
  28. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  29. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 27
  30. 2025Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Taskscited by 27
  31. 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 25
  32. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 25
  33. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  34. 2024AutoDev: Automated AI-Driven Developmentcited by 24
  35. 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 24
  36. 2024Unsupervised Evaluation of Code LLMs with Round-Trip Correctnesscited by 23
  37. 2025What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyondcited by 22
  38. 2024On the Impacts of Contexts on Repository-Level Code Generationcited by 21

50 most cited of 181 All 181 (JSON)