# Challenge on Optimization of Context Collection for Code Completion

arXiv: https://arxiv.org/abs/2510.04349 · HTML: https://mlsanity.com/paper/2510.04349/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [Repoformer: Selective Retrieval for Repository-Level Code Completion](https://mlsanity.com/paper/2403.10059/) (2024)
- [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022)
- [Long Code Arena: a Set of Benchmarks for Long-Context Code Models](https://mlsanity.com/paper/2406.11612/) (2024)
- [RLCoder: Reinforcement Learning for Repository-Level Code Completion](https://mlsanity.com/paper/2407.19487/) (2024)
- [EvalAI: Towards Better Evaluation Systems for AI Agents](https://mlsanity.com/paper/1902.03570/) (2019)
- [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023)
- [RepoHyper: Search-Expand-Refine on Semantic Graphs for Repository-Level Code Completion](https://mlsanity.com/paper/2403.06095/) (2024)
- [Toloka Visual Question Answering Benchmark](https://mlsanity.com/paper/2309.16511/) (2023)

## Abstract (arXiv)

The rapid advancement of workflows and methods for software engineering using AI emphasizes the need for a systematic evaluation and analysis of their ability to leverage information from entire projects, particularly in large code bases. In this challenge on optimization of context collection for code completion, organized by JetBrains in collaboration with Mistral AI as part of the ASE 2025 conference, participants developed efficient mechanisms for collecting context from source code repositories to improve fill-in-the-middle code completions for Python and Kotlin. We constructed a large dataset of real-world code in these two programming languages using permissively licensed open-source projects. The submissions were evaluated based on their ability to maximize completion quality for multiple state-of-the-art neural models using the chrF metric. During the public phase of the competition, nineteen teams submitted solutions to the Python track and eight teams submitted solutions to the Kotlin track. In the private phase, six teams competed, of which five submitted papers to the workshop.