# CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding

arXiv: https://arxiv.org/abs/2606.11864 · HTML: https://mlsanity.com/paper/2606.11864/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 29 of 29)

- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [LocAgent: Graph-Guided LLM Agents for Code Localization](https://mlsanity.com/paper/2503.09089/) (2025)
- [jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval](https://mlsanity.com/paper/2506.18902/) (2025)
- [CoSQA: 20,000+ Web Queries for Code Search and Question Answering](https://mlsanity.com/paper/2105.13239/) (2021)
- [CoIR: A Comprehensive Benchmark for Code Information Retrieval Models](https://mlsanity.com/paper/2407.02883/) (2024)
- [CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation](https://mlsanity.com/paper/2310.04951/) (2023)
- [CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking](https://mlsanity.com/paper/2412.01007/) (2024)
- [ContextBench: A Benchmark for Context Retrieval in Coding Agents](https://mlsanity.com/paper/2602.05892/) (2026)
- [SweRank: Software Issue Localization with Code Ranking](https://mlsanity.com/paper/2505.07849/) (2025)
- [FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents](https://mlsanity.com/paper/2504.13128/) (2025)
- [Efficient Code Embeddings from Code Generation Models](https://mlsanity.com/paper/2508.21290/) (2025)
- [F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2603.19223/) (2026)
- [CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming](https://mlsanity.com/paper/2505.12925/) (2025)
- [C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling](https://mlsanity.com/paper/2512.21332/) (2025)
- [CLARC: C/C++ Benchmark for Robust Code Search](https://mlsanity.com/paper/2603.04484/) (2026)

## Built on by (most cited first, 7 of 7)

- [Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents](https://mlsanity.com/paper/2607.24882/) (2026)
- [Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents](https://mlsanity.com/paper/2608.16295/) (2026)
- [The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks](https://mlsanity.com/paper/2608.16630/) (2026)
- [VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval](https://mlsanity.com/paper/2608.17889/) (2026)
- [Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale](https://mlsanity.com/paper/2609.15939/) (2026)
- [The Missing Complement: State-Conditioned Minimal Sufficient Evidence for Coding Agents](https://mlsanity.com/paper/2609.20050/) (2026)
- [StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agents](https://mlsanity.com/paper/2609.36319/) (2026)

## Abstract (arXiv)

Code retrieval is becoming central to coding agents, but agentic coding requires more than matching a natural-language query to an isolated snippet. Given a user request, a coding agent needs to navigate a concrete repository state, locate relevant files and functions, gather supporting context, and filter similar in-repository distractors. Existing code retrieval benchmarks mainly evaluate docstring-to-function or snippet-level matching, thereby missing this requirement-driven repository search problem. To address this gap, we introduce CORE-Bench, a comprehensive benchmark for code retrieval in the era of agentic coding. CORE-Bench evaluates code retrieval ability at three levels: code understanding, issue-to-edit localization, and broader context retrieval. Built from curated code-search tasks and SWE-bench-series instances, CORE-Bench contains over 180K queries and 106K broader-context relevance labels. Experiments with representative embedding models show a sharp drop from traditional code search to code retrieval in agentic coding settings. Simple supervised fine-tuning of existing embedding models significantly improves performance in this setting, suggesting substantial room for further progress.