# Towards A Generalist Code Embedding Model Based On Massive Data Synthesis

arXiv: https://arxiv.org/abs/2505.12697 · HTML: https://mlsanity.com/paper/2505.12697/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RAG for code, one of 15 directions in Retrieval-augmented generation.
- First posted on arXiv 19 May 2025.

## Placed in

- Retrieval-augmented generation / [RAG for code](https://mlsanity.com/rag/rag-for-code/)

## Built on by (most cited first, 8 of 8)

- [ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval](https://mlsanity.com/paper/2510.08252/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling](https://mlsanity.com/paper/2512.21332/) (2025)
- [On the Challenges and Opportunities of Learned Sparse Retrieval for Code](https://mlsanity.com/paper/2603.22008/) (2026)
- [Evaluating Language Models on Cross-Language Code Functional Equivalence](https://mlsanity.com/paper/2608.23961/) (2026)
- [Pull Requests as a Training Signal for Repo-Level Code Editing](https://mlsanity.com/paper/2602.07457/) (2026)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [RTLCurator: Label-Efficient Data Curation for RTL Generation](https://mlsanity.com/paper/2607.29283/) (2026)

## Abstract (arXiv)

Code embedding models attract increasing attention due to the widespread popularity of retrieval-augmented generation (RAG) in software development. These models are expected to capture the rich semantic relationships inherent to code, which differ significantly from those found in text. However, existing models remain severely limited due to the scarcity of high-quality training data. In this work, we introduce CodeR (underlineCode underlineRetrieval), a state-of-the-art embedding model for general-purpose code retrieval. The superior performance of CodeR is built upon CodeR-Pile, a large-scale synthetic dataset constructed under the DRU (Diversity, Reliability, Usability) principle via a novel data synthesis pipeline. To optimize training effectiveness, we propose Annealing, a curriculum learning strategy that enables effective knowledge transfer across heterogeneous sources of data. We evaluate CodeR based on 16 diverse code retrieval tasks, where it significantly outperforms existing baselines and exhibits strong out-of-domain generalization performance. We have publicly released our code and the well-trained model to facilitate further research in this critical area. https://github.com/FlagOpen/FlagEmbedding/tree/master/research/BGECoder.