# CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval

arXiv: https://arxiv.org/abs/2411.12644 · HTML: https://mlsanity.com/paper/2411.12644/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 83 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 30 of 58)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)

## Built on by (most cited first, 30 of 34)

- [jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval](https://mlsanity.com/paper/2506.18902/) (2025)
- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches](https://mlsanity.com/paper/2510.04905/) (2025)
- [SweRank: Software Issue Localization with Code Ranking](https://mlsanity.com/paper/2505.07849/) (2025)
- [The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements](https://mlsanity.com/paper/2506.22419/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Bootstrapping Task Spaces for Self-Improvement](https://mlsanity.com/paper/2509.04575/) (2025)
- [CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges](https://mlsanity.com/paper/2603.11863/) (2026)
- [CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs](https://mlsanity.com/paper/2506.11059/) (2025)
- [Exploiting Leaderboards for Large-Scale Distribution of Malicious Models](https://mlsanity.com/paper/2507.08983/) (2025)
- [CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming](https://mlsanity.com/paper/2505.12925/) (2025)
- [C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling](https://mlsanity.com/paper/2512.21332/) (2025)
- [ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation](https://mlsanity.com/paper/2507.12674/) (2025)
- [AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming](https://mlsanity.com/paper/2507.15378/) (2025)
- [Utilizing Training Data to Improve LLM Reasoning for Tabular Understanding](https://mlsanity.com/paper/2508.18676/) (2025)
- [GAPrune: Gradient-Alignment Pruning for Domain-Aware Embeddings](https://mlsanity.com/paper/2509.10844/) (2025)
- [On the Challenges and Opportunities of Learned Sparse Retrieval for Code](https://mlsanity.com/paper/2603.22008/) (2026)
- [Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development](https://mlsanity.com/paper/2604.06793/) (2026)
- [SWE-QA: A Dataset and Benchmark for Complex Code Understanding](https://mlsanity.com/paper/2604.24814/) (2026)
- [The "I Don't Know" Filter: Enhancing Agentic Reliability in Function Calling](https://mlsanity.com/paper/2607.04034/) (2026)
- [MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization](https://mlsanity.com/paper/2607.15205/) (2026)
- [CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement](https://mlsanity.com/paper/2505.19757/) (2025)
- [OmniLLP: Enhancing LLM-based Log Level Prediction with Context-Aware Retrieval](https://mlsanity.com/paper/2508.08545/) (2025)
- [Cross-modal Retrieval Models for Stripped Binary Analysis](https://mlsanity.com/paper/2512.10393/) (2025)
- [LoRACode: LoRA Adapters for Code Embeddings](https://mlsanity.com/paper/2503.05315/) (2025)
- [Evaluating Out-of-Distribution Robustness in Graph-Based Android Malware Classification: A New Principled Benchmark](https://mlsanity.com/paper/2508.06734/) (2025)
- [On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detection](https://mlsanity.com/paper/2510.27675/) (2025)
- [CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval](https://mlsanity.com/paper/2604.15663/) (2026)
- [Do not copy and paste! Rewriting strategies for code retrieval](https://mlsanity.com/paper/2605.08299/) (2026)
- [Long Live the Librarian! A Persistent Search Sub-Agent for Energy-Efficient Multi-Agent Software Engineering Systems](https://mlsanity.com/paper/2605.27787/) (2026)

## Abstract (arXiv)

Despite the success of text retrieval in many NLP tasks, code retrieval remains a largely underexplored area. Most text retrieval systems are tailored for natural language queries, often neglecting the specific challenges of retrieving code. This gap leaves existing models unable to effectively capture the diversity of programming languages and tasks across different domains, highlighting the need for more focused research in code retrieval. To address this, we introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework, enhancing model generalizability and retrieval performance. Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark. In addition to excelling in code retrieval, our models demonstrate competitive performance on the widely adopted BeIR text retrieval benchmark, offering versatility across domains. Experimental results demonstrate that improving retrieval performance significantly enhances end-to-end Retrieval-Augmented Generation (RAG) performance for code-related tasks.