# CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search

arXiv: https://arxiv.org/abs/2201.10866 · HTML: https://mlsanity.com/paper/2201.10866/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 30 of 32)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup](https://mlsanity.com/paper/2101.06983/) (2021)
- [Adversarial Retriever-Ranker for dense text retrieval](https://mlsanity.com/paper/2110.03611/) (2021)
- [Contrastive Code Representation Learning](https://mlsanity.com/paper/2007.04973/) (2020)
- [CoSQA: 20,000+ Web Queries for Code Search and Question Answering](https://mlsanity.com/paper/2105.13239/) (2021)
- [CoTexT: Multi-task Learning with Code-Text Transformer](https://mlsanity.com/paper/2105.08645/) (2021)
- [Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations](https://mlsanity.com/paper/2009.02731/) (2020)
- [Learning and Evaluating Contextual Embedding of Source Code](https://mlsanity.com/paper/2001.00059/) (2019)
- [Exploring Software Naturalness through Neural Language Models](https://mlsanity.com/paper/2006.12641/) (2020)
- [StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow](https://mlsanity.com/paper/1803.09371/) (2018)
- [Query Expansion Based on Crowd Knowledge for Code Search](https://mlsanity.com/paper/1703.01443/) (2017)
- [SCELMo: Source Code Embeddings from Language Models](https://mlsanity.com/paper/2004.13214/) (2020)

## Built on by (most cited first, 21 of 21)

- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking](https://mlsanity.com/paper/2412.01007/) (2024)
- [Code Representation Learning At Scale](https://mlsanity.com/paper/2402.01935/) (2024)
- [Improving Code Localization with Repository Memory](https://mlsanity.com/paper/2510.01003/) (2025)
- [cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree](https://mlsanity.com/paper/2506.15655/) (2025)
- [ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search](https://mlsanity.com/paper/2403.16702/) (2024)
- [GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding](https://mlsanity.com/paper/2311.09707/) (2023)
- [Building A Coding Assistant via the Retrieval-Augmented Language Model](https://mlsanity.com/paper/2410.16229/) (2024)
- [UniHGKR: Unified Instruction-aware Heterogeneous Knowledge Retrievers](https://mlsanity.com/paper/2410.20163/) (2024)
- [Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking](https://mlsanity.com/paper/2508.19558/) (2025)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Multi-task retriever fine-tuning for domain-specific and efficient RAG](https://mlsanity.com/paper/2501.04652/) (2025)
- [OASIS: Order-Augmented Strategy for Improved Code Search](https://mlsanity.com/paper/2503.08161/) (2025)
- [Zero-Shot Cross-Domain Code Search without Fine-Tuning](https://mlsanity.com/paper/2504.07740/) (2025)
- [SemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction](https://mlsanity.com/paper/2511.07584/) (2025)
- [LSem2Vec: A Simple yet Effective Two-Stage Approach for Source Code Embedding](https://mlsanity.com/paper/2409.14644/) (2024)
- [SaraCoder: Orchestrating Semantic and Structural Cues for Resource-Optimized Repository-Level Code Completion](https://mlsanity.com/paper/2508.10068/) (2025)
- [SEAL: Structure and Element Aware Learning to Improve Long Structured Document Retrieval](https://mlsanity.com/paper/2508.20778/) (2025)
- [XSearch: Explainable Code Search via Concept-to-Code Alignment](https://mlsanity.com/paper/2605.16046/) (2026)
- [ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval](https://mlsanity.com/paper/2609.01865/) (2026)

## Abstract (arXiv)

In this paper, we propose the CodeRetriever model, which learns the function-level code semantic representations through large-scale code-text contrastive pre-training. We adopt two contrastive learning schemes in CodeRetriever: unimodal contrastive learning and bimodal contrastive learning. For unimodal contrastive learning, we design an unsupervised learning approach to build semantic-related code pairs based on the documentation and function name. For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build code-text pairs. Both contrastive objectives can fully leverage large-scale code corpus for pre-training. Extensive experimental results show that CodeRetriever achieves new state-of-the-art with significant improvement over existing code pre-trained models, on eleven domain/language-specific code search tasks with six programming languages in different code granularity (function-level, snippet-level and statement-level). These results demonstrate the effectiveness and robustness of CodeRetriever.