# Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations

arXiv: https://arxiv.org/abs/2009.02731 · HTML: https://mlsanity.com/paper/2009.02731/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2020.

## Built on by (most cited first, 30 of 49)

- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation](https://mlsanity.com/paper/2108.04556/) (2021)
- [CodeMark: Imperceptible Watermarking for Code Datasets against Neural Code Completion Models](https://mlsanity.com/paper/2308.14401/) (2023)
- [On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages](https://mlsanity.com/paper/2204.09653/) (2022)
- [Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding](https://mlsanity.com/paper/2112.02268/) (2021)
- [CoCoSoDa: Effective Contrastive Learning for Code Search](https://mlsanity.com/paper/2204.03293/) (2022)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search](https://mlsanity.com/paper/2201.10866/) (2022)
- [CodeTF: One-stop Transformer Library for State-of-the-art Code LLMs](https://mlsanity.com/paper/2306.00029/) (2023)
- [Target-aware Abstractive Related Work Generation with Contrastive Learning](https://mlsanity.com/paper/2205.13339/) (2022)
- [The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation](https://mlsanity.com/paper/2305.06156/) (2023)
- [Towards Learning (Dis)-Similarity of Source Code from Program Contrasts](https://mlsanity.com/paper/2110.03868/) (2021)
- [AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models](https://mlsanity.com/paper/2206.11719/) (2022)
- [CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training](https://mlsanity.com/paper/2205.02029/) (2022)
- [Code Search based on Context-aware Code Translation](https://mlsanity.com/paper/2202.08029/) (2022)
- [Rethinking Negative Pairs in Code Search](https://mlsanity.com/paper/2310.08069/) (2023)
- [ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search](https://mlsanity.com/paper/2403.16702/) (2024)
- [Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5](https://mlsanity.com/paper/2211.14875/) (2022)
- [HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representations](https://mlsanity.com/paper/2205.15479/) (2022)
- [Towards Causal Deep Learning for Vulnerability Detection](https://mlsanity.com/paper/2310.07958/) (2023)
- [Exploring Representation-Level Augmentation for Code Search](https://mlsanity.com/paper/2210.12285/) (2022)
- [Soft-Labeled Contrastive Pre-training for Function-level Code Representation](https://mlsanity.com/paper/2210.09597/) (2022)
- [VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs](https://mlsanity.com/paper/2408.04125/) (2024)
- [A Topic-aware Summarization Framework with Different Modal Side Information](https://mlsanity.com/paper/2305.11503/) (2023)
- [SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrieval](https://mlsanity.com/paper/2508.00546/) (2025)
- [MIXCODE: Enhancing Code Classification by Mixup-Based Data Augmentation](https://mlsanity.com/paper/2210.03003/) (2022)
- [TransformCode: A Contrastive Learning Framework for Code Embedding via Subtree Transformation](https://mlsanity.com/paper/2311.08157/) (2023)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [Self-Supervised Learning to Prove Equivalence Between Straight-Line Programs via Rewrite Rules](https://mlsanity.com/paper/2109.10476/) (2021)
- [NS3: Neuro-Symbolic Semantic Code Search](https://mlsanity.com/paper/2205.10674/) (2022)

## Abstract (arXiv)

We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through a contrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.