# Exploring Representation-Level Augmentation for Code Search arXiv: https://arxiv.org/abs/2210.12285 ยท HTML: https://mlsanity.com/paper/2210.12285/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 47 references that are in the map. - First posted on arXiv 21 Oct 2022. ## Builds on (most cited first, 29 of 29) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018) - [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019) - [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021) - [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021) - [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019) - [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019) - [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020) - [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022) - [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020) - [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017) - [CERT: Contrastive Self-supervised Learning for Language Understanding](https://mlsanity.com/paper/2005.12766/) (2020) - [Augmenting Data with Mixup for Sentence Classification: An Empirical Study](https://mlsanity.com/paper/1905.08941/) (2019) - [Contrastive Code Representation Learning](https://mlsanity.com/paper/2007.04973/) (2020) - [Towards Domain-Agnostic Contrastive Learning](https://mlsanity.com/paper/2011.04419/) (2020) - [CoSQA: 20,000+ Web Queries for Code Search and Question Answering](https://mlsanity.com/paper/2105.13239/) (2021) - [SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation](https://mlsanity.com/paper/2108.04556/) (2021) - [Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations](https://mlsanity.com/paper/2009.02731/) (2020) - [When Deep Learning Met Code Search](https://mlsanity.com/paper/1905.03813/) (2019) - [Opportunities and Challenges in Code Search Tools](https://mlsanity.com/paper/2011.02297/) (2020) - [Query Expansion Based on Crowd Knowledge for Code Search](https://mlsanity.com/paper/1703.01443/) (2017) - [Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks](https://mlsanity.com/paper/2010.02394/) (2020) - [Multimodal Representation for Neural Code Search](https://mlsanity.com/paper/2107.00992/) (2021) - [Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation](https://mlsanity.com/paper/2203.07735/) (2022) - [Constructing Contrastive samples via Summarization for Text Classification with limited annotations](https://mlsanity.com/paper/2104.05094/) (2021) ## Built on by (most cited first, 8 of 8) - [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024) - [Rewriting the Code: A Simple Method for Large Language Model Augmented Code Search](https://mlsanity.com/paper/2401.04514/) (2024) - [Rethinking Negative Pairs in Code Search](https://mlsanity.com/paper/2310.08069/) (2023) - [VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs](https://mlsanity.com/paper/2408.04125/) (2024) - [GiFT: Gibbs Fine-Tuning for Code Generation](https://mlsanity.com/paper/2502.11466/) (2025) - [GenCode: A Generic Data Augmentation Framework for Boosting Deep Learning-Based Code Understanding](https://mlsanity.com/paper/2402.15769/) (2024) - [A Study on Mixup-Inspired Augmentation Methods for Software Vulnerability Detection](https://mlsanity.com/paper/2504.15632/) (2025) - [XSearch: Explainable Code Search via Concept-to-Code Alignment](https://mlsanity.com/paper/2605.16046/) (2026) ## Abstract (arXiv) Code search, which aims at retrieving the most relevant code fragment for a given natural language query, is a common activity in software development practice. Recently, contrastive learning is widely used in code search research, where many data augmentation approaches for source code (e.g., semantic-preserving program transformation) are proposed to learn better representations. However, these augmentations are at the raw-data level, which requires additional code analysis in the preprocessing stage and additional training costs in the training stage. In this paper, we explore augmentation methods that augment data (both code and query) at representation level which does not require additional data processing and training, and based on this we propose a general format of representation-level augmentation that unifies existing methods. Then, we propose three new augmentation methods (linear extrapolation, binary interpolation, and Gaussian scaling) based on the general format. Furthermore, we theoretically analyze the advantages of the proposed augmentation methods over traditional contrastive learning methods on code search. We experimentally evaluate the proposed representation-level augmentation methods with state-of-the-art code search models on a large-scale public dataset consisting of six programming languages. The experimental results show that our approach can consistently boost the performance of the studied code search models. Our source code is available at https://github.com/Alex-HaochenLi/RACS.