# LoRACode: LoRA Adapters for Code Embeddings

arXiv: https://arxiv.org/abs/2503.05315 · HTML: https://mlsanity.com/paper/2503.05315/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [CoSQA: 20,000+ Web Queries for Code Search and Question Answering](https://mlsanity.com/paper/2105.13239/) (2021)
- [CoIR: A Comprehensive Benchmark for Code Information Retrieval Models](https://mlsanity.com/paper/2407.02883/) (2024)
- [XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence](https://mlsanity.com/paper/2206.08474/) (2022)
- [CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval](https://mlsanity.com/paper/2411.12644/) (2024)
- [ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning](https://mlsanity.com/paper/2301.09072/) (2023)
- [Survey of Code Search Based on Deep Learning](https://mlsanity.com/paper/2305.05959/) (2023)
- [Language Agnostic Code Embeddings](https://mlsanity.com/paper/2310.16803/) (2023)
- [Refining Joint Text and Source Code Embeddings for Retrieval Task with Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2405.04126/) (2024)
- [Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning](https://mlsanity.com/paper/2408.00690/) (2024)

## Abstract (arXiv)

Code embeddings are essential for semantic code search; however, current approaches often struggle to capture the precise syntactic and contextual nuances inherent in code. Open-source models such as CodeBERT and UniXcoder exhibit limitations in scalability and efficiency, while high-performing proprietary systems impose substantial computational costs. We introduce a parameter-efficient fine-tuning method based on Low-Rank Adaptation (LoRA) to construct task-specific adapters for code retrieval. Our approach reduces the number of trainable parameters to less than two percent of the base model, enabling rapid fine-tuning on extensive code corpora (2 million samples in 25 minutes on two H100 GPUs). Experiments demonstrate an increase of up to 9.1% in Mean Reciprocal Rank (MRR) for Code2Code search, and up to 86.69% for Text2Code search tasks across multiple programming languages. Distinction in task-wise and language-wise adaptation helps explore the sensitivity of code retrieval for syntactical and linguistic variations. To foster research in this area, we make our code and pre-trained models publicly available.