# Towards Better Code Understanding in Decoder-Only Models with Contrastive Learning

arXiv: https://arxiv.org/abs/2406.12326 · HTML: https://mlsanity.com/paper/2406.12326/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 12 of 12)

- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [CodeGemma: Open Code Models Based on Gemma](https://mlsanity.com/paper/2406.11409/) (2024)
- [CoSQA: 20,000+ Web Queries for Code Search and Question Answering](https://mlsanity.com/paper/2105.13239/) (2021)
- [SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation](https://mlsanity.com/paper/2108.04556/) (2021)
- [WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning](https://mlsanity.com/paper/2312.14187/) (2023)
- [TreeBERT: A Tree-Based Pre-Trained Model for Programming Language](https://mlsanity.com/paper/2105.12485/) (2021)
- [Contrastive Learning of Sentence Embeddings from Scratch](https://mlsanity.com/paper/2305.15077/) (2023)
- [CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training](https://mlsanity.com/paper/2205.02029/) (2022)
- [Scaling Laws Behind Code Understanding Model](https://mlsanity.com/paper/2402.12813/) (2024)

## Built on by (most cited first, 3 of 3)

- [SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents](https://mlsanity.com/paper/2609.04167/) (2026)
- [Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detection](https://mlsanity.com/paper/2605.02860/) (2026)
- [Invariant Pretraining for Robust Code Representations](https://mlsanity.com/paper/2608.15412/) (2026)

## Abstract (arXiv)

Recent advances in large-scale code generation models have led to remarkable progress in producing high-quality code. These models are trained in a self-supervised manner on extensive unlabeled code corpora using a decoder-only architecture. However, despite their generative strength, decoder-only models often exhibit limited performance on code understanding tasks such as code search and clone detection, primarily due to their generation-oriented training objectives. While training large encoder-only models from scratch on massive code datasets can improve understanding ability but remains computationally expensive and time-consuming. In this paper, we explore a more efficient alternative by transferring knowledge from pre-trained decoder-only code generation models to code understanding tasks. We investigate how decoder-only architectures can be effectively adapted to learn discriminative and semantically meaningful code representations. To this end, we propose CL4D, a contrastive learning framework tailored to strengthen the representation capabilities of decoder-only models. Extensive experiments on multiple benchmark datasets demonstrate that CL4D achieves competitive or superior performance compared to existing methods on representative code understanding tasks, including code search and clone detection. Further analysis reveals that CL4D substantially improves the semantic alignment of code representations by reducing the distance between semantically similar code snippets. These findings highlight the feasibility of leveraging decoder-only models as a unified backbone for both code generation and understanding.