# ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning

arXiv: https://arxiv.org/abs/2301.09072 · HTML: https://mlsanity.com/paper/2301.09072/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2023.

## Built on by (most cited first, 16 of 16)

- [Finetuning Large Language Models for Vulnerability Detection](https://mlsanity.com/paper/2401.17010/) (2024)
- [Multi-target Backdoor Attacks for Code Pre-trained Models](https://mlsanity.com/paper/2306.08350/) (2023)
- [Understanding Programs by Exploiting (Fuzzing) Test Cases](https://mlsanity.com/paper/2305.13592/) (2023)
- [Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey](https://mlsanity.com/paper/2310.17903/) (2023)
- [Code Representation Pre-training with Complements from Program Executions](https://mlsanity.com/paper/2309.09980/) (2023)
- [Neuro Symbolic Reasoning for Planning: Counterexample Guided Inductive Synthesis using Large Language Models and Satisfiability Solving](https://mlsanity.com/paper/2309.16436/) (2023)
- [The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification](https://mlsanity.com/paper/2305.04940/) (2023)
- [Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking](https://mlsanity.com/paper/2508.19558/) (2025)
- [A Black-Box Attack on Code Models via Representation Nearest Neighbor Search](https://mlsanity.com/paper/2305.05896/) (2023)
- [Solution-oriented Agent-based Models Generation with Verifier-assisted Iterative In-context Learning](https://mlsanity.com/paper/2402.02388/) (2024)
- [Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection](https://mlsanity.com/paper/2503.23448/) (2025)
- [Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models](https://mlsanity.com/paper/2505.09062/) (2025)
- [Source Code is a Graph, Not a Sequence: A Cross-Lingual Perspective on Code Clone Detection](https://mlsanity.com/paper/2312.16488/) (2023)
- [CLNX: Bridging Code and Natural Language for C/C++ Vulnerability-Contributing Commits Identification](https://mlsanity.com/paper/2409.07407/) (2024)
- [LoRACode: LoRA Adapters for Code Embeddings](https://mlsanity.com/paper/2503.05315/) (2025)
- [SoK: Understanding (New) Security Issues Across AI4Code Use Cases](https://mlsanity.com/paper/2512.18456/) (2025)

## Abstract (arXiv)

Large-scale pre-trained models such as CodeBERT, GraphCodeBERT have earned widespread attention from both academia and industry. Attributed to the superior ability in code representation, they have been further applied in multiple downstream tasks such as clone detection, code search and code translation. However, it is also observed that these state-of-the-art pre-trained models are susceptible to adversarial attacks. The performance of these pre-trained models drops significantly with simple perturbations such as renaming variable names. This weakness may be inherited by their downstream models and thereby amplified at an unprecedented scale. To this end, we propose an approach namely ContraBERT that aims to improve the robustness of pre-trained models via contrastive learning. Specifically, we design nine kinds of simple and complex data augmentation operators on the programming language (PL) and natural language (NL) data to construct different variants. Furthermore, we continue to train the existing pre-trained models by masked language modeling (MLM) and contrastive pre-training task on the original samples with their augmented variants to enhance the robustness of the model. The extensive experiments demonstrate that ContraBERT can effectively improve the robustness of the existing pre-trained models. Further study also confirms that these robustness-enhanced models provide improvements as compared to original models over four popular downstream tasks.