# CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training

arXiv: https://arxiv.org/abs/2205.02029 · HTML: https://mlsanity.com/paper/2205.02029/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 70 references that are in the map.
- First posted on arXiv 4 May 2022.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://mlsanity.com/paper/1904.12787/) (2019)
- [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [Contrastive Code Representation Learning](https://mlsanity.com/paper/2007.04973/) (2020)
- [CoSQA: 20,000+ Web Queries for Code Search and Question Answering](https://mlsanity.com/paper/2105.13239/) (2021)
- [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020)
- [SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation](https://mlsanity.com/paper/2108.04556/) (2021)
- [CoTexT: Multi-task Learning with Code-Text Transformer](https://mlsanity.com/paper/2105.08645/) (2021)
- [Neural Code Comprehension: A Learnable Representation of Code Semantics](https://mlsanity.com/paper/1806.07336/) (2018)
- [What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code](https://mlsanity.com/paper/2202.06840/) (2022)
- [Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations](https://mlsanity.com/paper/2009.02731/) (2020)
- [Compilable Neural Code Generation with Compiler Feedback](https://mlsanity.com/paper/2203.05132/) (2022)

## Built on by (most cited first, 15 of 15)

- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [PanGu-Coder: Program Synthesis with Function-Level Language Modeling](https://mlsanity.com/paper/2207.11280/) (2022)
- [CodeTF: One-stop Transformer Library for State-of-the-art Code LLMs](https://mlsanity.com/paper/2306.00029/) (2023)
- [Rethinking Negative Pairs in Code Search](https://mlsanity.com/paper/2310.08069/) (2023)
- [Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting](https://mlsanity.com/paper/2405.16133/) (2024)
- [Soft-Labeled Contrastive Pre-training for Function-level Code Representation](https://mlsanity.com/paper/2210.09597/) (2022)
- [NS3: Neuro-Symbolic Semantic Code Search](https://mlsanity.com/paper/2205.10674/) (2022)
- [Finding Reusable Machine Learning Components to Build Programming Language Processing Pipelines](https://mlsanity.com/paper/2208.05596/) (2022)
- [Towards Better Code Understanding in Decoder-Only Models with Contrastive Learning](https://mlsanity.com/paper/2406.12326/) (2024)
- [Code LLMs: A Taxonomy-based Survey](https://mlsanity.com/paper/2412.08291/) (2024)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Neuro-symbolic Zero-Shot Code Cloning with Cross-Language Intermediate Representation](https://mlsanity.com/paper/2304.13350/) (2023)
- [Contrastive Learning for API Aspect Analysis](https://mlsanity.com/paper/2307.16878/) (2023)
- [Contrastive Prompt Learning-based Code Search based on Interaction Matrix](https://mlsanity.com/paper/2310.06342/) (2023)
- [The Patchwork Problem in LLM-Generated Code](https://mlsanity.com/paper/2607.08981/) (2026)

## Abstract (arXiv)

Recent years have witnessed increasing interest in code representation learning, which aims to represent the semantics of source code into distributed vectors. Currently, various works have been proposed to represent the complex semantics of source code from different views, including plain text, Abstract Syntax Tree (AST), and several kinds of code graphs (e.g., Control/Data Flow Graph). However, most of them only consider a single view of source code independently, ignoring the correspondences among different views. In this paper, we propose to integrate different views with the natural-language description of source code into a unified framework with Multi-View contrastive Pre-training, and name our model as CODE-MVP. Specifically, we first extract multiple code views using compiler tools, and learn the complementary information among them under a contrastive learning framework. Inspired by the type checking in compilation, we also design a fine-grained type inference objective in the pre-training. Experiments on three downstream tasks over five datasets demonstrate the superiority of CODE-MVP when compared with several state-of-the-art baselines. For example, we achieve 2.4/2.3/1.1 gain in terms of MRR/MAP/Accuracy metrics on natural language code retrieval, code similarity, and code defect detection tasks, respectively.