# Learning and Evaluating Contextual Embedding of Source Code

arXiv: https://arxiv.org/abs/2001.00059 · HTML: https://mlsanity.com/paper/2001.00059/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2019.

## Built on by (most cited first, 30 of 33)

- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [PanGu-Coder: Program Synthesis with Function-Level Language Modeling](https://mlsanity.com/paper/2207.11280/) (2022)
- [TreeBERT: A Tree-Based Pre-Trained Model for Programming Language](https://mlsanity.com/paper/2105.12485/) (2021)
- [What do pre-trained code models know about code?](https://mlsanity.com/paper/2108.11308/) (2021)
- [On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages](https://mlsanity.com/paper/2204.09653/) (2022)
- [Large Language Models of Code Fail at Completing Code with Potential Bugs](https://mlsanity.com/paper/2306.03438/) (2023)
- [On Multi-Modal Learning of Editing Source Code](https://mlsanity.com/paper/2108.06645/) (2021)
- [Multi-target Backdoor Attacks for Code Pre-trained Models](https://mlsanity.com/paper/2306.08350/) (2023)
- [Understanding Neural Code Intelligence Through Program Simplification](https://mlsanity.com/paper/2106.03353/) (2021)
- [CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search](https://mlsanity.com/paper/2201.10866/) (2022)
- [Generating Bug-Fixes Using Pretrained Transformers](https://mlsanity.com/paper/2104.07896/) (2021)
- [DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons](https://mlsanity.com/paper/2105.09352/) (2021)
- [One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization](https://mlsanity.com/paper/2303.15822/) (2023)
- [Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities](https://mlsanity.com/paper/2212.10017/) (2022)
- [ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback](https://mlsanity.com/paper/2107.14035/) (2021)
- [A Simple, Yet Effective Approach to Finding Biases in Code Generation](https://mlsanity.com/paper/2211.00609/) (2022)
- [Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Models](https://mlsanity.com/paper/2104.01642/) (2021)
- [Soft-Labeled Contrastive Pre-training for Function-level Code Representation](https://mlsanity.com/paper/2210.09597/) (2022)
- [CommitBART: A Large Pre-trained Model for GitHub Commits](https://mlsanity.com/paper/2208.08100/) (2022)
- [Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehension](https://mlsanity.com/paper/2404.08885/) (2024)
- [Can we learn from developer mistakes? Learning to localize and repair real bugs from real bug fixes](https://mlsanity.com/paper/2207.00301/) (2022)
- [Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis](https://mlsanity.com/paper/2310.13669/) (2023)
- [Unsupervised Learning of General-Purpose Embeddings for Code Changes](https://mlsanity.com/paper/2106.02087/) (2021)
- [CoRet: Improved Retriever for Code Editing](https://mlsanity.com/paper/2505.24715/) (2025)
- [Semantic-Preserving Adversarial Code Comprehension](https://mlsanity.com/paper/2209.05130/) (2022)
- [INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers](https://mlsanity.com/paper/2312.05092/) (2023)
- [An Empirical Study of Knowledge Distillation for Code Understanding Tasks](https://mlsanity.com/paper/2508.15423/) (2025)
- [AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees](https://mlsanity.com/paper/2201.07984/) (2022)
- [Exploring Representation of Horn Clauses using GNNs (Extended Technical Report)](https://mlsanity.com/paper/2206.06986/) (2022)
- [Text-to-Code Generation with Modality-relative Pre-training](https://mlsanity.com/paper/2402.05783/) (2024)

## Abstract (arXiv)

Recent research has achieved impressive results on understanding and improving source code by building up on machine-learning techniques developed for natural languages. A significant advancement in natural-language understanding has come with the development of pre-trained contextual embeddings, such as BERT, which can be fine-tuned for downstream tasks with less labeled data and training budget, while achieving better accuracies. However, there is no attempt yet to obtain a high-quality contextual embedding of source code, and to evaluate it on multiple program-understanding tasks simultaneously; that is the gap that this paper aims to mitigate. Specifically, first, we curate a massive, deduplicated corpus of 7.4M Python files from GitHub, which we use to pre-train CuBERT, an open-sourced code-understanding BERT model; and, second, we create an open-sourced benchmark that comprises five classification tasks and one program-repair task, akin to code-understanding tasks proposed in the literature before. We fine-tune CuBERT on our benchmark tasks, and compare the resulting models to different variants of Word2Vec token embeddings, BiLSTM and Transformer models, as well as published state-of-the-art models, showing that CuBERT outperforms them all, even with shorter training, and with fewer labeled examples. Future work on source-code embedding can benefit from reusing our benchmark, and from comparing against CuBERT models as a strong baseline.