# Which Features are Learned by CodeBert: An Empirical Study of the BERT-based Source Code Representation Learning

arXiv: https://arxiv.org/abs/2301.08427 · HTML: https://mlsanity.com/paper/2301.08427/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 20 Jan 2023.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [PalmTree: Learning an Assembly Language Model for Instruction Embedding](https://mlsanity.com/paper/2103.03809/) (2021)
- [Using Deep Learning to Solve Computer Security Challenges: A Survey](https://mlsanity.com/paper/1912.05721/) (2019)
- [Spotting Silent Buffer Overflows in Execution Trace through Graph Neural Network Assisted Data Flow Analysis](https://mlsanity.com/paper/2102.10452/) (2021)

## Built on by (most cited first, 2 of 2)

- [From Texts to Shields: Convergence of Large Language Models and Cybersecurity](https://mlsanity.com/paper/2505.00841/) (2025)
- [On The Impact of Merge Request Deviations on Code Review Practices](https://mlsanity.com/paper/2506.08860/) (2025)

## Abstract (arXiv)

The Bidirectional Encoder Representations from Transformers (BERT) were proposed in the natural language process (NLP) and shows promising results. Recently researchers applied the BERT to source-code representation learning and reported some good news on several downstream tasks. However, in this paper, we illustrated that current methods cannot effectively understand the logic of source codes. The representation of source code heavily relies on the programmer-defined variable and function names. We design and implement a set of experiments to demonstrate our conjecture and provide some insights for future works.