# A Literature Study of Embeddings on Source Code

arXiv: https://arxiv.org/abs/1904.03061 · HTML: https://mlsanity.com/paper/1904.03061/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 18 of 18)

- [VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning](https://mlsanity.com/paper/2112.02650/) (2021)
- [Syntax-Guided Program Reduction for Understanding Neural Code Intelligence Models](https://mlsanity.com/paper/2205.14374/) (2022)
- [A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks](https://mlsanity.com/paper/2403.10646/) (2024)
- [Code2Snapshot: Using Code Snapshots for Learning Representations of Source Code](https://mlsanity.com/paper/2111.01097/) (2021)
- [A Survey of Deep Learning Models for Structural Code Understanding](https://mlsanity.com/paper/2205.01293/) (2022)
- [TransformCode: A Contrastive Learning Framework for Code Embedding via Subtree Transformation](https://mlsanity.com/paper/2311.08157/) (2023)
- [Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking](https://mlsanity.com/paper/2508.19558/) (2025)
- [Unsupervised Learning of General-Purpose Embeddings for Code Changes](https://mlsanity.com/paper/2106.02087/) (2021)
- [Test2Vec: An Execution Trace Embedding for Test Case Prioritization](https://mlsanity.com/paper/2206.15428/) (2022)
- [PERFOGRAPH: A Numerical Aware Program Graph Representation for Performance Optimization and Program Analysis](https://mlsanity.com/paper/2306.00210/) (2023)
- [LAMNER: Code Comment Generation Using Character Language Model and Named Entity Recognition](https://mlsanity.com/paper/2204.09654/) (2022)
- [An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code](https://mlsanity.com/paper/2307.02443/) (2023)
- [Identification of the Relevance of Comments in Codes Using Bag of Words and Transformer Based Models](https://mlsanity.com/paper/2308.06144/) (2023)
- [Extracting Label-specific Key Input Features for Neural Code Intelligence Models](https://mlsanity.com/paper/2202.06474/) (2022)
- [MQDD: Pre-training of Multimodal Question Duplicity Detection for Software Engineering Domain](https://mlsanity.com/paper/2203.14093/) (2022)
- [Test Case Recommendations with Distributed Representation of Code Syntactic Features](https://mlsanity.com/paper/2310.03174/) (2023)
- [Multi-modal Learning for WebAssembly Reverse Engineering](https://mlsanity.com/paper/2404.03171/) (2024)
- [Type-IV Code Clone Detection via Layer-Wise Non-Contrastive Representation Learning](https://mlsanity.com/paper/2609.17338/) (2026)

## Abstract (arXiv)

Natural language processing has improved tremendously after the success of word embedding techniques such as word2vec. Recently, the same idea has been applied on source code with encouraging results. In this survey, we aim to collect and discuss the usage of word embedding techniques on programs and source code. The articles in this survey have been collected by asking authors of related work and with an extensive search on Google Scholar. Each article is categorized into five categories: 1. embedding of tokens 2. embedding of functions or methods 3. embedding of sequences or sets of method calls 4. embedding of binary code 5. other embeddings. We also provide links to experimental data and show some remarkable visualization of code embeddings. In summary, word embedding has been successfully applied on different granularities of source code. With access to countless open-source repositories, we see a great potential of applying other data-driven natural language processing techniques on source code in the future.