# Towards Demystifying Dimensions of Source Code Embeddings

arXiv: https://arxiv.org/abs/2008.13064 · HTML: https://mlsanity.com/paper/2008.13064/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2020.

## Built on by (most cited first, 9 of 9)

- [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021)
- [Understanding Neural Code Intelligence Through Program Simplification](https://mlsanity.com/paper/2106.03353/) (2021)
- [Memorization and Generalization in Neural Code Intelligence Models](https://mlsanity.com/paper/2106.08704/) (2021)
- [Syntax-Guided Program Reduction for Understanding Neural Code Intelligence Models](https://mlsanity.com/paper/2205.14374/) (2022)
- [Code2Snapshot: Using Code Snapshots for Learning Representations of Source Code](https://mlsanity.com/paper/2111.01097/) (2021)
- [Study of Distractors in Neural Models of Code](https://mlsanity.com/paper/2303.01739/) (2023)
- [Redundancy and Concept Analysis for Code-trained Language Models](https://mlsanity.com/paper/2305.00875/) (2023)
- [Extracting Label-specific Key Input Features for Neural Code Intelligence Models](https://mlsanity.com/paper/2202.06474/) (2022)
- [A Study of Variable-Role-based Feature Enrichment in Neural Models of Code](https://mlsanity.com/paper/2303.04942/) (2023)

## Abstract (arXiv)

Source code representations are key in applying machine learning techniques for processing and analyzing programs. A popular approach in representing source code is neural source code embeddings that represents programs with high-dimensional vectors computed by training deep neural networks on a large volume of programs. Although successful, there is little known about the contents of these vectors and their characteristics. In this paper, we present our preliminary results towards better understanding the contents of code2vec neural source code embeddings. In particular, in a small case study, we use the code2vec embeddings to create binary SVM classifiers and compare their performance with the handcrafted features. Our results suggest that the handcrafted features can perform very close to the highly-dimensional code2vec embeddings, and the information gains are more evenly distributed in the code2vec embeddings compared to the handcrafted features. We also find that the code2vec embeddings are more resilient to the removal of dimensions with low information gains than the handcrafted features. We hope our results serve a stepping stone toward principled analysis and evaluation of these code representations.