# InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees

arXiv: https://arxiv.org/abs/2012.07023 · HTML: https://mlsanity.com/paper/2012.07023/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2020.

## Built on by (most cited first, 23 of 23)

- [How could Neural Networks understand Programs?](https://mlsanity.com/paper/2105.04297/) (2021)
- [HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent](https://mlsanity.com/paper/2406.00215/) (2024)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [CodeTF: One-stop Transformer Library for State-of-the-art Code LLMs](https://mlsanity.com/paper/2306.00029/) (2023)
- [The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation](https://mlsanity.com/paper/2305.06156/) (2023)
- [Graph Neural Networks: Methods, Applications, and Opportunities](https://mlsanity.com/paper/2108.10733/) (2021)
- [CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training](https://mlsanity.com/paper/2205.02029/) (2022)
- [A systematic literature review on source code similarity measurement and clone detection: techniques, applications, and challenges](https://mlsanity.com/paper/2306.16171/) (2023)
- [AutoPruner: Transformer-Based Call Graph Pruning](https://mlsanity.com/paper/2209.03230/) (2022)
- [CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking](https://mlsanity.com/paper/2402.11842/) (2024)
- [A Neural Network Architecture for Program Understanding Inspired by Human Behaviors](https://mlsanity.com/paper/2206.04730/) (2022)
- [Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Code](https://mlsanity.com/paper/2308.13507/) (2023)
- [A Library for Representing Python Programs as Graphs for Machine Learning](https://mlsanity.com/paper/2208.07461/) (2022)
- [MIXCODE: Enhancing Code Classification by Mixup-Based Data Augmentation](https://mlsanity.com/paper/2210.03003/) (2022)
- [TransformCode: A Contrastive Learning Framework for Code Embedding via Subtree Transformation](https://mlsanity.com/paper/2311.08157/) (2023)
- [Evaluation of Contrastive Learning with Various Code Representations for Code Clone Detection](https://mlsanity.com/paper/2206.08726/) (2022)
- [NeuDep: Neural Binary Memory Dependence Analysis](https://mlsanity.com/paper/2210.02853/) (2022)
- [Reliability Assurance for Deep Neural Network Architectures Against Numerical Defects](https://mlsanity.com/paper/2302.06086/) (2023)
- [DocChecker: Bootstrapping Code Large Language Model for Detecting and Resolving Code-Comment Inconsistencies](https://mlsanity.com/paper/2306.06347/) (2023)
- [xASTNN: Improved Code Representations for Industrial Practice](https://mlsanity.com/paper/2303.07104/) (2023)
- [A Study of Variable-Role-based Feature Enrichment in Neural Models of Code](https://mlsanity.com/paper/2303.04942/) (2023)
- [Analysing the Behaviour of Tree-Based Neural Networks in Regression Tasks](https://mlsanity.com/paper/2406.11437/) (2024)
- [LSem2Vec: A Simple yet Effective Two-Stage Approach for Source Code Embedding](https://mlsanity.com/paper/2409.14644/) (2024)

## Abstract (arXiv)

Building deep learning models on source code has found many successful software engineering applications, such as code search, code comment generation, bug detection, code migration, and so on. Current learning techniques, however, have a major drawback that these models are mostly trained on datasets labeled for particular downstream tasks, and code representations may not be suitable for other tasks. While some techniques produce representations from unlabeled code, they are far from satisfactory when applied to downstream tasks. Although certain techniques generate representations from unlabeled code when applied to downstream tasks they are far from satisfactory. This paper proposes InferCode to overcome the limitation by adapting the self-supervised learning mechanism to build source code model. The key novelty lies in training code representations by predicting automatically identified subtrees from the context of the ASTs. Subtrees in ASTs are treated with InferCode as the labels for training code representations without any human labeling effort or the overhead of expensive graph construction, and the trained representations are no longer tied to any specific downstream tasks or code units. We trained an InferCode model instance using the Tree-based CNN as the encoder of a large set of Java code and applied it to downstream unsupervised tasks such as code clustering, code clone detection, cross-language code search or reused under a transfer learning scheme to continue training the model weights for supervised tasks such as code classification and method name prediction. Compared to previous code learning techniques applied to the same downstream tasks, such as Code2Vec, Code2Seq, ASTNN, higher performance results are achieved using our pre-trained InferCode model with a significant margin for most tasks including those involving different programming languages.