# xASTNN: Improved Code Representations for Industrial Practice

arXiv: https://arxiv.org/abs/2303.07104 · HTML: https://mlsanity.com/paper/2303.07104/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 59 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks](https://mlsanity.com/paper/2102.02017/) (2021)
- [Neural Code Comprehension: A Learnable Representation of Code Semantics](https://mlsanity.com/paper/1806.07336/) (2018)
- [When Deep Learning Met Code Search](https://mlsanity.com/paper/1905.03813/) (2019)
- [SourcererCC: Scaling Code Clone Detection to Big Code](https://mlsanity.com/paper/1512.06448/) (2015)
- [What do pre-trained code models know about code?](https://mlsanity.com/paper/2108.11308/) (2021)
- [SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representations](https://mlsanity.com/paper/2201.01549/) (2022)
- [Counterfactual Explanations for Models of Code](https://mlsanity.com/paper/2111.05711/) (2021)
- [InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees](https://mlsanity.com/paper/2012.07023/) (2020)
- [Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder](https://mlsanity.com/paper/1707.05436/) (2017)
- [Learning to Represent Programs with Heterogeneous Graphs](https://mlsanity.com/paper/2012.04188/) (2020)
- [Structural Attention Neural Networks for improved sentiment analysis](https://mlsanity.com/paper/1701.01811/) (2017)
- [Improving Code Autocompletion with Transfer Learning](https://mlsanity.com/paper/2105.05991/) (2021)
- [On the Effectiveness of Transfer Learning for Code Search](https://mlsanity.com/paper/2108.05890/) (2021)
- [Learning Program Semantics with Code Representations: An Empirical Study](https://mlsanity.com/paper/2203.11790/) (2022)
- [A Comparison of Code Embeddings and Beyond](https://mlsanity.com/paper/2109.07173/) (2021)
- [Exploring and Evaluating Personalized Models for Code Generation](https://mlsanity.com/paper/2208.13928/) (2022)

## Built on by (most cited first, 2 of 2)

- [Deep Learning-based Intrusion Detection Systems: A Survey](https://mlsanity.com/paper/2504.07839/) (2025)
- [From Illusion to Insight: Change-Aware File-Level Software Defect Prediction Using Agentic AI](https://mlsanity.com/paper/2512.23875/) (2025)

## Abstract (arXiv)

The application of deep learning techniques in software engineering becomes increasingly popular. One key problem is developing high-quality and easy-to-use source code representations for code-related tasks. The research community has acquired impressive results in recent years. However, due to the deployment difficulties and performance bottlenecks, seldom these approaches are applied to the industry. In this paper, we present xASTNN, an eXtreme Abstract Syntax Tree (AST)-based Neural Network for source code representation, aiming to push this technique to industrial practice. The proposed xASTNN has three advantages. First, xASTNN is completely based on widely-used ASTs and does not require complicated data pre-processing, making it applicable to various programming languages and practical scenarios. Second, three closely-related designs are proposed to guarantee the effectiveness of xASTNN, including statement subtree sequence for code naturalness, gated recursive unit for syntactical information, and gated recurrent unit for sequential information. Third, a dynamic batching algorithm is introduced to significantly reduce the time complexity of xASTNN. Two code comprehension downstream tasks, code classification and code clone detection, are adopted for evaluation. The results demonstrate that our xASTNN can improve the state-of-the-art while being faster than the baselines.