# Analysing the Behaviour of Tree-Based Neural Networks in Regression Tasks

arXiv: https://arxiv.org/abs/2406.11437 · HTML: https://mlsanity.com/paper/2406.11437/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [On the Bottleneck of Graph Neural Networks and its Practical Implications](https://mlsanity.com/paper/2006.05205/) (2020)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [A Survey on Oversmoothing in Graph Neural Networks](https://mlsanity.com/paper/2303.10993/) (2023)
- [A Gentle Introduction to Deep Learning for Graphs](https://mlsanity.com/paper/1912.12693/) (2019)
- [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020)
- [Graph Neural Networks for temporal graphs: State of the art, open challenges, and opportunities](https://mlsanity.com/paper/2302.01018/) (2023)
- [Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations](https://mlsanity.com/paper/2009.02731/) (2020)
- [TreeBERT: A Tree-Based Pre-Trained Model for Programming Language](https://mlsanity.com/paper/2105.12485/) (2021)
- [The expressive power of pooling in Graph Neural Networks](https://mlsanity.com/paper/2304.01575/) (2023)
- [Learning and Evaluating Contextual Embedding of Source Code](https://mlsanity.com/paper/2001.00059/) (2019)
- [ProGraML: Graph-based Deep Learning for Program Optimization and Analysis](https://mlsanity.com/paper/2003.10536/) (2020)
- [InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees](https://mlsanity.com/paper/2012.07023/) (2020)
- [Graph Neural Networks Designed for Different Graph Types: A Survey](https://mlsanity.com/paper/2204.03080/) (2022)
- [Fault Localization with Code Coverage Representation Learning](https://mlsanity.com/paper/2103.00270/) (2021)
- [A Multi-Modal Transformer-based Code Summarization Approach for Smart Contracts](https://mlsanity.com/paper/2103.07164/) (2021)
- [Hypergraph Neural Networks through the Lens of Message Passing: A Common Perspective to Homophily and Architecture Design](https://mlsanity.com/paper/2310.07684/) (2023)
- [Modeling Functional Similarity in Source Code with Graph-Based Siamese Networks](https://mlsanity.com/paper/2011.11228/) (2020)
- [Meta-Path Learning for Multi-relational Graph Neural Networks](https://mlsanity.com/paper/2309.17113/) (2023)
- [Graph Neural Networks for Graph Drawing](https://mlsanity.com/paper/2109.10061/) (2021)
- [Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach](https://mlsanity.com/paper/2201.03326/) (2022)
- [TEP-GNN: Accurate Execution Time Prediction of Functional Tests using Graph Neural Networks](https://mlsanity.com/paper/2208.11947/) (2022)
- [Cross-Language Learning for Program Classification using Bilateral Tree-Based Convolutional Neural Networks](https://mlsanity.com/paper/1710.06159/) (2017)
- [A Unified Active Learning Framework for Annotating Graph Data with Application to Software Source Code Performance Prediction](https://mlsanity.com/paper/2304.13032/) (2023)
- [Emotion Analysis using Multi-Layered Networks for Graphical Representation of Tweets](https://mlsanity.com/paper/2207.00907/) (2022)

## Abstract (arXiv)

The landscape of deep learning has vastly expanded the frontiers of source code analysis, particularly through the utilization of structural representations such as Abstract Syntax Trees (ASTs). While these methodologies have demonstrated effectiveness in classification tasks, their efficacy in regression applications, such as execution time prediction from source code, remains underexplored. This paper endeavours to decode the behaviour of tree-based neural network models in the context of such regression challenges. We extend the application of established models--tree-based Convolutional Neural Networks (CNNs), Code2Vec, and Transformer-based methods--to predict the execution time of source code by parsing it to an AST. Our comparative analysis reveals that while these models are benchmarks in code representation, they exhibit limitations when tasked with regression. To address these deficiencies, we propose a novel dual-transformer approach that operates on both source code tokens and AST representations, employing cross-attention mechanisms to enhance interpretability between the two domains. Furthermore, we explore the adaptation of Graph Neural Networks (GNNs) to this tree-based problem, theorizing the inherent compatibility due to the graphical nature of ASTs. Empirical evaluations on real-world datasets showcase that our dual-transformer model outperforms all other tree-based neural networks and the GNN-based models. Moreover, our proposed dual transformer demonstrates remarkable adaptability and robust performance across diverse datasets.