# Performance Optimization using Multimodal Modeling and Heterogeneous GNN

arXiv: https://arxiv.org/abs/2304.12568 · HTML: https://mlsanity.com/paper/2304.12568/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 91 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 26 of 26)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://mlsanity.com/paper/1904.12787/) (2019)
- [A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources](https://mlsanity.com/paper/2011.14867/) (2020)
- [Neural Code Comprehension: A Learnable Representation of Code Semantics](https://mlsanity.com/paper/1806.07336/) (2018)
- [UNICORN: Runtime Provenance-Based Detector for Advanced Persistent Threats](https://mlsanity.com/paper/2001.01525/) (2020)
- [NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learning](https://mlsanity.com/paper/1909.13639/) (2019)
- [A General Path-Based Representation for Predicting Program Properties](https://mlsanity.com/paper/1803.09544/) (2018)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Autotuning PolyBench Benchmarks with LLVM Clang/Polly Loop Optimization Pragmas Using Bayesian Optimization](https://mlsanity.com/paper/2010.08040/) (2020)
- [Learning Intermediate Representations using Graph Neural Networks for NUMA and Prefetchers Optimization](https://mlsanity.com/paper/2203.00611/) (2022)
- [A deep tree-based model for software defect prediction](https://mlsanity.com/paper/1802.00921/) (2018)
- [GraphIt: A High-Performance DSL for Graph Analytics](https://mlsanity.com/paper/1805.00923/) (2018)
- [Customized Monte Carlo Tree Search for LLVM/Polly's Composable Loop Optimization Transformations](https://mlsanity.com/paper/2105.04555/) (2021)
- [Towards Autotuning of OpenMP Applications on Multicore Architectures](https://mlsanity.com/paper/1401.4063/) (2014)
- [Power Constrained Autotuning using Graph Neural Networks](https://mlsanity.com/paper/2302.11467/) (2023)
- [ParaGraph: Weighted Graph Representation for Performance Optimization of HPC Kernels](https://mlsanity.com/paper/2304.03487/) (2023)

## Built on by (most cited first, 7 of 7)

- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [The Landscape and Challenges of HPC Research and LLMs](https://mlsanity.com/paper/2402.02018/) (2024)
- [OMPGPT: A Generative Pre-trained Transformer Model for OpenMP](https://mlsanity.com/paper/2401.16445/) (2024)
- [MIREncoder: Multi-modal IR-based Pretrained Embeddings for Performance Optimizations](https://mlsanity.com/paper/2407.02238/) (2024)
- [Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network](https://mlsanity.com/paper/2408.00290/) (2024)
- [LLM-Based Embeddings for Program Analysis and Optimization](https://mlsanity.com/paper/2608.07894/) (2026)
- [Accelerating Transfer-Learning-Based Autotuning with Predictive LLVM IR Performance Ranking](https://mlsanity.com/paper/2609.15807/) (2026)

## Abstract (arXiv)

Growing heterogeneity and configurability in HPC architectures has made auto-tuning applications and runtime parameters on these systems very complex. Users are presented with a multitude of options to configure parameters. In addition to application specific solutions, a common approach is to use general purpose search strategies, which often might not identify the best configurations or their time to convergence is a significant barrier. There is, thus, a need for a general purpose and efficient tuning approach that can be easily scaled and adapted to various tuning tasks. We propose a technique for tuning parallel code regions that is general enough to be adapted to multiple tasks. In this paper, we analyze IR-based programming models to make task-specific performance optimizations. To this end, we propose the Multimodal Graph Neural Network and Autoencoder (MGA) tuner, a multimodal deep learning based approach that adapts Heterogeneous Graph Neural Networks and Denoizing Autoencoders for modeling IR-based code representations that serve as separate modalities. This approach is used as part of our pipeline to model a syntax, semantics, and structure-aware IR-based code representation for tuning parallel code regions/kernels. We extensively experiment on OpenMP and OpenCL code regions/kernels obtained from PolyBench, Rodinia, STREAM, DataRaceBench, AMD SDK, NPB, NVIDIA SDK, Parboil, SHOC, and LULESH benchmarks. We apply our multimodal learning techniques to the tasks of i) optimizing the number of threads, scheduling policy and chunk size in OpenMP loops and, ii) identifying the best device for heterogeneous device mapping of OpenCL kernels. Our experiments show that this multimodal learning based approach outperforms the state-of-the-art in all experiments.