# Learning Generalizable Program and Architecture Representations for Performance Modeling

arXiv: https://arxiv.org/abs/2310.16792 · HTML: https://mlsanity.com/paper/2310.16792/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 70 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [Neural Code Comprehension: A Learnable Representation of Code Semantics](https://mlsanity.com/paper/1806.07336/) (2018)
- [Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks](https://mlsanity.com/paper/1808.07412/) (2018)
- [Dynamic Neural Program Embedding for Program Repair](https://mlsanity.com/paper/1711.07163/) (2017)
- [Learning Program Embeddings to Propagate Feedback on Student Code](https://mlsanity.com/paper/1505.05969/) (2015)
- [DiffTune: Optimizing CPU Simulator Parameters with Learned Differentiable Surrogates](https://mlsanity.com/paper/2010.04017/) (2020)
- [Learning Execution through Neural Code Fusion](https://mlsanity.com/paper/1906.07181/) (2019)
- [GRANITE: A Graph Neural Network Model for Basic Block Throughput Estimation](https://mlsanity.com/paper/2210.03894/) (2022)
- [SimNet: Accurate and High-Performance Computer Architecture Simulation using Deep Learning](https://mlsanity.com/paper/2105.05821/) (2021)
- [Application Performance Modeling via Tensor Completion](https://mlsanity.com/paper/2210.10184/) (2022)

## Built on by (most cited first, 2 of 2)

- [PAI: Fast, Accurate, and Full Benchmark Performance Projection with AI](https://mlsanity.com/paper/2603.19330/) (2026)
- [On the Limits of Machine-Learned Ranking for Modern Microarchitectural Policies](https://mlsanity.com/paper/2608.01041/) (2026)

## Abstract (arXiv)

Performance modeling is an essential tool in many areas, including performance characterization/optimization, design space exploration, and resource allocation problems, to name a few. However, existing performance modeling approaches have limitations, such as high computational cost for discrete-event simulators, narrow flexibility of hardware emulators, or restricted accuracy/generality of analytical/data-driven models. To address these limitations, this paper proposes PerfVec, a novel deep learning-based performance modeling framework that learns high-dimensional and independent/orthogonal program and microarchitecture representations. Once learned, a program representation can be used to predict its performance on any microarchitecture, and likewise, a microarchitecture representation can be applied in the performance prediction of any program. Additionally, PerfVec yields a foundation model that captures the performance essence of instructions, which can be directly used by developers in numerous performance modeling related tasks without incurring its training cost. The evaluation demonstrates that PerfVec is more general and efficient than previous approaches.