# Data-efficient Performance Modeling via Pre-training

arXiv: https://arxiv.org/abs/2501.14438 · HTML: https://mlsanity.com/paper/2501.14438/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 17 of 17)

- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training](https://mlsanity.com/paper/2205.14401/) (2022)
- [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Learning to Optimize Tensor Programs](https://mlsanity.com/paper/1805.08166/) (2018)
- [Neural Code Comprehension: A Learnable Representation of Code Semantics](https://mlsanity.com/paper/1806.07336/) (2018)
- [Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code](https://mlsanity.com/paper/1804.10694/) (2018)
- [Large Language Models for Compiler Optimization](https://mlsanity.com/paper/2309.07062/) (2023)
- [Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks](https://mlsanity.com/paper/1808.07412/) (2018)
- [A Deep Learning Based Cost Model for Automatic Code Optimization](https://mlsanity.com/paper/2104.04955/) (2021)
- [Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation](https://mlsanity.com/paper/2001.08743/) (2020)
- [Sentence Bottleneck Autoencoders from Transformer Language Models](https://mlsanity.com/paper/2109.00055/) (2021)
- [Less is More: Surgical Phase Recognition from Timestamp Supervision](https://mlsanity.com/paper/2202.08199/) (2022)
- [Unsupervised Natural Language Generation with Denoising Autoencoders](https://mlsanity.com/paper/1804.07899/) (2018)
- [LOOPer: A Learned Automatic Code Optimizer For Polyhedral Compilers](https://mlsanity.com/paper/2403.11522/) (2024)

## Abstract (arXiv)

Performance models are essential for automatic code optimization, enabling compilers to predict the effects of code transformations on performance and guide search for optimal transformations. Building state-of-the-art performance models with deep learning, however, requires vast labeled datasets of random programs -- an expensive and time-consuming process, stretching over months. This paper introduces a self-supervised pre-training scheme with autoencoders to reduce the need for labeled data. By pre-training on a large dataset of random programs, the autoencoder learns representations of code and transformations, which are then used to embed programs for the performance model. Implemented in the Tiramisu autoscheduler, our approach improves model accuracy with less data. For example, to achieve a MAPE of 20.72%, the original model requires 18 million data points, whereas our method achieves a similar MAPE of 22.44% with only 3.6 million data points, reducing data requirements by 5x.