# FAIR: Flow Type-Aware Pre-Training of Compiler Intermediate Representations

arXiv: https://arxiv.org/abs/2309.04828 · HTML: https://mlsanity.com/paper/2309.04828/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2023.

## Built on by (most cited first, 4 of 4)

- [Can Large Language Models Understand Intermediate Representations in Compilers?](https://mlsanity.com/paper/2502.06854/) (2025)
- [Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents](https://mlsanity.com/paper/2602.23556/) (2026)
- [The New Compiler Stack: A Survey on the Synergy of LLMs and Compilers](https://mlsanity.com/paper/2601.02045/) (2026)
- [LLM-Based Embeddings for Program Analysis and Optimization](https://mlsanity.com/paper/2608.07894/) (2026)

## Abstract (arXiv)

While the majority of existing pre-trained models from code learn source code features such as code tokens and abstract syntax trees, there are some other works that focus on learning from compiler intermediate representations (IRs). Existing IR-based models typically utilize IR features such as instructions, control and data flow graphs (CDFGs), call graphs, etc. However, these methods confuse variable nodes and instruction nodes in a CDFG and fail to distinguish different types of flows, and the neural networks they use fail to capture long-distance dependencies and have over-smoothing and over-squashing problems. To address these weaknesses, we propose FAIR, a Flow type-Aware pre-trained model for IR that involves employing (1) a novel input representation of IR programs; (2) Graph Transformer to address over-smoothing, over-squashing and long-dependencies problems; and (3) five pre-training tasks that we specifically propose to enable FAIR to learn the semantics of IR tokens, flow type information, and the overall representation of IR. Experimental results show that FAIR can achieve state-of-the-art results on four code-related downstream tasks.