# Meta Large Language Model Compiler: Foundation Models of Compiler Optimization

arXiv: https://arxiv.org/abs/2407.02524 · HTML: https://mlsanity.com/paper/2407.02524/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 60 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 30 of 40)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023)
- [SantaCoder: don't reach for the stars!](https://mlsanity.com/paper/2301.03988/) (2023)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models](https://mlsanity.com/paper/2212.14834/) (2022)
- [Code Translation with Compiler Representations](https://mlsanity.com/paper/2207.03578/) (2022)
- [CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research](https://mlsanity.com/paper/2109.08267/) (2021)
- [MLGO: a Machine Learning Guided Compiler Optimizations Framework](https://mlsanity.com/paper/2101.04808/) (2021)
- [SLaDe: A Portable Small Language Model Decompiler for Optimized Assembly](https://mlsanity.com/paper/2305.12520/) (2023)
- [LLM-Vectorizer: LLM-based Verified Loop Vectorizer](https://mlsanity.com/paper/2406.04693/) (2024)
- [IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators](https://mlsanity.com/paper/2403.03894/) (2024)

## Built on by (most cited first, 30 of 39)

- [CWM: An Open-Weights LLM for Research on Code Generation with World Models](https://mlsanity.com/paper/2510.02387/) (2025)
- [AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?](https://mlsanity.com/paper/2507.15887/) (2025)
- [EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking](https://mlsanity.com/paper/2502.12466/) (2025)
- [cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree](https://mlsanity.com/paper/2506.15655/) (2025)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [Autocomp: A Powerful and Portable Code Optimizer for Tensor Accelerators](https://mlsanity.com/paper/2505.18574/) (2025)
- [Don't Transform the Code, Code the Transforms: Towards Precise Code Rewriting using LLMs](https://mlsanity.com/paper/2410.08806/) (2024)
- [ShieldedCode: Learning Robust Representations for Virtual Machine Protected Code](https://mlsanity.com/paper/2601.20679/) (2026)
- [MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs](https://mlsanity.com/paper/2411.03471/) (2024)
- [Compiler-R1: Towards Agentic Compiler Auto-tuning with Reinforcement Learning](https://mlsanity.com/paper/2506.15701/) (2025)
- [Tuning LLM-based Code Optimization via Meta-Prompting: An Industrial Perspective](https://mlsanity.com/paper/2508.01443/) (2025)
- [SuperCoder: Assembly Program Superoptimization with Large Language Models](https://mlsanity.com/paper/2505.11480/) (2025)
- [Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs](https://mlsanity.com/paper/2506.17353/) (2025)
- [Magellan: Autonomous Discovery of Novel Compiler Optimization Heuristics with AlphaEvolve](https://mlsanity.com/paper/2601.21096/) (2026)
- [ASMA-Tune: Unlocking LLMs' Assembly Code Comprehension via Structural-Semantic Instruction Tuning](https://mlsanity.com/paper/2503.11617/) (2025)
- [AwareCompiler: Agentic Context-Aware Compiler Optimization via a Synergistic Knowledge-Data Driven Framework](https://mlsanity.com/paper/2510.11759/) (2025)
- [Agentic Auto-Scheduling: An Experimental Study of LLM-Guided Loop Optimization](https://mlsanity.com/paper/2511.00592/) (2025)
- [Fine-tuned LLM-based Code Migration Framework](https://mlsanity.com/paper/2512.13515/) (2025)
- [Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decision](https://mlsanity.com/paper/2604.10182/) (2026)
- [Tadashi: Enabling AI-Based Automated Code Generation With Guaranteed Correctness](https://mlsanity.com/paper/2410.03210/) (2024)
- [Towards LLM-based optimization compilers. Can LLMs learn how to apply a single peephole optimization? Reasoning is all LLMs need!](https://mlsanity.com/paper/2412.12163/) (2024)
- [Can Large Language Models Understand Intermediate Representations in Compilers?](https://mlsanity.com/paper/2502.06854/) (2025)
- [Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHA](https://mlsanity.com/paper/2503.05840/) (2025)
- [Omniwise: Predicting GPU Kernels Performance with LLMs](https://mlsanity.com/paper/2506.20886/) (2025)
- [Rethinking LLM-Based RTL Code Optimization Via Timing Logic Metamorphosis](https://mlsanity.com/paper/2507.16808/) (2025)
- [LOOPRAG: Enhancing Loop Transformation Optimization with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2512.15766/) (2025)
- [AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning](https://mlsanity.com/paper/2606.20373/) (2026)
- [Guaranteed Guess: A Language Modeling Approach for CISC-to-RISC Transpilation with Testing Guarantees](https://mlsanity.com/paper/2506.14606/) (2025)
- [CRINN: Contrastive Reinforcement Learning for Approximate Nearest Neighbor Search](https://mlsanity.com/paper/2508.02091/) (2025)
- [QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code](https://mlsanity.com/paper/2511.01183/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities across a variety of software engineering and coding tasks. However, their application in the domain of code and compiler optimization remains underexplored. Training LLMs is resource-intensive, requiring substantial GPU hours and extensive data collection, which can be prohibitive. To address this gap, we introduce Meta Large Language Model Compiler (LLM Compiler), a suite of robust, openly available, pre-trained models specifically designed for code optimization tasks. Built on the foundation of Code Llama, LLM Compiler enhances the understanding of compiler intermediate representations (IRs), assembly language, and optimization techniques. The model has been trained on a vast corpus of 546 billion tokens of LLVM-IR and assembly code and has undergone instruction fine-tuning to interpret compiler behavior. LLM Compiler is released under a bespoke commercial license to allow wide reuse and is available in two sizes: 7 billion and 13 billion parameters. We also present fine-tuned versions of the model, demonstrating its enhanced capabilities in optimizing code size and disassembling from x86_64 and ARM assembly back into LLVM-IR. These achieve 77% of the optimising potential of an autotuning search, and 45% disassembly round trip (14% exact match). This release aims to provide a scalable, cost-effective foundation for further research and development in compiler optimization by both academic researchers and industry practitioners.