mlsanityData to 7 Oct 2026

Large Language Models for Compiler Optimization

arXiv 2309.07062 ↗First posted 11 Sep 2023cs.PL cs.AI cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

We explore the novel application of Large Language Models to code optimization. We present a 7B-parameter transformer model trained from scratch to optimize LLVM assembly for code size. The model takes as input unoptimized assembly and outputs a list of compiler options to best optimize the program. Crucially, during training, we ask the model to predict the instruction counts before and after optimization, and the optimized code itself. These auxiliary learning tasks significantly improve the optimization performance of the model and improve the model's depth of understanding. We evaluate on a large suite of test programs. Our approach achieves a 3.0% improvement in reducing instruction counts over the compiler, outperforming two state-of-the-art baselines that require thousands of compilations. Furthermore, the model shows surprisingly strong code reasoning abilities, generating compilable code 91% of the time and perfectly emulating the output of the compiler 70% of the time.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 46 of its 67 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

46 of 67 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
34 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  3. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  4. 2023StarCoder: may the source be with you!cited by 790
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  6. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  7. 2022PAL: Program-aided Language Modelscited by 635
  8. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  9. 2023Textbooks Are All You Needcited by 561
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  12. 2020Unsupervised Translation of Programming Languagescited by 262
  13. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  14. 2023SantaCoder: don't reach for the stars!cited by 169
  15. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  16. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  17. 2022A Length-Extrapolatable Transformercited by 123
  18. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 102
  19. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  20. 2022Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learningcited by 76
  21. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  22. 2022Code Translation with Compiler Representationscited by 42
  23. 2021CompilerGym: Robust, Performant Compiler Optimization Environments for AI Researchcited by 41
  24. 2021MLGO: a Machine Learning Guided Compiler Optimizations Frameworkcited by 40
  25. 2019NeuroVectorizer: End-to-End Vectorization with Deep Reinforcement Learningcited by 37
  26. 2023SLaDe: A Portable Small Language Model Decompiler for Optimized Assemblycited by 28
  27. 2020AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learningcited by 24
  28. 2023Limits for Learning with Language Modelscited by 17
  29. 2022Beyond the C: Retargetable Decompilation using Neural Machine Translationcited by 15
  30. 2018Machine Learning in Compiler Optimisationcited by 13
  31. 2021Automated Conformance Testing for JavaScript Engines via Deep Compiler Fuzzingcited by 12
  32. 2021Learning C to x86 Translation: An Experiment in Neural Compilationcited by 12
  33. 2023Learning Compiler Pass Orders using Coreset and Normalized Value Predictioncited by 11
  34. 2022MLGOPerf: An ML Guided Inliner to Optimize Performancecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 19; 2026: 12.

52 papers.

Sort:

By area:

  1. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  3. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 82
  4. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  5. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  6. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2024LLM-Vectorizer: LLM-based Verified Loop Vectorizercited by 25
  9. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 25
  10. 2025EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checkingcited by 23
  11. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
  12. 2024UniCoder: Scaling Code Large Language Model via Universal Codecited by 18
38 more
  1. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  2. 2024How Multimodal Integration Boost the Performance of LLM for Optimization: Case Study on Capacitated Vehicle Routing Problemscited by 12
  3. 2025Autocomp: A Powerful and Portable Code Optimizer for Tensor Acceleratorscited by 12
  4. 2024Compiler generated feedback for Large Language Modelscited by 9
  5. 2024Priority Sampling of Large Language Models for Compilerscited by 8
  6. 2024LLM-Aided Compilation for Tensor Acceleratorscited by 8
  7. 2025Towards the Worst-case Robustness of Large Language Modelscited by 7
  8. 2025Compiler-R1: Towards Agentic Compiler Auto-tuning with Reinforcement Learningcited by 7
  9. 2025Tuning LLM-based Code Optimization via Meta-Prompting: An Industrial Perspectivecited by 7
  10. 2024CompilerDream: Learning a Compiler World Model for General Code Optimizationcited by 6
  11. 2026Magellan: Autonomous Discovery of Novel Compiler Optimization Heuristics with AlphaEvolvecited by 5
  12. 2025Improving Existing Optimization Algorithms with LLMscited by 4
  13. 2025VecTrans: Enhancing Compiler Auto-Vectorization through LLM-Assisted Code Transformationscited by 4
  14. 2025AwareCompiler: Agentic Context-Aware Compiler Optimization via a Synergistic Knowledge-Data Driven Frameworkcited by 4
  15. 2025Agentic Auto-Scheduling: An Experimental Study of LLM-Guided Loop Optimizationcited by 4
  16. 2026InCoder-32B-Thinking: Industrial Code World Model for Thinkingcited by 4
  17. 2025REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Servingcited by 3
  18. 2024MIREncoder: Multi-modal IR-based Pretrained Embeddings for Performance Optimizationscited by 2
  19. 2024ViC: Virtual Compiler Is All You Need For Assembly Code Searchcited by 2
  20. 2025Attention Distance: A Novel Metric for Directed Fuzzing with Large Language Modelscited by 2
  21. 2024Tadashi: Enabling AI-Based Automated Code Generation With Guaranteed Correctnesscited by 1
  22. 2024Towards LLM-based optimization compilers. Can LLMs learn how to apply a single peephole optimization? Reasoning is all LLMs need!cited by 1
  23. 2025Bringing Structure to Naturalness: On the Naturalness of ASTscited by 1
  24. 2025LOOPRAG: Enhancing Loop Transformation Optimization with Retrieval-Augmented Large Language Modelscited by 1
  25. 2026Protean Compiler: An Agile Framework to Drive Fine-grain Phase Orderingcited by 1
  26. 2024Green Runner: A tool for efficient deep learning component selectioncited by 0
  27. 2024The why, what, and how of AI-based coding in scientific researchcited by 0
  28. 2025Data-efficient Performance Modeling via Pre-trainingcited by 0
  29. 2025Behavioral Embeddings of Programs: A Quasi-Dynamic Approach for Optimization Predictioncited by 0
  30. 2025QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Codecited by 0
  31. 2025Exploring the Feasibility of End-to-End Large Language Model as a Compilercited by 0
  32. 2026Rudder: Steering Prefetching in Distributed GNN Training using LLM Agentscited by 0
  33. 2026LLM Translation of Compiler Intermediate Representationcited by 0
  34. 2026Step-TP: A Grounded, Step-Level Dataset with Chain-of-Thought Reasoning for LLM-Guided Tensor Program Optimizationcited by 0
  35. 2026TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Frameworkcited by 0
  36. 2026AgentCompile: An LLM-Guided Compiler for Direct CUDA Inferencecited by 0
  37. 2026Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizationscited by 0
  38. 2026Position: Quantum Program Generation Must Prioritize Validity Over Probabilistic Scalingcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Should AI Optimize Your Code? A Comparative Study of Classical Optimizing Compilers Versus Current Large Language Models
  2. 2024Optimizing High-Level Synthesis Designs with Retrieval-Augmented Large Language Models
  3. 2026LiteCoOp: Lightweight Multi-LLM Shared-Tree Reasoning for Model-Serving Compiler Optimizations
  4. 2025Can Large Language Models Understand Intermediate Representations in Compilers?
  5. 2025Small Language Models as Compiler Experts: Auto-Parallelization for Heterogeneous Systems
  6. 2026AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning
  7. 2025LPO: Discovering Missed Peephole Optimizations with Large Language Models
  8. 2025LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests ↗