mlsanityData to 7 Oct 2026

Learning Performance-Improving Code Edits

arXiv 2302.07867 ↗First posted 15 Feb 2023cs.SE cs.AI cs.LGcited by 103

Not on the map. Why?

Abstract (arXiv)

With the decline of Moore's law, optimizing program performance has become a major focus of software research. However, high-level optimizations such as API and algorithm changes remain elusive due to the difficulty of understanding the semantics of code. Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks. To that end, we introduce a framework for adapting LLMs to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs, accompanied by extensive unit tests. A major challenge is the significant variability of measuring performance on commodity hardware, which can lead to spurious "improvements." To isolate and reliably evaluate the impact of program optimizations, we design an environment based on the gem5 full system simulator, the de facto simulator used in academia and industry. Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play. A combination of these techniques achieves a mean speedup of 6.86 with eight generations, higher than average optimizations from individual programmers (3.66). Using our model's fastest generations, we set a new upper limit on the fastest speedup possible for our dataset at 9.64 compared to using the fastest human submissions available (9.56).

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 50 of its 94 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

50 of 94 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
38 more
  1. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  5. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  6. 2022CodeT: Code Generation with Generated Testscited by 369
  7. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  8. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  9. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  10. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  11. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  12. 2016DeepCoder: Learning to Write Programscited by 145
  13. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  14. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  15. 2018Automated Algorithm Selection: Survey and Perspectivescited by 111
  16. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  17. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  18. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  19. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 93
  20. 2016Neuro-Symbolic Program Synthesiscited by 84
  21. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 83
  22. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  23. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  24. 2023TrojanPuzzle: Covertly Poisoning Code-Suggestion Modelscited by 47
  25. 2019Learning to Infer Program Sketchescited by 44
  26. 2018Neural-Guided Deductive Search for Real-Time Program Synthesis from Examplescited by 39
  27. 2021An Evaluation of Edge TPU Accelerators for Convolutional Neural Networkscited by 39
  28. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  29. 2021Data-Driven Offline Optimization For Architecting Hardware Acceleratorscited by 29
  30. 2017Neural Program Meta-Inductioncited by 23
  31. 2020Learning to Represent Programs with Property Signaturescited by 16
  32. 2017Souper: A Synthesizing Superoptimizercited by 12
  33. 2022DeepPERF: A Deep Learning-Based Approach For Improving Software Performancecited by 6
  34. 2022Triangulating Python Performance Issues with Scalenecited by 5
  35. 2021Learning to Make Compiler Optimizations More Effectivecited by 4
  36. 2021Learning to Superoptimize Real-world Programscited by 4
  37. 2022Learning to Improve Code Efficiencycited by 4
  38. 2022MAGPIE: Machine Automated General Performance Improvement via Evolution of Softwarecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 33; 2025: 35; 2026: 16.

103 papers.

Sort:

By area:

  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2023Teaching Large Language Models to Self-Debugcited by 963
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  5. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  6. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  7. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 266
  8. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
  9. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  10. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  11. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  12. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
38 more
  1. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
  2. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  3. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 83
  4. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learningcited by 74
  5. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  6. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  7. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
  8. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 55
  9. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  10. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
  11. 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39
  12. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  13. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  14. 2023InstructCoder: Instruction Tuning Large Language Models for Code Editingcited by 28
  15. 2024EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimizationcited by 28
  16. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 27
  17. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  18. 2024Search-Based LLMs for Code Optimizationcited by 27
  19. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  20. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Codecited by 24
  21. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  22. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 24
  23. 2024NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctnesscited by 23
  24. 2025EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checkingcited by 23
  25. 2024CodeEditorBench: Evaluating Code Editing Capability of Large Language Modelscited by 22
  26. 2024PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedbackcited by 22
  27. 2023ChatGPT Participates in a Computer Science Examcited by 19
  28. 2024SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language modelscited by 18
  29. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  30. 2023Supersonic: Learning to Generate Source Code Optimizations in C/C++cited by 15
  31. 2024EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuningcited by 15
  32. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 15
  33. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimizationcited by 14
  34. 2025Autocomp: A Powerful and Portable Code Optimizer for Tensor Acceleratorscited by 12
  35. 2025SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimizationcited by 11
  36. 2023PerfRL: A Small Language Model Framework for Efficient Code Optimizationcited by 8
  37. 2024Python is Not Always the Best Choice: Embracing Multilingual Program of Thoughtscited by 8
  38. 2024Don't Transform the Code, Code the Transforms: Towards Precise Code Rewriting using LLMscited by 8

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs
  2. 2024Performance-Aligned LLMs for Generating Fast Code
  3. 2026Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination
  4. 2026EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems
  5. 2026Rethinking Code Performance Benchmarks for LLMs
  6. 2025Evaluating and Improving Large Language Models for Competitive Program Generation
  7. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  8. 2025Seed-Coder: Let the Code Model Curate Data for Itself