mlsanityData to 14 Oct 2026

Code Execution with Pre-trained Language Models

arXiv 2305.05383 ↗First posted 8 May 2023cs.PL cs.AI cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Code execution is a fundamental aspect of programming language semantics that reflects the exact behavior of the code. However, most pre-trained models for code intelligence ignore the execution trace and only rely on source code and syntactic structures. In this paper, we investigate how well pre-trained models can understand and perform code execution. We develop a mutation-based data augmentation technique to create a large-scale and realistic Python dataset and task for code execution, which challenges existing models such as Codex. We then present CodeExecutor, a Transformer model that leverages code execution pre-training and curriculum learning to enhance its semantic comprehension. We evaluate CodeExecutor on code execution and show its promising performance and limitations. We also demonstrate its potential benefits for code intelligence tasks such as zero-shot code-to-code search and text-to-code generation. Our analysis provides insights into the learning and generalization abilities of pre-trained models for code execution.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

35 of 51 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  9. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  11. 2020Efficient Transformers: A Surveycited by 790
  12. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
23 more
  1. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  2. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  4. 2018Universal Transformerscited by 647
  5. 2014Neural Turing Machinescited by 644
  6. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  8. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  9. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  10. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  11. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  12. 2015Neural GPUs Learn Algorithmscited by 121
  13. 2019Neural Execution of Graph Algorithmscited by 121
  14. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112
  15. 2014Learning to Executecited by 106
  16. 2015Neural Programmer-Interpreterscited by 101
  17. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  18. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 54
  19. 2020Pointer Graph Networkscited by 53
  20. 2020Neural Execution Engines: Learning to Execute Subroutinescited by 36
  21. 2020Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networkscited by 35
  22. 2015Neural Random-Access Machinescited by 34
  23. 2020Learning Semantic Program Embeddings with Graph Interval Neural Networkcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 13; 2026: 10.

34 papers.

Sort:

By area:

  1. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  2. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  3. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 45
  4. 2025EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checkingcited by 24
  5. 2024Can Transformers Reason Logically? A Study in SAT Solvingcited by 16
  6. 2025CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesiscited by 16
  7. 2024Code Simulation Challenges for Large Language Modelscited by 15
  8. 2024Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursioncited by 14
  9. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoningcited by 14
  10. 2024Beyond Graphs: Can Large Language Models Comprehend Hypergraphs?cited by 11
  11. 2023Predicting Code Coverage without Executioncited by 9
  12. 2023Code Representation Pre-training with Complements from Program Executionscited by 8
22 more
  1. 2025SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysiscited by 8
  2. 2025A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Modelscited by 7
  3. 2025Code Simulation as a Proxy for High-order Tasks in Large Language Modelscited by 7
  4. 2026Learning State-Tracking from Code Using Linear RNNscited by 6
  5. 2026ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoningcited by 6
  6. 2023Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generationcited by 5
  7. 2026Debugging code world modelscited by 5
  8. 2024Can Language Models Pretend Solvers? Logic Code Simulation with LLMscited by 4
  9. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  10. 2025Gradient-Based Program Repair: Fixing Bugs in Continuous Program Spacescited by 4
  11. 2025CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learningcited by 4
  12. 2025Gistify! Codebase-Level Understanding via Runtime Executioncited by 4
  13. 2025A Semantic-based Optimization Approach for Repairing LLMs: Case Study on Code Generationcited by 2
  14. 2026StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learningcited by 2
  15. 2025Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Taskscited by 1
  16. 2026Towards a Neural Debugger for Pythoncited by 1
  17. 2026Training Transformers as a Universal Computercited by 1
  18. 2025Themisto: Jupyter-Based Runtime Benchmarkcited by 0
  19. 2026SemRep: Generative Code Representation Learning with Code Transformationscited by 0
  20. 2026Teaching LLMs Program Semantics via Symbolic Execution Tracescited by 0
  21. 2026Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomycited by 0
  22. 2026PyINE: A Framework for Scalable Elicitation and Oversight via Code Executioncited by 0