# Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptions

arXiv: https://arxiv.org/abs/2203.03771 · HTML: https://mlsanity.com/paper/2203.03771/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Neural GPUs Learn Algorithms](https://mlsanity.com/paper/1511.08228/) (2015)
- [Learning to Execute](https://mlsanity.com/paper/1410.4615/) (2014)
- [Neural Programmer-Interpreters](https://mlsanity.com/paper/1511.06279/) (2015)
- [DeepBugs: A Learning Approach to Name-based Bug Detection](https://mlsanity.com/paper/1805.11683/) (2018)
- [Self-Supervised Bug Detection and Repair](https://mlsanity.com/paper/2105.12787/) (2021)
- [Neural Program Repair by Jointly Learning to Localize and Repair](https://mlsanity.com/paper/1904.01720/) (2019)
- [How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset](https://mlsanity.com/paper/1905.13334/) (2019)
- [Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks](https://mlsanity.com/paper/2010.12621/) (2020)
- [Programming with a Differentiable Forth Interpreter](https://mlsanity.com/paper/1605.06640/) (2016)
- [Differentiable Programs with Neural Libraries](https://mlsanity.com/paper/1611.02109/) (2016)
- [Representing Partial Programs with Blended Abstract Semantics](https://mlsanity.com/paper/2012.12964/) (2020)
- [Learning to Combine Per-Example Solutions for Neural Program Synthesis](https://mlsanity.com/paper/2106.07175/) (2021)
- [Comparing the Max and Noisy-Or Pooling Functions in Multiple Instance Learning for Weakly Supervised Sequence Learning Tasks](https://mlsanity.com/paper/1804.01146/) (2018)
- [Beep: Fine-grained Fix Localization by Learning to Predict Buggy Code Elements](https://mlsanity.com/paper/2111.07739/) (2021)

## Built on by (most cited first, 11 of 11)

- [NExT: Teaching Large Language Models to Reason about Code Execution](https://mlsanity.com/paper/2404.14662/) (2024)
- [Code Representation Learning At Scale](https://mlsanity.com/paper/2402.01935/) (2024)
- [What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces](https://mlsanity.com/paper/2503.05703/) (2025)
- [Exploiting Code Symmetries for Learning Program Semantics](https://mlsanity.com/paper/2308.03312/) (2023)
- [LExecutor: Learning-Guided Execution](https://mlsanity.com/paper/2302.02343/) (2023)
- [SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation](https://mlsanity.com/paper/2310.00796/) (2023)
- [A Library for Representing Python Programs as Graphs for Machine Learning](https://mlsanity.com/paper/2208.07461/) (2022)
- [COMEX: A Tool for Generating Customized Source Code Representations](https://mlsanity.com/paper/2307.04693/) (2023)
- [REDO: Execution-Free Runtime Error Detection for COding Agents](https://mlsanity.com/paper/2410.09117/) (2024)
- [How Robustly do LLMs Understand Execution Semantics?](https://mlsanity.com/paper/2604.16320/) (2026)
- [Towards a Neural Debugger for Python](https://mlsanity.com/paper/2603.09951/) (2026)

## Abstract (arXiv)

The execution behavior of a program often depends on external resources, such as program inputs or file contents, and so cannot be run in isolation. Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors as early as possible, even before programs can be compiled and run. This presents an interesting machine learning challenge: can we predict runtime errors in a "static" setting, where program execution is not possible? Here, we introduce a real-world dataset and task for predicting runtime errors, which we show is difficult for generic models like Transformers. We approach this task by developing an interpreter-inspired architecture with an inductive bias towards mimicking program executions, which models exception handling and "learns to execute" descriptions of the contents of external resources. Surprisingly, we show that the model can also predict the location of the error, despite being trained only on labels indicating the presence/absence and kind of error. In total, we present a practical and difficult-yet-approachable challenge problem related to learning program execution and we demonstrate promising new capabilities of interpreter-inspired machine learning models for code.