# Predicting Program Exit Code with LLMs and Programming Language Semantics

arXiv: https://arxiv.org/abs/2609.00579 · HTML: https://mlsanity.com/paper/2609.00579/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 27 of 27)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [LongCoder: A Long-Range Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2306.14893/) (2023)
- [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024)
- [NExT: Teaching Large Language Models to Reason about Code Execution](https://mlsanity.com/paper/2404.14662/) (2024)
- [Exploring and Unleashing the Power of Large Language Models in Automated Code Translation](https://mlsanity.com/paper/2404.14646/) (2024)
- [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024)
- [A Deep Dive into Large Language Models for Automated Bug Localization and Repair](https://mlsanity.com/paper/2404.11595/) (2024)
- [Unveiling Memorization in Code Models](https://mlsanity.com/paper/2308.09932/) (2023)
- [InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translation](https://mlsanity.com/paper/2411.01063/) (2024)
- [Large Language Models as Code Executors: An Exploratory Study](https://mlsanity.com/paper/2410.06667/) (2024)
- [Code Summarization Beyond Function Level](https://mlsanity.com/paper/2502.16704/) (2025)
- [LLMs Lean on Priors, Not Programming Language Semantics](https://mlsanity.com/paper/2510.03415/) (2025)
- [How Does Naming Affect LLMs on Code Analysis Tasks?](https://mlsanity.com/paper/2307.12488/) (2023)
- [Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs](https://mlsanity.com/paper/2604.13997/) (2026)
- [CodeFlow: Program Behavior Prediction with Dynamic Dependencies Learning](https://mlsanity.com/paper/2408.02816/) (2024)
- [Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias](https://mlsanity.com/paper/2508.17361/) (2025)
- [Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding](https://mlsanity.com/paper/2505.13353/) (2025)
- [DCE-LLM: Dead Code Elimination with Large Language Models](https://mlsanity.com/paper/2506.11076/) (2025)
- [LLMs versus the Halting Problem: Characterizing Program Termination Reasoning](https://mlsanity.com/paper/2601.18987/) (2026)
- [Detecting Functional Memorization in Code Language Models](https://mlsanity.com/paper/2606.12764/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation. However, a key limitation in their performance may be their (lack of) understanding of programming-language semantics. Even when explicit semantics are given, it remains unclear whether LLMs apply those rules or lean on priors learned during pre-training instead. We study if LLMs lean on priors or given semantics with a novel task--Program Executability Prediction (PrEx)--that asks models to predict whether a program is semantically valid or invalid (and, if invalid, which formal rule it violates) given the program's syntax and operational semantics. Because PrEx requires both valid and invalid programs, we build a dataset with systematically generated invalid transformations derived from valid programs. We evaluate open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits. Our findings show that LLMs lean on pre-training priors rather than systematically applying the given rules, performing especially poorly on modified semantics and degrading further as program complexity increases. PrEx is available at https://github.com/EngineeringSoftware/prex.