# TRACED: Execution-aware Pre-training for Source Code

arXiv: https://arxiv.org/abs/2306.07487 · HTML: https://mlsanity.com/paper/2306.07487/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2023.

## Built on by (most cited first, 27 of 27)

- [Clover: Closed-Loop Verifiable Code Generation](https://mlsanity.com/paper/2310.17807/) (2023)
- [LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning](https://mlsanity.com/paper/2401.16185/) (2024)
- [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024)
- [Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain](https://mlsanity.com/paper/2310.14053/) (2023)
- [Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap](https://mlsanity.com/paper/2410.06107/) (2024)
- [Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion](https://mlsanity.com/paper/2401.12947/) (2024)
- [CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning](https://mlsanity.com/paper/2506.00750/) (2025)
- [Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs](https://mlsanity.com/paper/2505.04441/) (2025)
- [Exploiting Code Symmetries for Learning Program Semantics](https://mlsanity.com/paper/2308.03312/) (2023)
- [Towards Causal Deep Learning for Vulnerability Detection](https://mlsanity.com/paper/2310.07958/) (2023)
- [Locus: Agentic Predicate Synthesis for Directed Fuzzing](https://mlsanity.com/paper/2508.21302/) (2025)
- [A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2507.22659/) (2025)
- [CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking](https://mlsanity.com/paper/2402.11842/) (2024)
- [LLMs Lean on Priors, Not Programming Language Semantics](https://mlsanity.com/paper/2510.03415/) (2025)
- [Trace Gadgets: Minimizing Code Context for Machine Learning-Based Vulnerability Prediction](https://mlsanity.com/paper/2504.13676/) (2025)
- [Debugging code world models](https://mlsanity.com/paper/2602.07672/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [Gistify! Codebase-Level Understanding via Runtime Execution](https://mlsanity.com/paper/2510.26790/) (2025)
- [DynaFix: Iterative Automated Program Repair Driven by Execution-Level Dynamic Information](https://mlsanity.com/paper/2512.24635/) (2025)
- [StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning](https://mlsanity.com/paper/2605.11922/) (2026)
- [Morescient GAI for Software Engineering (Extended Version)](https://mlsanity.com/paper/2406.04710/) (2024)
- [Investigating Execution-Aware Language Models for Code Optimization](https://mlsanity.com/paper/2503.08228/) (2025)
- [Knowledge-Guided Prompt Learning for Request Quality Assurance in Public Code Review](https://mlsanity.com/paper/2410.21673/) (2024)
- [Themisto: Jupyter-Based Runtime Benchmark](https://mlsanity.com/paper/2504.12365/) (2025)
- [Generating Verifiable Chain of Thoughts from Exection-Traces](https://mlsanity.com/paper/2512.00127/) (2025)
- [Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study](https://mlsanity.com/paper/2609.03702/) (2026)

## Abstract (arXiv)

Most existing pre-trained language models for source code focus on learning the static code text, typically augmented with static code structures (abstract syntax tree, dependency graphs, etc.). However, program semantics will not be fully exposed before the real execution. Without an understanding of the program execution, statically pre-trained models fail to comprehensively capture the dynamic code properties, such as the branch coverage and the runtime variable values, and they are consequently less effective at code understanding tasks, such as retrieving semantic clones and detecting software vulnerabilities. To close the gap between the static nature of language models and the dynamic characteristics of programs, we introduce TRACED, an execution-aware pre-training strategy for source code. Specifically, we pre-train code language models with a combination of source code, executable inputs, and corresponding execution traces. Our goal is to teach code models the complicated execution logic during the pre-training, enabling the model to statically estimate the dynamic code properties without repeatedly executing code during task-specific fine-tuning. To illustrate the effectiveness of our proposed approach, we fine-tune and evaluate TRACED on three downstream tasks: static execution estimation, clone retrieval, and vulnerability detection. The empirical results show that TRACED relatively improves the statically pre-trained code models by 12.4% for complete execution path prediction and by 25.2% for runtime variable value predictions. TRACED also significantly outperforms statically pre-trained models in clone retrieval and vulnerability detection across four public benchmarks.