# SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning

arXiv: https://arxiv.org/abs/2406.01006 · HTML: https://mlsanity.com/paper/2406.01006/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 8 of its 57 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 8 of 8)

- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)

## Built on by (most cited first, 27 of 27)

- [Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://mlsanity.com/paper/2502.19411/) (2025)
- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [Reverse Thinking Makes LLMs Stronger Reasoners](https://mlsanity.com/paper/2411.19865/) (2024)
- [EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking](https://mlsanity.com/paper/2502.12466/) (2025)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis](https://mlsanity.com/paper/2503.23145/) (2025)
- [What can Large Language Models Capture about Code Functional Equivalence?](https://mlsanity.com/paper/2408.11081/) (2024)
- [CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning](https://mlsanity.com/paper/2506.00750/) (2025)
- [Scaling Parameter-Constrained Language Models with Quality Data](https://mlsanity.com/paper/2410.03083/) (2024)
- [What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces](https://mlsanity.com/paper/2503.05703/) (2025)
- [The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation](https://mlsanity.com/paper/2505.18759/) (2025)
- [Locus: Agentic Predicate Synthesis for Directed Fuzzing](https://mlsanity.com/paper/2508.21302/) (2025)
- [CoCoNUT: Structural Code Understanding does not fall out of a tree](https://mlsanity.com/paper/2501.16456/) (2025)
- [A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models](https://mlsanity.com/paper/2501.18482/) (2025)
- [Disproving Program Equivalence with LLMs](https://mlsanity.com/paper/2502.18473/) (2025)
- [Trace Gadgets: Minimizing Code Context for Machine Learning-Based Vulnerability Prediction](https://mlsanity.com/paper/2504.13676/) (2025)
- [FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning](https://mlsanity.com/paper/2601.19001/) (2026)
- [The Hitchhiker's Guide to Program Analysis, Part II: Deep Thoughts by LLMs](https://mlsanity.com/paper/2504.11711/) (2025)
- [EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention](https://mlsanity.com/paper/2508.16771/) (2025)
- [Mastering the Craft of Data Synthesis for CodeLLMs](https://mlsanity.com/paper/2411.00005/) (2024)
- [Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models](https://mlsanity.com/paper/2509.11686/) (2025)
- [CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Models](https://mlsanity.com/paper/2503.23231/) (2025)
- [Morescient GAI for Software Engineering (Extended Version)](https://mlsanity.com/paper/2406.04710/) (2024)
- [Investigating Execution-Aware Language Models for Code Optimization](https://mlsanity.com/paper/2503.08228/) (2025)
- [Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning](https://mlsanity.com/paper/2605.17958/) (2026)
- [Generating Verifiable Chain of Thoughts from Exection-Traces](https://mlsanity.com/paper/2512.00127/) (2025)
- [Improved Bug Localization with AI Agents Leveraging Hypothesis and Dynamic Cognition](https://mlsanity.com/paper/2601.12522/) (2026)

## Abstract (arXiv)

Code Large Language Models (Code LLMs) have excelled at tasks like code completion but often miss deeper semantics such as execution effects and dynamic states. This paper aims to bridge the gap between Code LLMs' reliance on static text data and the need for semantic understanding for complex tasks like debugging and program repair. We introduce a novel strategy, monologue reasoning, to train Code LLMs to reason comprehensive semantics, encompassing high-level functional descriptions, local execution effects of individual statements, and overall input/output behavior, thereby linking static code text with dynamic execution states. We begin by collecting PyX, a clean Python corpus of fully executable code samples with functional descriptions and test cases. We propose training Code LLMs not only to write code but also to understand code semantics by reasoning about key properties, constraints, and execution behaviors using natural language, mimicking human verbal debugging, i.e., rubber-duck debugging. This approach led to the development of SemCoder, a Code LLM with only 6.7B parameters, which shows competitive performance with GPT-3.5-turbo on code generation and execution reasoning tasks. SemCoder achieves 79.3% on HumanEval (GPT-3.5-turbo: 76.8%), 63.6% on CRUXEval-I (GPT-3.5-turbo: 50.3%), and 63.9% on CRUXEval-O (GPT-3.5-turbo: 59.0%). We also study the effectiveness of SemCoder's monologue-style execution reasoning compared to concrete scratchpad reasoning, showing that our approach integrates semantics from multiple dimensions more smoothly. Finally, we demonstrate the potential of applying learned semantics to improve Code LLMs' debugging and self-refining capabilities. Our data, code, and models are available at: https://github.com/ARiSE-Lab/SemCoder.