mlsanityData to 14 Oct 2026

Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models

arXiv 2509.11686 ↗First posted 15 Sep 2025cs.SE cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Code Large Language Models (Code LLMs) have opened a new era in programming with their impressive capabilities. However, recent research has revealed critical limitations in their ability to reason about runtime behavior and understand the actual functionality of programs, which poses significant challenges for their post-training and practical deployment. Specifically, Code LLMs encounter two principal issues: (1) a lack of proficiency in reasoning about program execution behavior, as they struggle to interpret what programs actually do during runtime, and (2) the inconsistent and fragmented representation of semantic information, such as execution traces, across existing methods, which hinders their ability to generalize and reason effectively. These challenges underscore the necessity for more systematic approaches to enhance the reasoning capabilities of Code LLMs. To address these issues, we introduce a generic framework to support integrating semantic information~(e.g., execution trace) to code task-relevant prompts, and conduct a comprehensive study to explore the role of semantic information in enhancing the reasoning ability of Code LLMs accordingly. Specifically, we focus on investigating the usefulness of trace-based semantic information in boosting supervised fine-tuning~(SFT) and post-phase inference of Code LLMs. The experimental results surprisingly disagree with previous works and demonstrate that semantic information has limited usefulness for SFT and test time scaling of Code LLM.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

23 of 36 references.

References · 23

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  7. 2023Teaching Large Language Models to Self-Debugcited by 988
  8. 2021Measuring Coding Challenge Competence With APPScited by 902
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  10. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  12. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
11 more
  1. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  2. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 144
  3. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  4. 2020Contrastive Code Representation Learningcited by 76
  5. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 63
  6. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 62
  7. 2024Finetuning Large Language Models for Vulnerability Detectioncited by 40
  8. 2024Densing Law of LLMscited by 40
  9. 2024SemCoder: Training Code Language Models with Comprehensive Semantics Reasoningcited by 27
  10. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  11. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  2. 2025WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Qualitycited by 11
  3. 2026Debugging code world modelscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
  2. 2026An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning
  3. 2025A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models
  4. 2026Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
  5. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
  6. 2025Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference
  7. 2024CodeMind: Evaluating Large Language Models for Code Reasoning
  8. 2025Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs