# Assessing Code Understanding in LLMs

arXiv: https://arxiv.org/abs/2504.00065 · HTML: https://mlsanity.com/paper/2504.00065/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Fine-tuning ChatGPT for Automatic Scoring](https://mlsanity.com/paper/2310.10072/) (2023)
- [Semantic Robustness of Models of Source Code](https://mlsanity.com/paper/2002.03043/) (2020)
- [Large Language Models for Code Analysis: Do LLMs Really Do Their Job?](https://mlsanity.com/paper/2310.12357/) (2023)
- [A Review of Formal Methods applied to Machine Learning](https://mlsanity.com/paper/2104.02466/) (2021)
- [A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding](https://mlsanity.com/paper/2101.07910/) (2021)
- [Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations](https://mlsanity.com/paper/2004.07313/) (2020)
- [Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis](https://mlsanity.com/paper/2412.14841/) (2024)
- [Neural Networks Reduction via Lumping](https://mlsanity.com/paper/2209.07475/) (2022)
- [Vulnerability Detection in Ethereum Smart Contracts via Machine Learning: A Qualitative Analysis](https://mlsanity.com/paper/2407.18639/) (2024)

## Built on by (most cited first, 2 of 2)

- [Improving LLM Code Reasoning via Semantic Equivalence Self-Play with Formal Verification](https://mlsanity.com/paper/2604.17010/) (2026)
- [Do not copy and paste! Rewriting strategies for code retrieval](https://mlsanity.com/paper/2605.08299/) (2026)

## Abstract (arXiv)

We present an empirical evaluation of Large Language Models in code understanding associated with non-trivial, semantic-preserving program transformations such as copy propagation or constant folding. Our findings show that LLMs fail to judge semantic equivalence in approximately 41% of cases when no context is provided and in 29% when given a simple generic context. To improve accuracy, we advocate integrating LLMs with code-optimization tools to enhance training and facilitate more robust program understanding.