mlsanityData to 14 Oct 2026

Assessing Code Understanding in LLMs

arXiv 2504.00065 ↗First posted 31 Mar 2025cs.SE cs.AI cs.PLcited by 2

Not on the map. Why?

Abstract (arXiv)

We present an empirical evaluation of Large Language Models in code understanding associated with non-trivial, semantic-preserving program transformations such as copy propagation or constant folding. Our findings show that LLMs fail to judge semantic equivalence in approximately 41% of cases when no context is provided and in 29% when given a simple generic context. To improve accuracy, we advocate integrating LLMs with code-optimization tools to enhance training and facilitate more robust program understanding.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 17 of its 32 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

17 of 32 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2024Mixtral of Expertscited by 1,823
  4. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  5. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  6. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  7. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  8. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  9. 2023Fine-tuning ChatGPT for Automatic Scoringcited by 51
  10. 2020Semantic Robustness of Models of Source Codecited by 45
  11. 2023Large Language Models for Code Analysis: Do LLMs Really Do Their Job?cited by 45
  12. 2021A Review of Formal Methods applied to Machine Learningcited by 33
5 more
  1. 2021A Search-Based Testing Framework for Deep Neural Networks of Source Code Embeddingcited by 13
  2. 2020Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformationscited by 9
  3. 2024Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysiscited by 6
  4. 2022Neural Networks Reduction via Lumpingcited by 2
  5. 2024Vulnerability Detection in Ethereum Smart Contracts via Machine Learning: A Qualitative Analysiscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Improving LLM Code Reasoning via Semantic Equivalence Self-Play with Formal Verificationcited by 0
  2. 2026Do not copy and paste! Rewriting strategies for code retrievalcited by 0