mlsanityData to 7 Oct 2026

Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates

arXiv 2402.05980 ↗First posted 8 Feb 2024cs.SE cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Large Language Models' success on text generation has also made them better at code generation and coding tasks. While a lot of work has demonstrated their remarkable performance on tasks such as code completion and editing, it is still unclear as to why. We help bridge this gap by exploring to what degree auto-regressive models understand the logical constructs of the underlying programs. We propose Counterfactual Analysis for Programming Concept Predicates (CACP) as a counterfactual testing framework to evaluate whether Large Code Models understand programming concepts. With only black-box access to the model, we use CACP to evaluate ten popular Large Code Models for four different programming concepts. Our findings suggest that current models lack understanding of concepts such as data flow and control flow.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 26 of its 45 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

26 of 45 references.

References · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2023Teaching Large Language Models to Self-Debugcited by 963
  8. 2023StarCoder: may the source be with you!cited by 790
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
14 more
  1. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  2. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  3. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  4. 2021CURE: Code-Aware Neural Machine Translation for Automatic Program Repaircited by 85
  5. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  6. 2021Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanationscited by 53
  7. 2022CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Modelscited by 48
  8. 2020Semantic Robustness of Models of Source Codecited by 43
  9. 2020Typilus: Neural Type Hintscited by 29
  10. 2021Counterfactual Explanations for Models of Codecited by 24
  11. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  12. 2023Toward a Theory of Causation for Interpreting Neural Code Modelscited by 12
  13. 2023Evaluating and Explaining Large Language Models for Code Using Syntactic Structurescited by 9
  14. 2019Recovering Variable Names for Minified Code with Usage Contextscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 3.

14 papers.

Sort:

By area:

  1. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  2. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  3. 2026Large Language Model Reasoning Failurescited by 20
  4. 2024Understanding How CodeLLMs (Mis)Predict Types with Activation Steeringcited by 4
  5. 2025When Names Disappear: Revealing What LLMs Actually Understand About Codecited by 4
  6. 2026Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibilitycited by 4
  7. 2024Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Abilitycited by 3
  8. 2025Trust Me, I Know This Function: Hijacking LLM Static Analysis using Biascited by 3
  9. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settingscited by 3
  10. 2025Neuron-Guided Interpretation of Code LLMs: Where, Why, and How?cited by 3
  11. 2024CounterFace: A Synthetic Face Dataset for Fine-Grained Counterfactual Evaluation of Face Recognition Systemscited by 2
  12. 2025Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactualscited by 1
2 more
  1. 2026(How) Do Large Language Models Understand High-Level Message Sequence Charts?cited by 1
  2. 2025Auto-SPT: Automating Semantic Preserving Transformations for Codecited by 0