mlsanityData to 7 Oct 2026

What can Large Language Models Capture about Code Functional Equivalence?

arXiv 2408.11081 ↗First posted 20 Aug 2024cs.SE cs.AI cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Code-LLMs, LLMs pre-trained on large code corpora, have shown great progress in learning rich representations of the structure and syntax of code, successfully using it to generate or classify code fragments. At the same time, understanding if they are able to do so because they capture code semantics, and how well, is still an open question. In this paper, we tackle this problem by introducing SeqCoBench, a benchmark for systematically assessing how Code-LLMs can capture code functional equivalence. SeqCoBench contains over 20 code transformations that either preserve or alter the semantics of Python programs. We conduct extensive evaluations in different settings, including zero-shot and parameter-efficient finetuning methods on state-of-the-art (Code)-LLMs to see if they can discern semantically equivalent or different pairs of programs in SeqCoBench. We find that the performance gap between these LLMs and classical match-based retrieval scores is minimal, with both approaches showing a concerning lack of depth in understanding code semantics.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

35 of 54 references.

References · 35

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  12. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
23 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  3. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  4. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  5. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  6. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  7. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  8. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  9. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  10. 2022Natural Attack for Pre-trained Models of Codecited by 72
  11. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  12. 2023CodeScore: Evaluating Code Generation by Learning Code Executioncited by 66
  13. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  14. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  15. 2022CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Modelscited by 48
  16. 2020On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformationscited by 41
  17. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 31
  18. 2023The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Pythoncited by 29
  19. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 27
  20. 2024SemCoder: Training Code Language Models with Comprehensive Semantics Reasoningcited by 27
  21. 2023A Static Evaluation of Code Completion by Large Language Modelscited by 14
  22. 2023Backdooring Neural Code Searchcited by 13
  23. 2020Embedding Java Classes with code2vec: Improvements from Variable Obfuscationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checkingcited by 23
  2. 2025LLMs Lean on Priors, Not Programming Language Semanticscited by 5
  3. 2025PyVeritas: On Verifying Python via LLM-Based Transpilation and Bounded Model Checking for Ccited by 4
  4. 2025MatchFixAgent: Language-Agnostic Autonomous Repository-Level Code Translation Validation and Repaircited by 3
  5. 2026RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbationscited by 3
  6. 2025LLM-Supported Natural Language to Bash Translationcited by 2
  7. 2025Program Semantic Inequivalence Game with Large Language Modelscited by 1
  8. 2025CETBench: A Novel Dataset constructed via Transformations over Programs for Benchmarking LLMs for Code-Equivalence Checkingcited by 1
  9. 2025AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibrationcited by 1
  10. 2026Rethinking Code Similarity for Automated Algorithm Design with LLMscited by 1
  11. 2026FASE: Fast Adaptive Semantic Entropy for Code Qualitycited by 1
  12. 2026Evaluating Language Models on Cross-Language Code Functional Equivalencecited by 1
1 more
  1. 2026LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoningcited by 0