mlsanityData to 14 Oct 2026

Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain

arXiv 2310.14053 ↗First posted 21 Oct 2023cs.LG cs.CL cs.SEcited by 31

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Code Large Language Models (Code LLMs) are being increasingly employed in real-life applications, so evaluating them is critical. While the conventional accuracy evaluates the performance of Code LLMs on a set of individual tasks, their self-consistency across different tasks is overlooked. Intuitively, a trustworthy model should be self-consistent when generating natural language specifications for its own code and generating code for its own specifications. Failure to preserve self-consistency reveals a lack of understanding of the shared semantics underlying natural language and programming language, and therefore undermines the trustworthiness of a model. In this paper, we first formally define the self-consistency of Code LLMs and then design a framework, IdentityChain, which effectively and efficiently evaluates the self-consistency and conventional accuracy of a model at the same time. We study eleven Code LLMs and show that they fail to preserve self-consistency, which is indeed a distinct aspect from conventional accuracy. Furthermore, we show that IdentityChain can be used as a model debugging tool to expose weaknesses of Code LLMs by demonstrating three major weaknesses that we identify in current models using IdentityChain. Our code is available at https://github.com/marcusm117/IdentityChain.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 22 of its 27 references that are in the map. First posted on arXiv 21 Oct 2023.

Builds on

22 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  6. 2021Measuring Coding Challenge Competence With APPScited by 902
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  8. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  9. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  10. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  12. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
9 more
  1. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  2. 2018Mapping Language to Code in Programmatic Contextcited by 104
  3. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
  4. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 78
  5. 2019A Logic-Driven Framework for Consistency of Neural Modelscited by 64
  6. 2018Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledgecited by 46
  7. 2023TRACED: Execution-aware Pre-training for Source Codecited by 27
  8. 2021Towards Learning (Dis)-Similarity of Source Code from Program Contrastscited by 17
  9. 2023Separating form and meaning: Using self-consistency to quantify task understanding across multiple sensescited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 15; 2026: 7.

31 papers.

Sort:

By area:

  1. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  2. 2024CodeMind: Evaluating Large Language Models for Code Reasoningcited by 32
  3. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  4. 2025Measuring General Intelligence with Generated Gamescited by 24
  5. 2024Unsupervised Evaluation of Code LLMs with Round-Trip Correctnesscited by 23
  6. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  7. 2025ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generationcited by 11
  8. 2023Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Codecited by 9
  9. 2025SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinationscited by 9
  10. 2026LLMs Corrupt Your Documents When You Delegatecited by 9
  11. 2025A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Modelscited by 7
  12. 2025Leveraging Large Language Models to Develop Heuristics for Emerging Optimization Problemscited by 6
19 more
  1. 2025Chain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarizationcited by 6
  2. 2024tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs)cited by 5
  3. 2025ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilitiescited by 5
  4. 2025Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Modelscited by 5
  5. 2026Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibilitycited by 4
  6. 2025Preventing Rogue Agents Improves Multi-Agent Collaborationcited by 3
  7. 2025Integrating Expert Knowledge into Logical Programs via LLMscited by 3
  8. 2026SHIFT: Gate-Modulated Activation Steering for Knowledge Conflict Mitigation in Retrieval-Augmented Generationcited by 3
  9. 2025CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Modelscited by 2
  10. 2026Probing the Lack of Stable Internal Beliefs in LLMscited by 2
  11. 2024Knowledge-based Consistency Testing of Large Language Modelscited by 1
  12. 2025Can Large Language Models Understand Intermediate Representations in Compilers?cited by 1
  13. 2025EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programmingcited by 1
  14. 2025Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMscited by 1
  15. 2026Algorithmic Prompt-Augmentation for Efficient LLM-Based Heuristic Design for A* Searchcited by 1
  16. 2026Boosting Self-Consistency with Rankingcited by 1
  17. 2024LeCov: Multi-level Testing Criteria for Large Language Modelscited by 0
  18. 2025Learning by Teaching: Engaging Students as Instructors of Large Language Models in Computer Science Educationcited by 0
  19. 2026Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency
  2. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  3. 2026Evaluating Language Models on Cross-Language Code Functional Equivalence
  4. 2026ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation
  5. 2025Existing LLMs Are Not Self-Consistent For Simple Tasks
  6. 2025Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?
  7. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules
  8. 2025On Code-Induced Reasoning in LLMs