mlsanityData to 7 Oct 2026

Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Models

arXiv 2407.00456 ↗First posted 29 Jun 2024cs.SE cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have brought a paradigm shift to the field of code generation, offering the potential to enhance the software development process. However, previous research mainly focuses on the accuracy of code generation, while coding style differences between LLMs and human developers remain under-explored. In this paper, we empirically analyze the differences in coding style between the code generated by mainstream Code LLMs and the code written by human developers, and summarize coding style inconsistency taxonomy. Specifically, we first summarize the types of coding style inconsistencies by manually analyzing a large number of generation results. We then compare the code generated by Code LLMs with the code written by human programmers in terms of readability, conciseness, and robustness. The results reveal that LLMs and developers have different coding styles. Additionally, we study the possible causes of these inconsistencies and provide some solutions to alleviate the problem.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 61 of its 105 references that are in the map. First posted on arXiv 29 Jun 2024.

Builds on

61 of 105 references.

References · 61

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2023StarCoder: may the source be with you!cited by 790
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  10. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  11. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  12. 2023Self-collaboration Code Generation via ChatGPTcited by 255
38 more
  1. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  2. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  3. 2023Planning with Large Language Models for Code Generationcited by 179
  4. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  5. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 139
  6. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  7. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  8. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  9. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  10. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  11. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  12. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  13. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  14. 2023Towards an Understanding of Large Language Models in Software Engineering Taskscited by 67
  15. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  16. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  17. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 42
  18. 2023SkCoder: A Sketch-based Approach for Automatic Code Generationcited by 40
  19. 2023Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Modelscited by 37
  20. 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 37
  21. 2022StructCoder: Structure-Aware Transformer for Code Generationcited by 33
  22. 2023Evaluation of ChatGPT Model for Vulnerability Detectioncited by 33
  23. 2021Code Completion by Modeling Flattened Abstract Syntax Trees as Graphscited by 30
  24. 2023Distilled GPT for Source Code Summarizationcited by 30
  25. 2023Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attackscited by 29
  26. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  27. 2023Exploring Continual Learning for Code Generation Modelscited by 22
  28. 2024STALL+: Boosting LLM-based Repository-level Code Completion with Static Analysiscited by 21
  29. 2023Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyondcited by 20
  30. 2023Guiding Language Models of Code with Global Context using Monitorscited by 20
  31. 2023Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generationcited by 19
  32. 2024Analyzing the Performance of Large Language Models on Code Summarizationcited by 19
  33. 2024Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMscited by 17
  34. 2024Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Studycited by 17
  35. 2024Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarizationcited by 16
  36. 2023ChatCoder: Chat-based Refine Requirement Improves LLMs' Code Generationcited by 13
  37. 2023Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Modelscited by 13
  38. 2022Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systemscited by 12

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 8.

15 papers.

Sort:

By area:

  1. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  2. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 22
  3. 2026Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysiscited by 15
  4. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 10
  5. 2025Towards an Understanding of Context Utilization in Code Intelligencecited by 10
  6. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  7. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  8. 2025Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generationcited by 4
  9. 2025Knowledge Graph Based Repository-Level Code Generationcited by 4
  10. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  11. 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
  12. 2026PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agentscited by 2
3 more
  1. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  2. 2026Interactive LLM-assisted Curriculum Learning for Multi-Task Evolutionary Policy Searchcited by 0
  3. 2026AI Writes Code, Humans Pay the Debt. An Empirical Study on the Sustainability and Evolution of Agent-Generated Codecited by 0