mlsanityData to 7 Oct 2026

Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study

arXiv 2309.08221 ↗First posted 15 Sep 2023cs.SEcited by 44

Not on the map. Why?

Abstract (arXiv)

Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given code reviews. To conduct the study, we select the existing benchmark CodeReview and construct a new code review dataset with high quality. We use CodeReviewer, a state-of-the-art code review tool, as a baseline for comparison with ChatGPT. Our results show that ChatGPT outperforms CodeReviewer in code refinement tasks. Specifically, our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset. We further identify the root causes for ChatGPT's underperformance and propose several strategies to mitigate these challenges. Our study provides insights into the potential of ChatGPT in automating the code review process, and highlights the potential research directions.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Sep 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 21; 2026: 8.

44 papers.

Sort:

By area:

  1. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  2. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHubcited by 53
  3. 2024CYCLE: Learning to Self-Refine the Code Generationcited by 39
  4. 2024Automatic Programming: Large Language Models and Beyondcited by 35
  5. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  6. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 27
  7. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 24
  8. 2024A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?cited by 9
  9. 2024Model Editing for LLMs4Code: How Far are We?cited by 8
  10. 2025CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scalecited by 8
  11. 2025LLMs' Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopterscited by 8
  12. 2025LAURA: Enhancing Code Review Generation with Context-Enriched Retrieval-Augmented LLMcited by 8
32 more
  1. 2024Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generationcited by 7
  2. 2024An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factorscited by 6
  3. 2024Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineeringcited by 6
  4. 2025Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Taskscited by 6
  5. 2024Looking into Black Box Code Language Modelscited by 5
  6. 2024Leveraging Reviewer Experience in Code Review Comment Generationcited by 4
  7. 2025Issue-Oriented Agent-Based Framework for Automated Review Comment Generationcited by 4
  8. 2024Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trendscited by 3
  9. 2024CodeCloak: A Method for Evaluating and Mitigating Code Leakage by LLM Code Assistantscited by 3
  10. 2024Log Parsing using LLMs with Self-Generated In-Context Learning and Self-Correctioncited by 3
  11. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  12. 2025CodeReviewQA: The Code Review Comprehension Assessment for Large Language Modelscited by 3
  13. 2025EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMscited by 3
  14. 2026Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agentscited by 3
  15. 2025SDLog: A Deep Learning Framework for Detecting Sensitive Information in Software Logscited by 2
  16. 2025SAEL: Leveraging Large Language Models with Adaptive Mixture-of-Experts for Smart Contract Vulnerability Detectioncited by 2
  17. 2025Code Review Without Borders: Evaluating Synthetic vs. Real Data for Review Recommendationcited by 1
  18. 2026Understanding the Limits of Automated Evaluation for Code Review Bots in Practicecited by 1
  19. 2025Towards Refining Developer Questions using LLM-Based Named Entity Recognition for Developer Chatroom Conversationscited by 0
  20. 2025Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMscited by 0
  21. 2025Improving Automated Secure Code Reviews: A Synthetic Dataset for Code Vulnerability Flawscited by 0
  22. 2025Fine-Tuning Multilingual Language Models for Code Review: An Empirical Study on Industrial C# Projectscited by 0
  23. 2025Fine-Tuning Code Language Models to Detect Cross-Language Bugscited by 0
  24. 2025TGPR: Tree-Guided Policy Refinement for Robust Self-Debugging of LLMscited by 0
  25. 2025A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Casecited by 0
  26. 2025Agentic LLMs for REST API Test Amplification: A Comparative Study Across Cloud Applicationscited by 0
  27. 2026Improving MPI Error Detection and Repair with Large Language Models and Bug Referencescited by 0
  28. 2026Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revisioncited by 0
  29. 2026Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automationcited by 0
  30. 2026Software Self-Extension with SelfEvolve: an Agentic Architecture for Runtime Code Generationcited by 0
  31. 2026CIT-CAD: Constraint Intent Tree-based CAD Code Generation and Verificationcited by 0
  32. 2026CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?cited by 0