mlsanityData to 14 Oct 2026

ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks

arXiv 2305.16837 ↗First posted 26 May 2023cs.SE cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

ChatGPT (Chat Generative Pre-trained Transformer) is a chatbot launched by OpenAI on November 30, 2022. OpenAI's GPT-3 family of large language models serve as the foundation for ChatGPT. ChatGPT is fine-tuned with both supervised and reinforcement learning techniques and has received widespread attention for its articulate responses across diverse domains of knowledge. In this study, we explore how ChatGPT can be used to help with common software engineering tasks. Many of the ubiquitous tasks covering the breadth of software engineering such as ambiguity resolution in software requirements, method name suggestion, test case prioritization, code review, log summarization can potentially be performed using ChatGPT. In this study, we explore fifteen common software engineering tasks using ChatGPT. We juxtapose and analyze ChatGPT's answers with the respective state of the art outputs (where available) and/or human expert ground truth. Our experiments suggest that for many tasks, ChatGPT does perform credibly and the response from it is detailed and often better than the human expert output or the state of the art output. However, for a few other tasks, ChatGPT in its present form provides incorrect answers and hence is not suited for such tasks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 15 of its 41 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

15 of 41 references.

References · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  4. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  5. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  6. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  7. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 179
  8. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 75
  9. 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
  10. 2020Reinforcement Learning for Test Case Prioritizationcited by 13
  11. 2021Static Inference Meets Deep Learning: A Hybrid Type Inference Approach for Pythoncited by 12
  12. 2016Why We Refactor? Confessions of GitHub Contributorscited by 10
3 more
  1. 2022Duplicate Bug Report Detection: How Far Are We?cited by 8
  2. 2020Summarizing Unstructured Logs in Online Servicescited by 5
  3. 2021Program Merge Conflict Resolution via Neural Transformerscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 7.

12 papers.

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  3. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  4. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 86
  5. 2024Automatic Programming: Large Language Models and Beyondcited by 36
  6. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  7. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Modelscited by 12
  8. 2023An Assessment of ChatGPT on Log Datacited by 12
  9. 2024Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generationcited by 11
  10. 2024Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineeringcited by 6
  11. 2024Icing on the Cake: Automatic Code Summarization at Ericssoncited by 2
  12. 2024Binary Hypothesis Testing for Softmax Models and Leverage Score Modelscited by 1