# Improving ChatGPT Prompt for Code Generation

arXiv: https://arxiv.org/abs/2305.08360 · HTML: https://mlsanity.com/paper/2305.08360/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2023.

## Built on by (most cited first, 28 of 28)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Large Language Models are Complex Table Parsers](https://mlsanity.com/paper/2312.11521/) (2023)
- [LLM-Assisted Code Cleaning For Training Accurate Code Generators](https://mlsanity.com/paper/2311.14904/) (2023)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation](https://mlsanity.com/paper/2305.12050/) (2023)
- [UniCoder: Scaling Code Large Language Model via Universal Code](https://mlsanity.com/paper/2406.16441/) (2024)
- [Supersonic: Learning to Generate Source Code Optimizations in C/C++](https://mlsanity.com/paper/2309.14846/) (2023)
- [OMPGPT: A Generative Pre-trained Transformer Model for OpenMP](https://mlsanity.com/paper/2401.16445/) (2024)
- [Benchmarking Causal Study to Interpret Large Language Models for Source Code](https://mlsanity.com/paper/2308.12415/) (2023)
- [A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentation](https://mlsanity.com/paper/2407.02742/) (2024)
- [FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair](https://mlsanity.com/paper/2307.00012/) (2023)
- [Coding by Design: GPT-4 empowers Agile Model Driven Development](https://mlsanity.com/paper/2310.04304/) (2023)
- [Exploring the Limits of ChatGPT in Software Security Applications](https://mlsanity.com/paper/2312.05275/) (2023)
- [SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation](https://mlsanity.com/paper/2404.01923/) (2024)
- [Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering](https://mlsanity.com/paper/2411.15501/) (2024)
- [A Survey on Neural Question Generation: Methods, Applications, and Prospects](https://mlsanity.com/paper/2402.18267/) (2024)
- [Research and application of artificial intelligence based webshell detection model: A literature review](https://mlsanity.com/paper/2405.00066/) (2024)
- [SmartLLMSentry: A Comprehensive LLM Based Smart Contract Vulnerability Detection Framework](https://mlsanity.com/paper/2411.19234/) (2024)
- [Assessing the Promise and Pitfalls of ChatGPT for Automated Code Generation](https://mlsanity.com/paper/2311.02640/) (2023)
- [Large Language Models are Few-shot Generators: Proposing Hybrid Prompt Algorithm To Generate Webshell Escape Samples](https://mlsanity.com/paper/2402.07408/) (2024)
- [Exploring Large Language Models in Resolving Environment-Related Crash Bugs: Localizing and Repairing](https://mlsanity.com/paper/2312.10448/) (2023)
- [Plan with Code: Comparing approaches for robust NL to DSL generation](https://mlsanity.com/paper/2408.08335/) (2024)
- [LLM as a code generator in Agile Model Driven Development](https://mlsanity.com/paper/2410.18489/) (2024)
- [SnipGen: A Mining Repository Framework for Evaluating LLMs for Code](https://mlsanity.com/paper/2502.07046/) (2025)
- [Task-oriented Prompt Enhancement via Script Generation](https://mlsanity.com/paper/2409.16418/) (2024)
- [Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering](https://mlsanity.com/paper/2601.21070/) (2026)
- [Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality](https://mlsanity.com/paper/2607.14816/) (2026)
- [Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation](https://mlsanity.com/paper/2608.11513/) (2026)

## Abstract (arXiv)

Automated code generation can be a powerful technique for software development, significantly reducing developers' efforts and time required to create new code by generating it automatically based on requirements. Recently, OpenAI's language model ChatGPT has emerged as a powerful tool for generating human-like responses to a wide range of textual inputs (i.e., prompts), including those related to code generation. However, the effectiveness of ChatGPT for code generation is not well understood, and the generation performance could be heavily influenced by the choice of prompt. To answer these questions, we conducted experiments using the CodeXGlue dataset to evaluate ChatGPT's capabilities for two code generation tasks, including text-to-code and code-to-code generation. We designed prompts by leveraging the chain-of-thought strategy with multi-step optimizations. Our results showed that by carefully designing prompts to guide ChatGPT, the generation performance can be improved substantially. We also analyzed the factors that influenced the prompt design and provided insights that could guide future research.