# Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code

arXiv: https://arxiv.org/abs/2310.10508 · HTML: https://mlsanity.com/paper/2310.10508/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2023.

## Built on by (most cited first, 9 of 9)

- [Agent READMEs: An Empirical Study of Context Files for Agentic Coding](https://mlsanity.com/paper/2511.12884/) (2025)
- [Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation](https://mlsanity.com/paper/2506.02943/) (2025)
- [MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety](https://mlsanity.com/paper/2605.01687/) (2026)
- [Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering](https://mlsanity.com/paper/2412.03815/) (2024)
- [Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: A Systematic Evaluation](https://mlsanity.com/paper/2602.11904/) (2026)
- [Can AI Agents Generate Microservices? How Far are We?](https://mlsanity.com/paper/2603.09004/) (2026)
- [From Instruction to Output: The Role of Prompting in Modern NLG](https://mlsanity.com/paper/2602.11179/) (2026)
- [OMEGA: Optimizing Machine Learning by Evaluating Generated Algorithms](https://mlsanity.com/paper/2604.26211/) (2026)
- [Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns](https://mlsanity.com/paper/2607.00048/) (2026)

## Abstract (arXiv)

The rapid advancements in large language models (LLMs) have greatly expanded the potential for automated code-related tasks. Two primary methodologies are used in this domain: prompt engineering and fine-tuning. Prompt engineering involves applying different strategies to query LLMs, like ChatGPT, while fine-tuning further adapts pre-trained models, such as CodeBERT, by training them on task-specific data. Despite the growth in the area, there remains a lack of comprehensive comparative analysis between the approaches for code models. In this paper, we evaluate GPT-4 using three prompt engineering strategies -- basic prompting, in-context learning, and task-specific prompting -- and compare it against 17 fine-tuned models across three code-related tasks: code summarization, generation, and translation. Our results indicate that GPT-4 with prompt engineering does not consistently outperform fine-tuned models. For instance, in code generation, GPT-4 is outperformed by fine-tuned models by 28.3% points on the MBPP dataset. It also shows mixed results for code translation tasks. Additionally, a user study was conducted involving 27 graduate students and 10 industry practitioners. The study revealed that GPT-4 with conversational prompts, incorporating human feedback during interaction, significantly improved performance compared to automated prompting. Participants often provided explicit instructions or added context during these interactions. These findings suggest that GPT-4 with conversational prompting holds significant promise for automated code-related tasks, whereas fully automated prompt engineering without human involvement still requires further investigation.