# Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?

arXiv: https://arxiv.org/abs/2411.02093 · HTML: https://mlsanity.com/paper/2411.02093/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- First posted on arXiv 4 Nov 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Built on by (most cited first, 19 of 19)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [AgentRxiv: Towards Collaborative Autonomous Research](https://mlsanity.com/paper/2503.18102/) (2025)
- [Natural-Language Agent Harnesses](https://mlsanity.com/paper/2603.25723/) (2026)
- [The Impact of Prompt Programming on Function-Level Code Generation](https://mlsanity.com/paper/2412.20545/) (2024)
- [Multi-Agent Debate Strategies to Enhance Requirements Engineering with Large Language Models](https://mlsanity.com/paper/2507.05981/) (2025)
- [Revisiting Prompt Optimization with Large Reasoning Models-A Case Study on Event Extraction](https://mlsanity.com/paper/2504.07357/) (2025)
- [An Empirical Study of Harness Design for Coding Agents](https://mlsanity.com/paper/2609.20804/) (2026)
- [Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs?](https://mlsanity.com/paper/2503.05507/) (2025)
- [The Imitation Game: Turing Machine Imitator is Length Generalizable Reasoner](https://mlsanity.com/paper/2507.13332/) (2025)
- [Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skill](https://mlsanity.com/paper/2606.06454/) (2026)
- [LLM Agents for Generating Microservice-based Applications: how complex is your specification?](https://mlsanity.com/paper/2508.20119/) (2025)
- [Enhancing Factual Accuracy and Citation Generation in LLMs via Multi-Stage Self-Verification](https://mlsanity.com/paper/2509.05741/) (2025)
- [Exploringand Unleashing the Power of Large Language Models in CI/CD Configuration Translation](https://mlsanity.com/paper/2511.01316/) (2025)
- [A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis](https://mlsanity.com/paper/2512.19481/) (2025)
- [Human-in-the-Loop Control of Objective Drift in LLM-Assisted Computer Science Education](https://mlsanity.com/paper/2604.00281/) (2026)
- [Prompt-Driven Code Summarization: A Systematic Literature Review](https://mlsanity.com/paper/2604.15385/) (2026)
- [Less Back-and-Forth: A Comparative Study of Structured Prompting](https://mlsanity.com/paper/2605.20149/) (2026)
- [LLMs as Post-hoc Auditors of Physiological Plausibility in Symbolic Regression: A Clinician-Evaluated Case Study](https://mlsanity.com/paper/2609.11431/) (2026)
- [Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents](https://mlsanity.com/paper/2609.22259/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have significantly advanced software engineering (SE) tasks, with prompt engineering techniques enhancing their performance in code-related areas. However, the rapid development of foundational LLMs such as the non-reasoning model GPT-4o and the reasoning model o1 raises questions about the continued effectiveness of these prompt engineering techniques. This paper presents an extensive empirical study that reevaluates various prompt engineering techniques within the context of these advanced LLMs. Focusing on three representative SE tasks, i.e., code generation, code translation, and code summarization, we assess whether prompt engineering techniques still yield improvements with advanced models, the actual effectiveness of reasoning models compared to non-reasoning models, and whether the benefits of using these advanced models justify their increased costs. Our findings reveal that prompt engineering techniques developed for earlier LLMs may provide diminished benefits or even hinder performance when applied to advanced models. In reasoning LLMs, the ability of sophisticated built-in reasoning reduces the impact of complex prompts, sometimes making simple zero-shot prompting more effective. Furthermore, while reasoning models outperform non-reasoning models in tasks requiring complex reasoning, they offer minimal advantages in tasks that do not need reasoning and may incur unnecessary costs. Based on our study, we provide practical guidance for practitioners on selecting appropriate prompt engineering techniques and foundational LLMs, considering factors such as task requirements, operational costs, and environmental impact. Our work contributes to a deeper understanding of effectively harnessing advanced LLMs in SE tasks, informing future research and application development.