# Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering arXiv: https://arxiv.org/abs/2411.15501 ยท HTML: https://mlsanity.com/paper/2411.15501/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 57 references that are in the map. - First posted on arXiv 23 Nov 2024. ## Builds on (most cited first, 30 of 34) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019) - [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023) - [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022) - [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023) - [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023) - [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021) - [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020) - [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021) - [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023) - [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023) - [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023) - [Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning](https://mlsanity.com/paper/2304.11384/) (2023) - [TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks](https://mlsanity.com/paper/2305.11430/) (2023) - [What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?](https://mlsanity.com/paper/2304.07575/) (2023) - [AgentFL: Scaling LLM-based Fault Localization to Project-Level Context](https://mlsanity.com/paper/2403.16362/) (2024) - [Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study](https://mlsanity.com/paper/2309.08221/) (2023) - [Improving ChatGPT Prompt for Code Generation](https://mlsanity.com/paper/2305.08360/) (2023) - [A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair](https://mlsanity.com/paper/2304.08191/) (2023) - [CCT5: A Code-Change-Oriented Pre-Trained Model](https://mlsanity.com/paper/2305.10785/) (2023) - [Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability](https://mlsanity.com/paper/2308.00229/) (2023) - [ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks](https://mlsanity.com/paper/2305.16837/) (2023) - [Analyzing and Supporting Adaptation of Online Code Examples](https://mlsanity.com/paper/1905.12111/) (2019) - [From Query to Usable Code: An Analysis of Stack Overflow Code Snippets](https://mlsanity.com/paper/1605.04464/) (2016) - [Usage and Attribution of Stack Overflow Code Snippets in GitHub Projects](https://mlsanity.com/paper/1802.02938/) (2018) - [APIzation: Generating Reusable APIs from StackOverflow Code Snippets](https://mlsanity.com/paper/2109.01138/) (2021) ## Built on by (most cited first, 6 of 6) - [Agent READMEs: An Empirical Study of Context Files for Agentic Coding](https://mlsanity.com/paper/2511.12884/) (2025) - [Can LLMs Replace Humans During Code Chunking?](https://mlsanity.com/paper/2506.19897/) (2025) - [Large Language Model-Based Intelligent Antenna Design System](https://mlsanity.com/paper/2504.18271/) (2025) - [Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs](https://mlsanity.com/paper/2507.20977/) (2025) - [AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation](https://mlsanity.com/paper/2601.04540/) (2026) - [Wired for Reuse: Automating Context-Aware Code Adaptation in IDEs via LLM-Based Agent](https://mlsanity.com/paper/2507.01315/) (2025) ## Abstract (arXiv) Code snippet adaptation is a fundamental activity in the software development process. Unlike code generation, code snippet adaptation is not a "free creation", which requires developers to tailor a given code snippet in order to fit specific requirements and the code context. Recently, large language models (LLMs) have confirmed their effectiveness in the code generation task with promising results. However, their performance on adaptation, a reuse-oriented and context-dependent code change prediction task, is still unclear. To bridge this gap, we conduct an empirical study to investigate the performance and issues of LLMs on the adaptation task. We first evaluate the adaptation performances of three popular LLMs and compare them to the code generation task. Our result indicates that their adaptation ability is weaker than generation, with a nearly 15% decrease on pass@1 and more context-related errors. By manually inspecting 200 cases, we further investigate the causes of LLMs' sub-optimal performance, which can be classified into three categories, i.e., Unclear Requirement, Requirement Misalignment and Context Misapplication. Based on the above empirical research, we propose an interactive prompting approach to eliciting LLMs' adaptation ability. Experimental result reveals that our approach greatly improve LLMs' adaptation performance. The best-performing Human-LLM interaction successfully solves 159 out of the 202 identified defects and improves the pass@1 and pass@5 by over 40% compared to the initial instruction-based prompt. Considering human efforts, we suggest multi-agent interaction as a trade-off, which can achieve comparable performance with excellent generalization ability. We deem that our approach could provide methodological assistance for autonomous code snippet reuse and adaptation with LLMs.