What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Not on the map. Why?
Abstract (arXiv)
Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications.
Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 43 of its 87 references that are in the map. First posted on arXiv 13 Sep 2024.
Builds on
43 of 87 references.
References · 43
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2019How Can We Know What Language Models Know?cited by 953
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
- 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
- 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
31 more
- 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
- 2023Using an LLM to Help With Code Understandingcited by 222
- 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
- 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
- 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 134
- 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
- 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 93
- 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
- 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
- 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
- 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 71
- 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 62
- 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 61
- 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
- 2023"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Modelscited by 48
- 2023Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4cited by 48
- 2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
- 2023ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principlescited by 33
- 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
- 2023Prompt Problems: A New Programming Exercise for the Generative AI Eracited by 31
- 2024How Beginning Programmers and Code LLMs (Mis)read Each Othercited by 31
- 2023How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debuggingcited by 29
- 2016Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Explorationcited by 26
- 2024Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chatcited by 23
- 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
- 2023CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programmingcited by 16
- 2024A First Look at GPT Apps: Landscape and Vulnerabilitycited by 15
- 2023Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMscited by 9
- 2024LLMCRIT: Teaching Large Language Models to Use Criteriacited by 9
- 2023Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunitiescited by 8
- 2018NLP-assisted software testing: A systematic mapping of the literaturecited by 4
Built on by
Citing papers per year: 2024: 2; 2025: 12; 2026: 10.
24 papers.
Sort:
By area:
- 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 72
- 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
- 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
- 2025Taxonomy of User Needs and Actionscited by 13
- 2024Substance Beats Style: Why Beginning Students Fail to Code with LLMscited by 5
- 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3
- 2025LLM-HyPZ: Hardware Vulnerability Discovery using an LLM-Assisted Hybrid Platform for Zero-Shot Knowledge Extraction and Refinementcited by 2
- 2025Does GenAI Rewrite How We Write? An Empirical Study on Two-Million Preprintscited by 2
- 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 2
- 2025Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocationscited by 1
- 2025Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behaviorcited by 1
- 2025U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibilitycited by 1
12 more
- 2024Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Samplingcited by 0
- 2025VISION: Prompting Ocean Vertical Velocity Reconstruction from Incomplete Observationscited by 0
- 2025Optimizing delivery for quick commerce factoring qualitative assessment of generated routescited by 0
- 2025Agile Deliberation: Concept Deliberation for Subjective Visual Classificationcited by 0
- 2026PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactionscited by 0
- 2026Large Language Model and Formal Concept Analysis: a comparative study for Topic Modelingcited by 0
- 2026Are Open-Weight LLMs Ready for Social Media Moderation? A Comparative Study on Blueskycited by 0
- 2026"How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversationscited by 0
- 2026Alignment has a Fantasia Problemcited by 0
- 2026PromptDecipher: Supporting AI Tutor Authoring Through Editable Simulated Interactionscited by 0
- 2026Less Back-and-Forth: A Comparative Study of Structured Promptingcited by 0
- 2026VibeJam: A User Study Platform for Web Development with Agentscited by 0
- 2026VibeJam: A User Study Platform for Web Development with Agents30 Aug 2026
- 2026Less Back-and-Forth: A Comparative Study of Structured Prompting19 May 2026
- 2026PromptDecipher: Supporting AI Tutor Authoring Through Editable Simulated Interactions15 May 2026
- 2026Alignment has a Fantasia Problem23 Apr 2026
- 2026"How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations20 Feb 2026
- 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges5 Feb 2026
- 2026Are Open-Weight LLMs Ready for Social Media Moderation? A Comparative Study on Bluesky5 Feb 2026
- 2026Large Language Model and Formal Concept Analysis: a comparative study for Topic Modeling2 Feb 2026
- 2026PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactions22 Jan 2026
- 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents5 Jan 2026
- 2025Agile Deliberation: Concept Deliberation for Subjective Visual Classification11 Dec 2025
- 2025U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibility5 Nov 2025
12 more
- 2025Does GenAI Rewrite How We Write? An Empirical Study on Two-Million Preprints18 Oct 2025
- 2025Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behavior14 Oct 2025
- 2025Optimizing delivery for quick commerce factoring qualitative assessment of generated routes9 Oct 2025
- 2025Taxonomy of User Needs and Actions7 Oct 2025
- 2025Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations29 Sep 2025
- 2025VISION: Prompting Ocean Vertical Velocity Reconstruction from Incomplete Observations25 Sep 2025
- 2025LLM-HyPZ: Hardware Vulnerability Discovery using an LLM-Assisted Hybrid Platform for Zero-Shot Knowledge Extraction and Refinement31 Aug 2025
- 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI26 May 2025
- 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts19 May 2025
- 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scale12 Apr 2025
- 2024Substance Beats Style: Why Beginning Students Fail to Code with LLMs15 Oct 2024
- 2024Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Sampling8 Oct 2024
- 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 72
- 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
- 2025Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocationscited by 1
- 2025U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibilitycited by 1
- 2026VibeJam: A User Study Platform for Web Development with Agentscited by 0
- 2026VibeJam: A User Study Platform for Web Development with Agents30 Aug 2026
- 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents5 Jan 2026
- 2025U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibility5 Nov 2025
- 2025Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations29 Sep 2025
- 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI26 May 2025
- 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
- 2025Taxonomy of User Needs and Actionscited by 13
- 2024Substance Beats Style: Why Beginning Students Fail to Code with LLMscited by 5
- 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3
- 2025LLM-HyPZ: Hardware Vulnerability Discovery using an LLM-Assisted Hybrid Platform for Zero-Shot Knowledge Extraction and Refinementcited by 2
- 2025Does GenAI Rewrite How We Write? An Empirical Study on Two-Million Preprintscited by 2
- 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 2
- 2025Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behaviorcited by 1
- 2025VISION: Prompting Ocean Vertical Velocity Reconstruction from Incomplete Observationscited by 0
- 2025Optimizing delivery for quick commerce factoring qualitative assessment of generated routescited by 0
- 2025Agile Deliberation: Concept Deliberation for Subjective Visual Classificationcited by 0
- 2026PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactionscited by 0
12 most cited of 17 All 17 (JSON)
- 2026Less Back-and-Forth: A Comparative Study of Structured Prompting19 May 2026
- 2026PromptDecipher: Supporting AI Tutor Authoring Through Editable Simulated Interactions15 May 2026
- 2026Alignment has a Fantasia Problem23 Apr 2026
- 2026"How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations20 Feb 2026
- 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges5 Feb 2026
- 2026Large Language Model and Formal Concept Analysis: a comparative study for Topic Modeling2 Feb 2026
- 2026PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactions22 Jan 2026
- 2025Agile Deliberation: Concept Deliberation for Subjective Visual Classification11 Dec 2025
- 2025Does GenAI Rewrite How We Write? An Empirical Study on Two-Million Preprints18 Oct 2025
- 2025Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behavior14 Oct 2025
- 2025Optimizing delivery for quick commerce factoring qualitative assessment of generated routes9 Oct 2025
- 2025Taxonomy of User Needs and Actions7 Oct 2025
12 newest of 17 All 17 (JSON)