mlsanityData to 7 Oct 2026

What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use

arXiv 2409.08775 ↗First posted 13 Sep 2024cs.HC cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 43 of its 87 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

43 of 87 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  11. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  12. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
31 more
  1. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  2. 2023Using an LLM to Help With Code Understandingcited by 222
  3. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  4. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  5. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 134
  6. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
  7. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 93
  8. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  9. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  10. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
  11. 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 71
  12. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 62
  13. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 61
  14. 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
  15. 2023"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Modelscited by 48
  16. 2023Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4cited by 48
  17. 2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
  18. 2023ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principlescited by 33
  19. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  20. 2023Prompt Problems: A New Programming Exercise for the Generative AI Eracited by 31
  21. 2024How Beginning Programmers and Code LLMs (Mis)read Each Othercited by 31
  22. 2023How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debuggingcited by 29
  23. 2016Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Explorationcited by 26
  24. 2024Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chatcited by 23
  25. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
  26. 2023CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programmingcited by 16
  27. 2024A First Look at GPT Apps: Landscape and Vulnerabilitycited by 15
  28. 2023Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMscited by 9
  29. 2024LLMCRIT: Teaching Large Language Models to Use Criteriacited by 9
  30. 2023Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunitiescited by 8
  31. 2018NLP-assisted software testing: A systematic mapping of the literaturecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 10.

24 papers.

Sort:

By area:

  1. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 72
  2. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  3. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
  4. 2025Taxonomy of User Needs and Actionscited by 13
  5. 2024Substance Beats Style: Why Beginning Students Fail to Code with LLMscited by 5
  6. 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3
  7. 2025LLM-HyPZ: Hardware Vulnerability Discovery using an LLM-Assisted Hybrid Platform for Zero-Shot Knowledge Extraction and Refinementcited by 2
  8. 2025Does GenAI Rewrite How We Write? An Empirical Study on Two-Million Preprintscited by 2
  9. 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 2
  10. 2025Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocationscited by 1
  11. 2025Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behaviorcited by 1
  12. 2025U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibilitycited by 1
12 more
  1. 2024Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Samplingcited by 0
  2. 2025VISION: Prompting Ocean Vertical Velocity Reconstruction from Incomplete Observationscited by 0
  3. 2025Optimizing delivery for quick commerce factoring qualitative assessment of generated routescited by 0
  4. 2025Agile Deliberation: Concept Deliberation for Subjective Visual Classificationcited by 0
  5. 2026PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactionscited by 0
  6. 2026Large Language Model and Formal Concept Analysis: a comparative study for Topic Modelingcited by 0
  7. 2026Are Open-Weight LLMs Ready for Social Media Moderation? A Comparative Study on Blueskycited by 0
  8. 2026"How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversationscited by 0
  9. 2026Alignment has a Fantasia Problemcited by 0
  10. 2026PromptDecipher: Supporting AI Tutor Authoring Through Editable Simulated Interactionscited by 0
  11. 2026Less Back-and-Forth: A Comparative Study of Structured Promptingcited by 0
  12. 2026VibeJam: A User Study Platform for Web Development with Agentscited by 0