# What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use

arXiv: https://arxiv.org/abs/2409.08775 · HTML: https://mlsanity.com/paper/2409.08775/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 87 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models](https://mlsanity.com/paper/2310.20410/) (2023)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents](https://mlsanity.com/paper/2402.09205/) (2024)
- [Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles](https://mlsanity.com/paper/2407.00870/) (2024)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [Interactive Code Generation via Test-Driven User-Intent Formalization](https://mlsanity.com/paper/2208.05950/) (2022)
- ["What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models](https://mlsanity.com/paper/2304.06597/) (2023)
- [Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4](https://mlsanity.com/paper/2312.16171/) (2023)
- [Investigating Explainability of Generative AI for Code through Scenario-based Design](https://mlsanity.com/paper/2202.04903/) (2022)
- [ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles](https://mlsanity.com/paper/2310.15428/) (2023)

## Built on by (most cited first, 24 of 24)

- [Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents](https://mlsanity.com/paper/2601.01885/) (2026)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts](https://mlsanity.com/paper/2505.13360/) (2025)
- [Taxonomy of User Needs and Actions](https://mlsanity.com/paper/2510.06124/) (2025)
- [Substance Beats Style: Why Beginning Students Fail to Code with LLMs](https://mlsanity.com/paper/2410.19792/) (2024)
- [Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scale](https://mlsanity.com/paper/2504.09283/) (2025)
- [A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges](https://mlsanity.com/paper/2602.05883/) (2026)
- [LLM-HyPZ: Hardware Vulnerability Discovery using an LLM-Assisted Hybrid Platform for Zero-Shot Knowledge Extraction and Refinement](https://mlsanity.com/paper/2509.00647/) (2025)
- [Does GenAI Rewrite How We Write? An Empirical Study on Two-Million Preprints](https://mlsanity.com/paper/2510.17882/) (2025)
- [Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations](https://mlsanity.com/paper/2509.25492/) (2025)
- [Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behavior](https://mlsanity.com/paper/2510.12728/) (2025)
- [U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibility](https://mlsanity.com/paper/2511.03517/) (2025)
- [Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Sampling](https://mlsanity.com/paper/2410.18105/) (2024)
- [VISION: Prompting Ocean Vertical Velocity Reconstruction from Incomplete Observations](https://mlsanity.com/paper/2509.21477/) (2025)
- [Optimizing delivery for quick commerce factoring qualitative assessment of generated routes](https://mlsanity.com/paper/2510.08671/) (2025)
- [Agile Deliberation: Concept Deliberation for Subjective Visual Classification](https://mlsanity.com/paper/2512.10821/) (2025)
- [PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactions](https://mlsanity.com/paper/2601.15575/) (2026)
- [Large Language Model and Formal Concept Analysis: a comparative study for Topic Modeling](https://mlsanity.com/paper/2602.01933/) (2026)
- [Are Open-Weight LLMs Ready for Social Media Moderation? A Comparative Study on Bluesky](https://mlsanity.com/paper/2602.05189/) (2026)
- ["How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations](https://mlsanity.com/paper/2602.18372/) (2026)
- [Alignment has a Fantasia Problem](https://mlsanity.com/paper/2604.21827/) (2026)
- [PromptDecipher: Supporting AI Tutor Authoring Through Editable Simulated Interactions](https://mlsanity.com/paper/2605.16605/) (2026)
- [Less Back-and-Forth: A Comparative Study of Structured Prompting](https://mlsanity.com/paper/2605.20149/) (2026)
- [VibeJam: A User Study Platform for Web Development with Agents](https://mlsanity.com/paper/2608.29889/) (2026)

## Abstract (arXiv)

Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications.