mlsanityData to 7 Oct 2026

What Makes a Good Natural Language Prompt?

arXiv 2506.06950 ↗First posted 7 Jun 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) have progressed towards more human-like and human--AI communications have become prevalent, prompting has emerged as a decisive component. However, there is limited conceptual consensus on what exactly quantifies natural language prompts. We attempt to address this question by conducting a meta-analysis surveying more than 150 prompting-related papers from leading NLP and AI conferences from 2022 to 2025 and blogs. We propose a property- and human-centric framework for evaluating prompt quality, encompassing 21 properties categorized into six dimensions. We then examine how existing studies assess their impact on LLMs, revealing their imbalanced support across models and tasks, and substantial research gaps. Further, we analyze correlations among properties in high-quality natural language prompts, deriving prompting recommendations. We then empirically explore multi-property prompt enhancements in reasoning tasks, observing that single-property enhancements often have the greatest impact. Finally, we discover that instruction-tuning on property-enhanced prompts can result in better reasoning models. Our findings establish a foundation for property-centric prompt evaluation and optimization, bridging the gaps between human--AI communication and opening new prompting research directions.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 132 of its 203 references that are in the map. First posted on arXiv 7 Jun 2025.

Builds on

132 of 203 references.

References · 132

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  13. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  14. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  15. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  16. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  17. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  18. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  19. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  20. 2023Augmented Language Models: a Surveycited by 398
  21. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  22. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  23. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  24. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  25. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  26. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  27. 2022Prompting GPT-3 To Be Reliablecited by 258
  28. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
  29. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  30. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  31. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  32. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  33. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  34. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  35. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendationscited by 130
  36. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  37. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  38. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115

50 most cited of 132 All 132 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
  2. 2026ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QA
  3. 2025DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Models
  4. 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
  5. 2025Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages
  6. 2023Prompt Your Mind: Refine Personalized Text Prompts within Your Mind
  7. 2024Efficient Prompting Methods for Large Language Models: A Survey
  8. 2026When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness