# Prompting Science Report 1: Prompt Engineering is Complicated and Contingent

arXiv: https://arxiv.org/abs/2503.04818 · HTML: https://mlsanity.com/paper/2503.04818/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 5 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Builds on (most cited first, 2 of 2)

- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance](https://mlsanity.com/paper/2402.14531/) (2024)

## Built on by (most cited first, 8 of 8)

- [Beyond "Hallucinations": A Framework for Stable Human-AI Reasoning](https://mlsanity.com/paper/2510.14665/) (2025)
- [Prompting as Scientific Inquiry](https://mlsanity.com/paper/2507.00163/) (2025)
- [Prompting Science Report 3: I'll pay you or I'll kill you -- but will you care?](https://mlsanity.com/paper/2508.00614/) (2025)
- [Instance-level Randomization: Toward More Stable LLM Evaluations](https://mlsanity.com/paper/2509.12678/) (2025)
- [Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy](https://mlsanity.com/paper/2512.05858/) (2025)
- [Ghost Framing Theory: Exploring the role of generative AI in new venture rhetorical legitimation](https://mlsanity.com/paper/2603.11384/) (2026)
- [Human-in-the-Loop Control of Objective Drift in LLM-Assisted Computer Science Education](https://mlsanity.com/paper/2604.00281/) (2026)
- [Understanding Tone-Dependent Inference Cost in Large Language Models](https://mlsanity.com/paper/2607.23915/) (2026)

## Abstract (arXiv)

This is the first of a series of short reports that seek to help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. In this report, we demonstrate two things: - There is no single standard for measuring whether a Large Language Model (LLM) passes a benchmark, and that choosing a standard has a big impact on how well the LLM does on that benchmark. The standard you choose will depend on your goals for using an LLM in a particular case. - It is hard to know in advance whether a particular prompting approach will help or harm the LLM's ability to answer any particular question. Specifically, we find that sometimes being polite to the LLM helps performance, and sometimes it lowers performance. We also find that constraining the AI's answers helps performance in some cases, though it may lower performance in other cases. Taken together, this suggests that benchmarking AI performance is not one-size-fits-all, and also that particular prompting formulas or approaches, like being polite to the AI, are not universally valuable.