# Benchmarking Prompt Sensitivity in Large Language Models

arXiv: https://arxiv.org/abs/2502.06065 · HTML: https://mlsanity.com/paper/2502.06065/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [Information Retrieval: Recent Advances and Beyond](https://mlsanity.com/paper/2301.08801/) (2023)
- [What's the Magic Word? A Control Theory of LLM Prompting](https://mlsanity.com/paper/2310.04444/) (2023)
- [Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters](https://mlsanity.com/paper/2312.17476/) (2023)
- [Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Science](https://mlsanity.com/paper/2305.14310/) (2023)
- [Query Performance Prediction using Relevance Judgments Generated by Large Language Models](https://mlsanity.com/paper/2404.01012/) (2024)
- [Query Performance Prediction: From Ad-hoc to Conversational Search](https://mlsanity.com/paper/2305.10923/) (2023)
- [Assessing and Verifying Task Utility in LLM-Powered Applications](https://mlsanity.com/paper/2405.02178/) (2024)
- [Semantic Consistency for Assuring Reliability of Large Language Models](https://mlsanity.com/paper/2308.09138/) (2023)
- [Testing LLMs on Code Generation with Varying Levels of Prompt Specificity](https://mlsanity.com/paper/2311.07599/) (2023)
- [Unsupervised Question Clarity Prediction Through Retrieved Item Coherency](https://mlsanity.com/paper/2208.04882/) (2022)
- [PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction](https://mlsanity.com/paper/2406.04746/) (2024)

## Built on by (most cited first, 30 of 38)

- [EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering](https://mlsanity.com/paper/2604.05149/) (2026)
- [JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems](https://mlsanity.com/paper/2604.23478/) (2026)
- [PRL: Prompts from Reinforcement Learning](https://mlsanity.com/paper/2505.14412/) (2025)
- [Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits](https://mlsanity.com/paper/2605.18890/) (2026)
- [The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility](https://mlsanity.com/paper/2605.19537/) (2026)
- [Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation](https://mlsanity.com/paper/2506.17442/) (2025)
- [MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering](https://mlsanity.com/paper/2505.24040/) (2025)
- [Network-Level Prompt and Trait Leakage in Local Research Agents](https://mlsanity.com/paper/2508.20282/) (2025)
- [Measuring Pragmatic Influence in Large Language Model Instructions](https://mlsanity.com/paper/2602.21223/) (2026)
- [LLM Agents Are the Antidote to Walled Gardens](https://mlsanity.com/paper/2506.23978/) (2025)
- [Structured Prompts Improve Evaluation of Language Models](https://mlsanity.com/paper/2511.20836/) (2025)
- [GPS: General Per-Sample Prompter](https://mlsanity.com/paper/2511.21714/) (2025)
- [Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis](https://mlsanity.com/paper/2507.22936/) (2025)
- [Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis](https://mlsanity.com/paper/2509.00038/) (2025)
- [Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewards](https://mlsanity.com/paper/2510.04392/) (2025)
- [Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models](https://mlsanity.com/paper/2512.21964/) (2025)
- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)
- [Text-to-SQL Task-oriented Dialogue Ontology Construction](https://mlsanity.com/paper/2507.23358/) (2025)
- [CAMA: Enhancing Mathematical Reasoning in Large Language Models with Causal Knowledge](https://mlsanity.com/paper/2508.02583/) (2025)
- [Leveraging LLMs for Semi-Automatic Corpus Filtration in Systematic Literature Reviews](https://mlsanity.com/paper/2510.11409/) (2025)
- [When F1 Fails: Granularity-Aware Evaluation for Dialogue Topic Segmentation](https://mlsanity.com/paper/2512.17083/) (2025)
- [TATRA: Training-Free Instance-Adaptive Prompting Through Rephrasing and Aggregation](https://mlsanity.com/paper/2603.03298/) (2026)
- [Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Models](https://mlsanity.com/paper/2606.04109/) (2026)
- [Document-Authored Control-Signal Impersonation: A Low-Cost Indirect Prompt Attack on RAG Safety Boundaries](https://mlsanity.com/paper/2606.09005/) (2026)
- [Mitigating Context Interference for Reliable and Efficient Search Agents](https://mlsanity.com/paper/2608.10743/) (2026)
- [Rethinking Reasoning in LLMs: Neuro-Symbolic Local RetoMaton Beyond ICL and CoT](https://mlsanity.com/paper/2508.19271/) (2025)
- [TopiCLEAR: Adaptive embedding clustering for interpretable topic discovery from short texts](https://mlsanity.com/paper/2512.06694/) (2025)
- [PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses](https://mlsanity.com/paper/2603.10477/) (2026)
- [On Emotion-Sensitive Decision Making of Small Language Model Agents](https://mlsanity.com/paper/2604.06562/) (2026)
- [Green Shielding: A User-Centric Approach Towards Trustworthy AI](https://mlsanity.com/paper/2604.24700/) (2026)

## Abstract (arXiv)

Large language Models (LLMs) are highly sensitive to variations in prompt formulation, which can significantly impact their ability to generate accurate responses. In this paper, we introduce a new task, Prompt Sensitivity Prediction, and a dataset PromptSET designed to investigate the effects of slight prompt variations on LLM performance. Using TriviaQA and HotpotQA datasets as the foundation of our work, we generate prompt variations and evaluate their effectiveness across multiple LLMs. We benchmark the prompt sensitivity prediction task employing state-of-the-art methods from related tasks, including LLM-based self-evaluation, text classification, and query performance prediction techniques. Our findings reveal that existing methods struggle to effectively address prompt sensitivity prediction, underscoring the need to understand how information needs should be phrased for accurate LLM responses.