mlsanityData to 7 Oct 2026

Benchmarking Prompt Sensitivity in Large Language Models

arXiv 2502.06065 ↗First posted 9 Feb 2025cs.CL cs.AI cs.IRcited by 38

Not on the map. Why?

Abstract (arXiv)

Large language Models (LLMs) are highly sensitive to variations in prompt formulation, which can significantly impact their ability to generate accurate responses. In this paper, we introduce a new task, Prompt Sensitivity Prediction, and a dataset PromptSET designed to investigate the effects of slight prompt variations on LLM performance. Using TriviaQA and HotpotQA datasets as the foundation of our work, we generate prompt variations and evaluate their effectiveness across multiple LLMs. We benchmark the prompt sensitivity prediction task employing state-of-the-art methods from related tasks, including LLM-based self-evaluation, text classification, and query performance prediction techniques. Our findings reveal that existing methods struggle to effectively address prompt sensitivity prediction, underscoring the need to understand how information needs should be phrased for accurate LLM responses.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 16 of its 41 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

16 of 41 references.

References · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  5. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  6. 2023Information Retrieval: Recent Advances and Beyondcited by 49
  7. 2023What's the Magic Word? A Control Theory of LLM Promptingcited by 41
  8. 2023Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameterscited by 32
  9. 2023Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Sciencecited by 18
  10. 2024Query Performance Prediction using Relevance Judgments Generated by Large Language Modelscited by 18
  11. 2023Query Performance Prediction: From Ad-hoc to Conversational Searchcited by 16
  12. 2023Semantic Consistency for Assuring Reliability of Large Language Modelscited by 15
4 more
  1. 2024Assessing and Verifying Task Utility in LLM-Powered Applicationscited by 15
  2. 2023Testing LLMs on Code Generation with Varying Levels of Prompt Specificitycited by 12
  3. 2022Unsupervised Question Clarity Prediction Through Retrieved Item Coherencycited by 8
  4. 2024PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Predictioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 21.

38 papers.

Sort:

By area:

  1. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 13
  2. 2026JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systemscited by 10
  3. 2025PRL: Prompts from Reinforcement Learningcited by 9
  4. 2025Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradationcited by 6
  5. 2026Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Auditscited by 6
  6. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 6
  7. 2025MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answeringcited by 5
  8. 2025Network-Level Prompt and Trait Leakage in Local Research Agentscited by 5
  9. 2025LLM Agents Are the Antidote to Walled Gardenscited by 3
  10. 2025Structured Prompts Improve Evaluation of Language Modelscited by 3
  11. 2025GPS: General Per-Sample Promptercited by 3
  12. 2026Measuring Pragmatic Influence in Large Language Model Instructionscited by 3
26 more
  1. 2025Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysiscited by 2
  2. 2025Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesiscited by 2
  3. 2025Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewardscited by 2
  4. 2025Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Modelscited by 2
  5. 2026How Open Must Language Models be to Enable Reliable Scientific Inference?cited by 2
  6. 2025Text-to-SQL Task-oriented Dialogue Ontology Constructioncited by 1
  7. 2025CAMA: Enhancing Mathematical Reasoning in Large Language Models with Causal Knowledgecited by 1
  8. 2025Leveraging LLMs for Semi-Automatic Corpus Filtration in Systematic Literature Reviewscited by 1
  9. 2025When F1 Fails: Granularity-Aware Evaluation for Dialogue Topic Segmentationcited by 1
  10. 2026TATRA: Training-Free Instance-Adaptive Prompting Through Rephrasing and Aggregationcited by 1
  11. 2026Mitigating Context Interference for Reliable and Efficient Search Agentscited by 1
  12. 2025Rethinking Reasoning in LLMs: Neuro-Symbolic Local RetoMaton Beyond ICL and CoTcited by 0
  13. 2025TopiCLEAR: Adaptive embedding clustering for interpretable topic discovery from short textscited by 0
  14. 2026PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responsescited by 0
  15. 2026On Emotion-Sensitive Decision Making of Small Language Model Agentscited by 0
  16. 2026Green Shielding: A User-Centric Approach Towards Trustworthy AIcited by 0
  17. 2026Predicting Performance of Symbolic and Prompt Programs with Examplescited by 0
  18. 2026Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Modelscited by 0
  19. 2026Document-Authored Control-Signal Impersonation: A Low-Cost Indirect Prompt Attack on RAG Safety Boundariescited by 0
  20. 2026AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Modelscited by 0
  21. 2026Understanding Evaluation Illusion in Diffusion Large Language Modelscited by 0
  22. 2026Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactionscited by 0
  23. 2026Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Modelscited by 0
  24. 2026A Comprehensive Review of Generative Physical Artificial Intelligencecited by 0
  25. 2026Hyper Algorithm Design Agent: Evolving Learnable Optimizer from Zerocited by 0
  26. 2026Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboardcited by 0