mlsanityData to 7 Oct 2026

ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs

arXiv 2410.12405 ↗First posted 16 Oct 2024cs.CLcited by 145

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but their performance is highly sensitive to the prompts utilized. This variability poses challenges for accurate assessment and user satisfaction. Current research frequently overlooks instance-level prompt variations and their implications on subjective evaluations. To address these shortcomings, we introduce ProSA, a framework designed to evaluate and comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity metric, PromptSensiScore, and leverages decoding confidence to elucidate underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers that prompt sensitivity fluctuates across datasets and models, with larger models exhibiting enhanced robustness. We observe that few-shot examples can alleviate this sensitivity issue, and subjective evaluations are also susceptible to prompt sensitivities, particularly in complex, reasoning-oriented tasks. Furthermore, our findings indicate that higher model confidence correlates with increased prompt robustness. We believe this work will serve as a helpful tool in studying prompt sensitivity of LLMs. The project is released at: https://github.com/open-compass/ProSA .

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 13 of its 24 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

13 of 24 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  11. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  12. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
1 more
  1. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 71; 2026: 72.

145 papers.

Sort:

By area:

  1. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  2. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  3. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
  4. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 19
  5. 2025Challenging the Evaluator: LLM Sycophancy Under User Rebuttalcited by 18
  6. 2025Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimizationcited by 15
  7. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 15
  8. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  9. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12
  10. 2026On Randomness in Agentic Evalscited by 12
  11. 2025Triangulating LLM Progress through Benchmarks, Games, and Cognitive Testscited by 11
  12. 2025Quantifying and Improving the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Datacited by 10
38 more
  1. 2025ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Momentscited by 10
  2. 2025When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMscited by 10
  3. 2025Culture is Everywhere: A Call for Intentionally Cultural Evaluationcited by 10
  4. 2025Silenced Biases: The Dark Side LLMs Learned to Refusecited by 10
  5. 2026JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systemscited by 10
  6. 2025PRL: Prompts from Reinforcement Learningcited by 9
  7. 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failurecited by 9
  8. 2025Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefscited by 8
  9. 2025Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMscited by 8
  10. 2025CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experimentscited by 8
  11. 2025Meaningless Tokens, Meaningful Gains: How Activation Shifts Enhance LLM Reasoningcited by 8
  12. 2026The Fragility Of Moral Judgment In Large Language Modelscited by 8
  13. 2025Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probingcited by 7
  14. 2025Knowledge Integration for Physics-informed Symbolic Regression Using Pre-trained Large Language Modelscited by 7
  15. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 7
  16. 2025Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradationcited by 6
  17. 2026ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Modelscited by 6
  18. 2025Limits of LLM Text Detectors in Educationcited by 5
  19. 2025Conflict-Aware Soft Prompting for Retrieval-Augmented Generationcited by 5
  20. 2025If We May De-Presuppose: Robustly Verifying Claims through Presupposition-Free Question Decompositioncited by 5
  21. 2025Human Preferences in Large Language Model Latent Space: A Technical Analysis on the Reliability of Synthetic Data in Voting Outcome Predictioncited by 4
  22. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  23. 2025Talk Less, Call Right: Enhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Promptingcited by 4
  24. 2025Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysiscited by 4
  25. 2025EvalCards: A Framework for Standardized Evaluation Reportingcited by 4
  26. 2025GREATERPROMPT: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimizationcited by 3
  27. 2025Leveraging LLM Inconsistency to Boost Pass@k Performancecited by 3
  28. 2025A Multimodal, Multilingual, and Multidimensional Pipeline for Fine-grained Crowdsourcing Earthquake Damage Evaluationcited by 3
  29. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
  30. 2025AHAMask: Reliable Task Specification for Large Audio Language Models without Instructionscited by 3
  31. 2025Representing LLMs in Prompt Semantic Task Spacecited by 3
  32. 2025Lita: Light Agent Uncovers the Agentic Coding Capabilities of LLMscited by 3
  33. 2025ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Modelscited by 3
  34. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  35. 2026PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Modelscited by 3
  36. 2025Choices Speak Louder than Questionscited by 2
  37. 2025Weaker LLMs' Opinions Also Matter: Mixture of Opinions Enhances LLM's Mathematical Reasoningcited by 2
  38. 2025Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMscited by 2

50 most cited of 145 All 145 (JSON)