mlsanityData to 14 Oct 2026

How are Prompts Different in Terms of Sensitivity?

arXiv 2311.07230 ↗First posted 13 Nov 2023cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) has become one of the most popular learning paradigms. While there is a growing body of literature focusing on prompt engineering, there is a lack of systematic analysis comparing the effects of prompts across different models and tasks. To address this gap, we present a comprehensive prompt analysis based on the sensitivity of a function. Our analysis reveals that sensitivity is an unsupervised proxy for model performance, as it exhibits a strong negative correlation with accuracy. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output, resulting in different levels of sensitivity. Furthermore, we introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding. We show that this approach is particularly helpful when information in the input is scarce. Our work provides a fresh perspective on the analysis of prompts, and contributes to a better understanding of the mechanism of ICL.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 48 of its 60 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

48 of 60 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
36 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  3. 2018Hierarchical Neural Story Generationcited by 1,193
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  7. 2018Neural Network Acceptability Judgmentscited by 958
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  11. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  12. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  13. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  14. 2023In-Context Learning Creates Task Vectorscited by 312
  15. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  16. 2022Can language models learn from explanations in context?cited by 249
  17. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  18. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  19. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  20. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 214
  21. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  22. 2015Visualizing and Understanding Neural Models in NLPcited by 200
  23. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 169
  24. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  25. 2023The Learnability of In-Context Learningcited by 127
  26. 2022Interpreting Language Models with Contrastive Explanationscited by 92
  27. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  28. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  29. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  30. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
  31. 2022Data Curation Alone Can Stabilize In-context Learningcited by 59
  32. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
  33. 2021Sensitivity as a Complexity Measure for Sequence Classification Taskscited by 28
  34. 2021Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiabilitycited by 18
  35. 2022Can In-context Learners Learn a Reasoning Concept from Demonstrations?cited by 7
  36. 2022On Measuring the Intrinsic Few-Shot Hardness of Datasetscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 11; 2026: 12.

35 papers.

Sort:

By area:

  1. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
  2. 2024Do LLMs "know" internally when they follow instructions?cited by 26
  3. 2024Finding Visual Task Vectorscited by 24
  4. 2024Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuningcited by 16
  5. 2024Paraphrase Types Elicit Prompt Engineering Capabilitiescited by 15
  6. 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgmentcited by 15
  7. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  8. 2025What Makes a Good Natural Language Prompt?cited by 7
  9. 2024Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metriccited by 6
  10. 2024Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languagescited by 6
  11. 2025Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repaircited by 5
  12. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
23 more
  1. 2025FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarkingcited by 3
  2. 2024E-Bench: Towards Evaluating the Ease-of-Use of Large Language Modelscited by 2
  3. 2025Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articlescited by 2
  4. 2024Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Promptingcited by 1
  5. 2025RCScore: Quantifying Response Consistency in Large Language Modelscited by 1
  6. 2026PLR: Plackett-Luce for Reordering In-Context Learning Examplescited by 1
  7. 2026Document-Authored Control-Signal Impersonation: A Low-Cost Indirect Prompt Attack on RAG Safety Boundariescited by 1
  8. 2024LLMs as Function Approximators: Terminology, Taxonomy, and Questions for Evaluationcited by 0
  9. 2024Information Anxiety in Large Language Modelscited by 0
  10. 2025SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generationcited by 0
  11. 2025Towards Contamination Resistant Benchmarkscited by 0
  12. 2025Improving Document Retrieval Coherence for Semantically Equivalent Queriescited by 0
  13. 2025DICE: Discrete Interpretable Comparative Evaluation with Probabilistic Scoring for Retrieval-Augmented Generationcited by 0
  14. 2026Evaluating Adjective-Noun Compositionality in LLMs: Functional vs Representational Perspectivescited by 0
  15. 2026Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictionscited by 0
  16. 2026Prompt-Driven Code Summarization: A Systematic Literature Reviewcited by 0
  17. 2026From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Mediacited by 0
  18. 2026Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?cited by 0
  19. 2026Harnessing non-adversarial robustness in large language modelscited by 0
  20. 2026Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluationcited by 0
  21. 2026Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactionscited by 0
  22. 2026Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Qualitycited by 0
  23. 2026RCCM: Diagnosing Gains and Harms in Reasoning-Prompt Transfercited by 0