mlsanityData to 14 Oct 2026

Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

arXiv 2310.11324 ↗First posted 17 Oct 2023cs.CL cs.AI cs.LGcited by 817

Not on the map. Why?

Cited by 86 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As large language models (LLMs) are adopted as a fundamental component of language technologies, it is crucial to accurately characterize their performance. Because choices in prompt design can strongly influence model behavior, this design process is critical in effectively using any modern pre-trained generative language model. In this work, we focus on LLM sensitivity to a quintessential class of meaning-preserving design choices: prompt formatting. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B. Sensitivity remains even when increasing model size, the number of few-shot examples, or performing instruction tuning. Our analysis suggests that work evaluating LLMs with prompting-based methods would benefit from reporting a range of performance across plausible prompt formats, instead of the currently-standard practice of reporting performance on a single format. We also show that format performance only weakly correlates between models, which puts into question the methodological validity of comparing models with an arbitrarily chosen, fixed prompt format. To facilitate systematic analysis we propose FormatSpread, an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights. Furthermore, we present a suite of analyses that characterize the nature of this sensitivity, including exploring the influence of particular atomic perturbations and the internal representation of particular formats.

Key facts. Cited by 817 papers in the map (data through 14 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

27 of 42 references.

References · 27

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  9. 2022Editing Models with Task Arithmeticcited by 1,120
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  11. 2019How Can We Know What Language Models Know?cited by 959
  12. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
15 more
  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  2. 2017Deep Reinforcement Learning that Matterscited by 754
  3. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  5. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  6. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  7. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  8. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  9. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  10. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  11. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  12. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  13. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  14. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  15. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 202; 2025: 208; 2026: 398.

817 papers.

Sort:

By area:

  1. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  3. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  4. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  5. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  6. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  7. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  8. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  9. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  10. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  11. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  12. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
38 more
  1. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
  2. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  3. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  4. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  5. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  6. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  7. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  9. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  10. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  11. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  12. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  13. 2026Self-Harness: Harnesses That Improve Themselvescited by 107
  14. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  15. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 100
  16. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
  17. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  18. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  19. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  20. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  21. 2024Can large language models explore in-context?cited by 76
  22. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
  23. 2024Efficient multi-prompt evaluation of LLMscited by 73
  24. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  25. 2024Repetition Improves Language Model Embeddingscited by 72
  26. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 68
  27. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  28. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  29. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 59
  30. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
  31. 2024Compression Represents Intelligence Linearlycited by 56
  32. 2024Quantifying Variance in Evaluation Benchmarkscited by 54
  33. 2024Large Language Models Assume People are More Rational than We Really arecited by 54
  34. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  35. 2023Prompt Engineering a Prompt Engineercited by 53
  36. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
  37. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  38. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 52

50 most cited of 817 All 817 (JSON)