mlsanityData to 14 Oct 2026

When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs

arXiv 2508.11383 ↗First posted 15 Aug 2025cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are highly sensitive to subtle, non-semantic variations in prompt phrasing and formatting. In this work, we present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework. We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset. Our evaluation covers robustness methods from both fine-tuned and in-context learning paradigms, and tests their generalization against multiple types of distribution shifts. Finally, we extend our analysis to GPT-4.1 and DeepSeek V3 to assess frontier models' current robustness to format perturbations. Our findings offer actionable insights into the relative effectiveness of these robustness methods, enabling practitioners to make informed decisions when aiming for stable and reliable LLM performance in real-world applications. Code: https://github.com/AIRI-Institute/when-punctuation-matters.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

22 of 31 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  11. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  12. 2022A Survey on In-context Learningcited by 721
10 more
  1. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  2. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  3. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  4. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  5. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  6. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  7. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  8. 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
  9. 2023Supervised Knowledge Makes Large Language Models Better In-context Learnerscited by 20
  10. 2024Learning to Retrieve Iteratively for In-Context Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025The Rogue Scalpel: Activation Steering Compromises LLM Safetycited by 21
  2. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 8
  3. 2025Prompt Stability in Code LLMs: Measuring Sensitivity across Emotion- and Personality-Driven Variationscited by 1
  4. 2025Evidence-Driven Decision Support for AI Model Selection in Research Software Engineeringcited by 1
  5. 2026Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Modelscited by 1
  6. 2026Voice Under Revision: Large Language Models and the Normalization of Personal Narrativecited by 0
  7. 2026Harnessing non-adversarial robustness in large language modelscited by 0
  8. 2026Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Computecited by 0
  9. 2026Prompt-Robust Language Models: Which Training Strategies Work?cited by 0
  10. 2026Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMscited by 0