# Towards LLMs Robustness to Changes in Prompt Format Styles

arXiv: https://arxiv.org/abs/2504.06969 · HTML: https://mlsanity.com/paper/2504.06969/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 9 Apr 2025.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Understanding the Failure Modes of Out-of-Distribution Generalization](https://mlsanity.com/paper/2010.15775/) (2020)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [On Calibration and Out-of-domain Generalization](https://mlsanity.com/paper/2102.10395/) (2021)
- [Does Invariant Risk Minimization Capture Invariance?](https://mlsanity.com/paper/2101.01134/) (2021)
- [Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering](https://mlsanity.com/paper/2309.17249/) (2023)
- [Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements](https://mlsanity.com/paper/2401.06766/) (2024)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [Better Zero-Shot Reasoning with Self-Adaptive Prompting](https://mlsanity.com/paper/2305.14106/) (2023)
- [Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning](https://mlsanity.com/paper/2306.01150/) (2023)
- [Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization](https://mlsanity.com/paper/2406.15708/) (2024)
- [Optimization-based Causal Estimation from Heterogenous Environments](https://mlsanity.com/paper/2109.11990/) (2021)
- [Simple Disentanglement of Style and Content in Visual Representations](https://mlsanity.com/paper/2302.09795/) (2023)
- [Searching for consistent associations with a multi-environment knockoff filter](https://mlsanity.com/paper/2106.04118/) (2021)

## Built on by (most cited first, 21 of 21)

- [MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them](https://mlsanity.com/paper/2507.21017/) (2025)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [Paraphrasing Adversarial Attack on LLM-as-a-Reviewer](https://mlsanity.com/paper/2601.06884/) (2026)
- [Compositional Steering of Large Language Models with Steering Tokens](https://mlsanity.com/paper/2601.05062/) (2026)
- [Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs](https://mlsanity.com/paper/2510.14242/) (2025)
- [Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents](https://mlsanity.com/paper/2510.14453/) (2025)
- [PRISON: Unmasking the Criminal Potential of Large Language Models](https://mlsanity.com/paper/2506.16150/) (2025)
- [Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3](https://mlsanity.com/paper/2603.27844/) (2026)
- [Pioneer Agent: Continual Improvement of Small Language Models in Production](https://mlsanity.com/paper/2604.09791/) (2026)
- [PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection](https://mlsanity.com/paper/2605.24171/) (2026)
- [Mitigating LLM-based p-Hacking by Preregistering for the Next LLM](https://mlsanity.com/paper/2606.27687/) (2026)
- [Instance-level Randomization: Toward More Stable LLM Evaluations](https://mlsanity.com/paper/2509.12678/) (2025)
- [Large Language Models for Geolocation Extraction in Humanitarian Crisis Response](https://mlsanity.com/paper/2602.08872/) (2026)
- [AI Agent Systems for Supply Chains: Structured Decision Prompts and Memory Retrieval](https://mlsanity.com/paper/2602.05524/) (2026)
- [Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare](https://mlsanity.com/paper/2603.00917/) (2026)
- [Natural Language Interfaces for Spatial and Temporal Databases: A Comprehensive Overview of Methods, Taxonomy, and Future Directions](https://mlsanity.com/paper/2603.23375/) (2026)
- [Voice Under Revision: Large Language Models and the Normalization of Personal Narrative](https://mlsanity.com/paper/2604.22142/) (2026)
- [Green Shielding: A User-Centric Approach Towards Trustworthy AI](https://mlsanity.com/paper/2604.24700/) (2026)
- [Greener Than Humans? Environmental Attitudes in Large Language Models](https://mlsanity.com/paper/2606.02741/) (2026)
- [Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality](https://mlsanity.com/paper/2608.20349/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have gained popularity in recent years for their utility in various applications. However, they are sensitive to non-semantic changes in prompt formats, where small changes in the prompt format can lead to significant performance fluctuations. In the literature, this problem is commonly referred to as prompt brittleness. Previous research on prompt engineering has focused mainly on developing techniques for identifying the optimal prompt for specific tasks. Some studies have also explored the issue of prompt brittleness and proposed methods to quantify performance variations; however, no simple solution has been found to address this challenge. We propose Mixture of Formats (MOF), a simple and efficient technique for addressing prompt brittleness in LLMs by diversifying the styles used in the prompt few-shot examples. MOF was inspired by computer vision techniques that utilize diverse style datasets to prevent models from associating specific styles with the target variable. Empirical results show that our proposed technique reduces style-induced prompt brittleness in various LLMs while also enhancing overall performance across prompt variations and different datasets.