# The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs

arXiv: https://arxiv.org/abs/2502.04134 · HTML: https://mlsanity.com/paper/2502.04134/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Builds on (most cited first, 14 of 14)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [Evaluating the Factual Consistency of Large Language Models Through News Summarization](https://mlsanity.com/paper/2211.08412/) (2022)
- [Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models](https://mlsanity.com/paper/2407.04121/) (2024)
- [DelucionQA: Detecting Hallucinations in Domain-specific Question Answering](https://mlsanity.com/paper/2312.05200/) (2023)
- [Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility](https://mlsanity.com/paper/2305.10235/) (2023)
- [Shaping the Emerging Norms of Using Large Language Models in Social Computing Research](https://mlsanity.com/paper/2307.04280/) (2023)
- [Order-Independence Without Fine Tuning](https://mlsanity.com/paper/2406.06581/) (2024)

## Built on by (most cited first, 19 of 19)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [A Taxonomy of Prompt Defects in LLM Systems](https://mlsanity.com/paper/2509.14404/) (2025)
- [Meaningless Tokens, Meaningful Gains: How Activation Shifts Enhance LLM Reasoning](https://mlsanity.com/paper/2510.01032/) (2025)
- [Training-Free Active Learning Framework in Materials Science with Large Language Models](https://mlsanity.com/paper/2511.19730/) (2025)
- [ATA: A Neuro-Symbolic Approach to Implement Autonomous and Trustworthy Agents](https://mlsanity.com/paper/2510.16381/) (2025)
- [Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models](https://mlsanity.com/paper/2607.15277/) (2026)
- [Judge a Book by its Cover: Investigating Multi-Modal LLMs for Multi-Page Handwritten Document Transcription](https://mlsanity.com/paper/2502.20295/) (2025)
- [Prompt Fairness: Sub-group Disparities in LLMs](https://mlsanity.com/paper/2511.19956/) (2025)
- [Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models](https://mlsanity.com/paper/2601.22478/) (2026)
- [Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays](https://mlsanity.com/paper/2602.04604/) (2026)
- [Fine-Tuning Large Language Models for Cooperative Tactical Deconfliction of Small Unmanned Aerial Systems](https://mlsanity.com/paper/2603.28561/) (2026)
- [DynaTab: Dynamic Feature Ordering as Neural Rewiring for High-Dimensional Tabular Data](https://mlsanity.com/paper/2605.03430/) (2026)
- [Detecting Differences Is Not Understanding Structure: Large Language Models Fail at Graph Isomorphism](https://mlsanity.com/paper/2606.09484/) (2026)
- [Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs](https://mlsanity.com/paper/2506.17630/) (2025)
- [When Cow Urine Cures Constipation on YouTube: Limits of LLMs in Detecting Culture-specific Health Misinformation](https://mlsanity.com/paper/2604.22002/) (2026)
- [Green Shielding: A User-Centric Approach Towards Trustworthy AI](https://mlsanity.com/paper/2604.24700/) (2026)
- [Counterfactual Benchmarking and Training for Factuality Consistency and Order-Robust Grounded Reasoning in LLMs over Heterogeneous Knowledge](https://mlsanity.com/paper/2608.07838/) (2026)
- [PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation](https://mlsanity.com/paper/2608.27716/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become integral to diverse applications, ensuring their reliability under varying input conditions is crucial. One key issue affecting this reliability is order sensitivity, wherein slight variations in the input arrangement can lead to inconsistent or biased outputs. Although recent advances have reduced this sensitivity, the problem remains unresolved. This paper investigates the extent of order sensitivity in LLMs whose internal components are hidden from users (such as closed-source models or those accessed via API calls). We conduct experiments across multiple tasks, including paraphrasing, relevance judgment, and multiple-choice questions. Our results show that input order significantly affects performance across tasks, with shuffled inputs leading to measurable declines in output accuracy. Few-shot prompting demonstrates mixed effectiveness and offers partial mitigation; however, fails to fully resolve the problem. These findings highlight persistent risks, particularly in high-stakes applications, and point to the need for more robust LLMs or improved input-handling techniques in future development.