# Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following

arXiv: https://arxiv.org/abs/2502.17204 · HTML: https://mlsanity.com/paper/2502.17204/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [Visualizing and Understanding Neural Models in NLP](https://mlsanity.com/paper/1506.01066/) (2015)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [Premise Order Matters in Reasoning with Large Language Models](https://mlsanity.com/paper/2402.08939/) (2024)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning](https://mlsanity.com/paper/2310.00492/) (2023)
- [Following Length Constraints in Instructions](https://mlsanity.com/paper/2406.17744/) (2024)
- [Testing AI on language comprehension tasks reveals insensitivity to underlying meaning](https://mlsanity.com/paper/2302.12313/) (2023)
- [The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models](https://mlsanity.com/paper/2406.19999/) (2024)
- [Contrastive Instruction Tuning](https://mlsanity.com/paper/2402.11138/) (2024)
- [LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty](https://mlsanity.com/paper/2402.10573/) (2024)
- [Kendall Tau Sequence Distance: Extending Kendall Tau from Ranks to Sequences](https://mlsanity.com/paper/1905.02752/) (2019)

## Built on by (most cited first, 12 of 12)

- [How Many Instructions Can LLMs Follow at Once?](https://mlsanity.com/paper/2507.11538/) (2025)
- [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://mlsanity.com/paper/2506.01413/) (2025)
- [More or Less Wrong: A Benchmark for Directional Bias in LLM Comparative Reasoning](https://mlsanity.com/paper/2506.03923/) (2025)
- [IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2508.04632/) (2025)
- [Revisiting the Reliability of Language Models in Instruction-Following](https://mlsanity.com/paper/2512.14754/) (2025)
- [Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages](https://mlsanity.com/paper/2601.06341/) (2026)
- [Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuning](https://mlsanity.com/paper/2505.11922/) (2025)
- [TransEvalnia: Reasoning-based Evaluation and Ranking of Translations](https://mlsanity.com/paper/2507.12724/) (2025)
- [To Copy or Not to Copy: Copying Is Easier to Induce Than Recall](https://mlsanity.com/paper/2601.12075/) (2026)
- [Financial Instruction Following Evaluation (FIFE)](https://mlsanity.com/paper/2512.08965/) (2025)
- [Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement](https://mlsanity.com/paper/2607.10590/) (2026)
- [Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction](https://mlsanity.com/paper/2608.12426/) (2026)

## Abstract (arXiv)

Real-world instructions with multiple constraints pose a significant challenge to existing large language models (LLMs). An observation is that the LLMs exhibit dramatic performance fluctuation when disturbing the order of the incorporated constraints. Yet, none of the existing works has systematically investigated this position bias problem in the field of multi-constraint instruction following. To bridge this gap, we design a probing task where we quantitatively measure the difficulty distribution of the constraints by a novel Difficulty Distribution Index (CDDI). Through the experimental results, we find that LLMs are more performant when presented with the constraints in a ``hard-to-easy'' order. This preference can be generalized to LLMs with different architecture or different sizes of parameters. Additionally, we conduct an explanation study, providing an intuitive insight into the correlation between the LLM's attention and constraint orders. Our code and dataset are publicly available at https://github.com/meowpass/PBIF.