mlsanityData to 7 Oct 2026

Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following

arXiv 2502.17204 ↗First posted 24 Feb 2025cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Real-world instructions with multiple constraints pose a significant challenge to existing large language models (LLMs). An observation is that the LLMs exhibit dramatic performance fluctuation when disturbing the order of the incorporated constraints. Yet, none of the existing works has systematically investigated this position bias problem in the field of multi-constraint instruction following. To bridge this gap, we design a probing task where we quantitatively measure the difficulty distribution of the constraints by a novel Difficulty Distribution Index (CDDI). Through the experimental results, we find that LLMs are more performant when presented with the constraints in a ``hard-to-easy'' order. This preference can be generalized to LLMs with different architecture or different sizes of parameters. Additionally, we conduct an explanation study, providing an intuitive insight into the correlation between the LLM's attention and constraint orders. Our code and dataset are publicly available at https://github.com/meowpass/PBIF.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

26 of 35 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
14 more
  1. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  2. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  3. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  4. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  5. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  6. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  7. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  8. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  9. 2023Testing AI on language comprehension tasks reveals insensitivity to underlying meaningcited by 45
  10. 2024Following Length Constraints in Instructionscited by 45
  11. 2024The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Modelscited by 18
  12. 2024Contrastive Instruction Tuningcited by 17
  13. 2024LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertaintycited by 12
  14. 2019Kendall Tau Sequence Distance: Extending Kendall Tau from Ranks to Sequencescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 4.

12 papers.

  1. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  2. 2025How Many Instructions Can LLMs Follow at Once?cited by 17
  3. 2025More or Less Wrong: A Benchmark for Directional Bias in LLM Comparative Reasoningcited by 6
  4. 2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 6
  5. 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 6
  6. 2026Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languagescited by 4
  7. 2025Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuningcited by 2
  8. 2025TransEvalnia: Reasoning-based Evaluation and Ranking of Translationscited by 1
  9. 2026To Copy or Not to Copy: Copying Is Easier to Induce Than Recallcited by 1
  10. 2025Financial Instruction Following Evaluation (FIFE)cited by 0
  11. 2026Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreementcited by 0
  12. 2026Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfactioncited by 0