Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models
Not on the map. Why?
Abstract (arXiv)
Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets. Through systematic architectural analysis, we identify causal attention as the core mechanism: in QOC prompts, the causal mask prevents option tokens from attending to context, creating an information bottleneck where context becomes invisible to options.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 20 Jan 2026.
Builds on
20 of 27 references.
References · 20
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
- 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2020Quantifying Attention Flow in Transformerscited by 775
- 2017Crowdsourcing Multiple Choice Science Questionscited by 765
8 more
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
- 2021Causal Abstractions of Neural Networkscited by 422
- 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
- 2024Make Your LLM Fully Utilize the Contextcited by 119
- 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
- 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
- 2025ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatabilitycited by 9
- 2024It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehensioncited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.