# Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation

arXiv: https://arxiv.org/abs/2605.01846 · HTML: https://mlsanity.com/paper/2605.01846/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 3 May 2026.

## Builds on (most cited first, 21 of 21)

- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [Questioning the Survey Responses of Large Language Models](https://mlsanity.com/paper/2306.07951/) (2023)
- [Can multiple-choice questions really be useful in detecting the abilities of LLMs?](https://mlsanity.com/paper/2403.17752/) (2024)
- [Do language models plan ahead for future tokens?](https://mlsanity.com/paper/2404.00859/) (2024)
- [Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo](https://mlsanity.com/paper/2504.13139/) (2025)
- [Beyond Performance: Quantifying and Mitigating Label Bias in LLMs](https://mlsanity.com/paper/2405.02743/) (2024)
- [Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2406.16033/) (2024)
- [Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models](https://mlsanity.com/paper/2406.03009/) (2024)
- [Emergent Response Planning in LLMs](https://mlsanity.com/paper/2502.06258/) (2025)
- [Mitigating Selection Bias with Node Pruning and Auxiliary Options](https://mlsanity.com/paper/2409.18857/) (2024)
- [Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models](https://mlsanity.com/paper/2410.14248/) (2024)
- [Extracting Paragraphs from LLM Token Activations](https://mlsanity.com/paper/2409.06328/) (2024)
- [Steer2Adapt: Dynamically Composing Steering Vectors Elicits Efficient Adaptation of LLMs](https://mlsanity.com/paper/2602.07276/) (2026)
- [What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering](https://mlsanity.com/paper/2601.20164/) (2026)
- [Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approach](https://mlsanity.com/paper/2511.21709/) (2025)

## Built on by (most cited first, 1 of 1)

- [When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora](https://mlsanity.com/paper/2610.00202/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to generate multiple-choice questions (MCQs), where correct answers should ideally be uniformly distributed across options. However, we observe that LLMs exhibit systematic position biases during generation. Through extensive experiments with 10 LLMs and 5 vision-language models (VLMs) on three MCQ generation tasks, we show that these biases are structured, with similar patterns emerging within model families. To investigate the underlying mechanisms, we conduct probing experiments and find that hidden representations in the question stem encode predictive signals of the correct answer position, suggesting that answer position may be implicitly planned during generation. Building on this insight, we apply activation steering to manipulate internal representations and influence answer position. Our results show that steering can partially control positional preferences and substantially shift answer position distributions. Our findings provide a practical framework for studying implicit positional planning in LLMs and highlight the importance of controllable generation for reliable MCQ construction and evaluation.