# Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models

arXiv: https://arxiv.org/abs/2406.03009 · HTML: https://mlsanity.com/paper/2406.03009/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Mitigating Label Biases for In-context Learning](https://mlsanity.com/paper/2305.19148/) (2023)
- [Adversarial Demonstration Attacks on Large Language Models](https://mlsanity.com/paper/2305.14950/) (2023)
- [Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations](https://mlsanity.com/paper/2305.13299/) (2023)

## Built on by (most cited first, 20 of 20)

- [Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?](https://mlsanity.com/paper/2508.17536/) (2025)
- [Do LLM Evaluators Prefer Themselves for a Reason?](https://mlsanity.com/paper/2504.03846/) (2025)
- [Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs](https://mlsanity.com/paper/2406.09367/) (2024)
- [Mitigating Selection Bias with Node Pruning and Auxiliary Options](https://mlsanity.com/paper/2409.18857/) (2024)
- [BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models](https://mlsanity.com/paper/2510.00307/) (2025)
- [Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering](https://mlsanity.com/paper/2504.07583/) (2025)
- [SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questions](https://mlsanity.com/paper/2506.00643/) (2025)
- [Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slow](https://mlsanity.com/paper/2408.08651/) (2024)
- [Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation](https://mlsanity.com/paper/2412.11831/) (2024)
- [None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answering](https://mlsanity.com/paper/2503.01550/) (2025)
- [Hearing the Order: Investigating Position Bias in Large Audio-Language Models](https://mlsanity.com/paper/2510.00628/) (2025)
- [Too Big to Fool: Resisting Deception in Language Models](https://mlsanity.com/paper/2412.10558/) (2024)
- [SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models](https://mlsanity.com/paper/2507.18182/) (2025)
- [Enhancing Clinical Multiple-Choice Questions Benchmarks with Knowledge Graph Guided Distractor Generation](https://mlsanity.com/paper/2506.00612/) (2025)
- [Evaluating Variance in Visual Question Answering Benchmarks](https://mlsanity.com/paper/2508.02645/) (2025)
- [Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation](https://mlsanity.com/paper/2605.01846/) (2026)
- [Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluation](https://mlsanity.com/paper/2608.08869/) (2026)
- [Corpus Prevalence of Multiple-Choice Question Options](https://mlsanity.com/paper/2602.17377/) (2026)
- [CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Models](https://mlsanity.com/paper/2603.14745/) (2026)
- [Set-shifting Behavioral Test for Harnessed Agents](https://mlsanity.com/paper/2607.13396/) (2026)

## Abstract (arXiv)

In this paper, we investigate the phenomena of "selection biases" in Large Language Models (LLMs), focusing on problems where models are tasked with choosing the optimal option from an ordered sequence. We delve into biases related to option order and token usage, which significantly impact LLMs' decision-making processes. We also quantify the impact of these biases through an extensive empirical analysis across multiple models and tasks. Furthermore, we propose mitigation strategies to enhance model performance. Our key contributions are threefold: 1) Precisely quantifying the influence of option order and token on LLMs, 2) Developing strategies to mitigate the impact of token and order sensitivity to enhance robustness, and 3) Offering a detailed analysis of sensitivity across models and tasks, which informs the creation of more stable and reliable LLM applications for selection problems.