mlsanityData to 7 Oct 2026

Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models

arXiv 2406.03009 ↗First posted 5 Jun 2024cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the phenomena of "selection biases" in Large Language Models (LLMs), focusing on problems where models are tasked with choosing the optimal option from an ordered sequence. We delve into biases related to option order and token usage, which significantly impact LLMs' decision-making processes. We also quantify the impact of these biases through an extensive empirical analysis across multiple models and tasks. Furthermore, we propose mitigation strategies to enhance model performance. Our key contributions are threefold: 1) Precisely quantifying the influence of option order and token on LLMs, 2) Developing strategies to mitigate the impact of token and order sensitivity to enhance robustness, and 3) Offering a detailed analysis of sensitivity across models and tasks, which informs the creation of more stable and reliable LLM applications for selection problems.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

19 of 30 references.

References · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  12. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
7 more
  1. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  2. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  3. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  4. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  5. 2023Mitigating Label Biases for In-context Learningcited by 88
  6. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 58
  7. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?cited by 74
  2. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  3. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 35
  4. 2024Mitigating Selection Bias with Node Pruning and Auxiliary Optionscited by 12
  5. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 9
  6. 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
  7. 2024Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slowcited by 8
  8. 2025BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Modelscited by 8
  9. 2024Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimationcited by 7
  10. 2025None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answeringcited by 5
  11. 2025Hearing the Order: Investigating Position Bias in Large Audio-Language Modelscited by 4
  12. 2024Too Big to Fool: Resisting Deception in Language Modelscited by 2
8 more
  1. 2025SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Modelscited by 2
  2. 2025Enhancing Clinical Multiple-Choice Questions Benchmarks with Knowledge Graph Guided Distractor Generationcited by 1
  3. 2025Evaluating Variance in Visual Question Answering Benchmarkscited by 1
  4. 2026Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generationcited by 1
  5. 2026Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluationcited by 1
  6. 2026Corpus Prevalence of Multiple-Choice Question Optionscited by 0
  7. 2026CAMD: Coverage-Aware Multimodal Decoding for Efficient Reasoning of Multimodal Large Language Modelscited by 0
  8. 2026Set-shifting Behavioral Test for Harnessed Agentscited by 0