# Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approach

arXiv: https://arxiv.org/abs/2511.21709 · HTML: https://mlsanity.com/paper/2511.21709/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Builds on (most cited first, 12 of 12)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering](https://mlsanity.com/paper/2309.17249/) (2023)
- [CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges](https://mlsanity.com/paper/2410.15393/) (2024)
- [Beyond Performance: Quantifying and Mitigating Label Bias in LLMs](https://mlsanity.com/paper/2405.02743/) (2024)
- [Mitigating Selection Bias with Node Pruning and Auxiliary Options](https://mlsanity.com/paper/2409.18857/) (2024)
- [QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling](https://mlsanity.com/paper/2409.14175/) (2024)

## Built on by (most cited first, 2 of 2)

- [Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation](https://mlsanity.com/paper/2605.01846/) (2026)
- [Consider, Then Choose: Order-Invariant Decisions in System One Models](https://mlsanity.com/paper/2610.18233/) (2026)

## Abstract (arXiv)

Multiple Choice Question (MCQ) answering is a widely used method for evaluating the performance of Large Language Models (LLMs). However, LLMs often exhibit selection bias in MCQ tasks, where their choices are influenced by factors like answer position or option symbols rather than the content. This bias undermines the reliability of MCQ as an evaluation framework. Most existing selection bias metrics require answer labels and measure divergences between prediction and answer distributions, but do not fully capture the consistency of a model's predictions across different orderings of answer choices. Existing selection bias mitigation strategies have notable limitations: majority voting, though effective, is computationally prohibitive; calibration-based methods require validation sets and often fail to generalize across datasets. To address these gaps, we propose three key contributions: (1) a new unsupervised label-free Permutation Bias Metric (PBM) that directly quantifies inconsistencies in model predictions across answer permutations, providing a more precise measure of selection bias, (2) an efficient majority voting approach called Batch Question-Context KV caching (BaQCKV), to significantly reduce computational costs while preserving bias mitigation effectiveness, and (3) an unsupervised Low-Rank Adaptation (LoRA-1) fine-tuning strategy based on our proposed metric and the BaQCKV that mitigates selection bias, providing a computationally efficient alternative that maintains model generalizability. Experiments across multiple MCQ benchmarks demonstrate that our approaches reduce bias, increasing consistency in accuracy while minimizing computational costs.