# Consider, Then Choose: Order-Invariant Decisions in System One Models

arXiv: https://arxiv.org/abs/2610.18233 · HTML: https://mlsanity.com/paper/2610.18233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 16 of 16)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering](https://mlsanity.com/paper/2309.17249/) (2023)
- [Eliminating Position Bias of Language Models: A Mechanistic Approach](https://mlsanity.com/paper/2407.01100/) (2024)
- [Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It](https://mlsanity.com/paper/2609.26758/) (2026)
- [POE: Process of Elimination for Multiple Choice Reasoning](https://mlsanity.com/paper/2310.15575/) (2023)
- [Order-Independence Without Fine Tuning](https://mlsanity.com/paper/2406.06581/) (2024)
- [One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation](https://mlsanity.com/paper/2604.27599/) (2026)
- [RoToR: Towards More Reliable Responses for Order-Invariant Inputs](https://mlsanity.com/paper/2502.08662/) (2025)
- [Option-ID Based Elimination For Multiple Choice Questions](https://mlsanity.com/paper/2501.15175/) (2025)
- [Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approach](https://mlsanity.com/paper/2511.21709/) (2025)

## Abstract (arXiv)

System One models answer typed questions in a single forward pass and promise the same answer for the same input. For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose Runoff, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.