mlsanityData to 14 Oct 2026

Consider, Then Choose: Order-Invariant Decisions in System One Models

arXiv 2610.18233 ↗First posted 13 Oct 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

System One models answer typed questions in a single forward pass and promise the same answer for the same input. For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose Runoff, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 35 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

16 of 35 references.

References · 16

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  6. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  7. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  8. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  9. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  10. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 25
  11. 2023POE: Process of Elimination for Multiple Choice Reasoningcited by 8
  12. 2024Order-Independence Without Fine Tuningcited by 6
4 more
  1. 2026One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendationcited by 5
  2. 2025RoToR: Towards More Reliable Responses for Order-Invariant Inputscited by 3
  3. 2025Option-ID Based Elimination For Multiple Choice Questionscited by 2
  4. 2025Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approachcited by 2

Built on by

Not cited yet.