mlsanityData to 14 Oct 2026

Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

arXiv 2608.26762 ↗First posted 27 Aug 2026cs.CL cs.IR cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one prompt, each still receiving its own score. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or which chosen/rejected pair enters preference training. Because the candidates share that prompt, reordering them changes their scores. The same query over the same candidates should still yield the same decision. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66-0.84 when reordered. No prompt-time change we test resolves that dependence: the only one that improves ranking quality does not measurably improve decision stability. We introduce order-consistency SFT (OC-SFT), which attenuates it in the weights by penalizing disagreement between a candidate's scores across orderings. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks. It is also more stable on 12 base models than order-averaged distillation, which trains on labels averaged across permutations. One OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison of such scorers should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://github.com/thomsonreuters/presentation-dependence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 52 of its 61 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

52 of 61 references.

References · 52

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  12. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
38 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  3. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  4. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  5. 2020Overview of the TREC 2019 deep learning trackcited by 286
  6. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  7. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  8. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  9. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  10. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  11. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  12. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  13. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  14. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  15. 2024How to Evaluate Reward Models for RLHFcited by 88
  16. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 70
  17. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 70
  18. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  19. 2024FIRST: Faster Improved Listwise Reranking with Single Token Decodingcited by 54
  20. 2019SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrievalcited by 48
  21. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  22. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  23. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 31
  24. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
  25. 2025jina-reranker-v3: Last but Not Late Interaction for Listwise Document Rerankingcited by 24
  26. 2024Top-Down Partitioning for Efficient List-Wise Rankingcited by 13
  27. 2024Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Modelscited by 12
  28. 2025ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Rerankingcited by 11
  29. 2025Batched Self-Consistency Improves LLM Relevance Assessment and Rankingcited by 8
  30. 2024Self-Calibrated Listwise Reranking with Large Language Modelscited by 7
  31. 2025Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Shortcited by 7
  32. 2024Order-Independence Without Fine Tuningcited by 6
  33. 2025An Empirical Study of Position Bias in Modern Information Retrievalcited by 6
  34. 2026Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generationcited by 5
  35. 2026One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendationcited by 5
  36. 2025Contextual Relevance and Adaptive Sampling for LLM-Based Document Rerankingcited by 4
  37. 2026Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibrationcited by 4
  38. 2026Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPOcited by 3

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Can LLM Rerankers Predict Their Own Ranking Performance?
  2. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  3. 2026Ranking Reasoning LLMs under Test-Time Scaling
  4. 2026Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge
  5. 2026Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs
  6. 2025How Reliable are LLMs for Reasoning on the Re-ranking task?
  7. 2022Ranking Inferences Based on the Top Choice of Multiway Comparisons
  8. 2025Reference-Free Rating of LLM Responses via Latent Information