mlsanityData to 7 Oct 2026

Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models

arXiv 2310.07712 ↗First posted 11 Oct 2023cs.CL cs.LGcited by 30

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking. To address this, we propose permutation self-consistency, a form of self-consistency over ranking list outputs of black-box LLMs. Our key idea is to marginalize out different list orders in the prompt to produce an order-independent ranking with less positional bias. First, given some input prompt, we repeatedly shuffle the list in the prompt and pass it through the LLM while holding the instructions the same. Next, we aggregate the resulting sample of rankings by computing the central ranking closest in distance to all of them, marginalizing out prompt order biases in the process. Theoretically, we prove the robustness of our method, showing convergence to the true ranking in the presence of random perturbations. Empirically, on five list-ranking datasets in sorting and passage reranking, our approach improves scores from conventional inference by up to 7-18% for GPT-3.5 and 8-16% for LLaMA v2 (70B), surpassing the previous state of the art in passage reranking. Our code is at https://github.com/castorini/perm-sc.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 37 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

27 of 37 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
15 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  5. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  6. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  7. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  8. 2020Overview of the TREC 2019 deep learning trackcited by 282
  9. 2021Overview of the TREC 2020 deep learning trackcited by 193
  10. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  11. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  12. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  13. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 81
  14. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 69
  15. 2023Unsupervised Contrast-Consistent Ranking with Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 6; 2026: 15.

30 papers.

Sort:

By area:

  1. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 42
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Rerankingcited by 34
  4. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  5. 2024GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understandingcited by 18
  6. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  7. 2023Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Modelscited by 14
  8. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QAcited by 13
  9. 2025Set-LLM: A Permutation-Invariant LLMcited by 10
  10. 2025Scalable In-context Ranking with Generative Modelscited by 8
  11. 2024An Investigation of Prompt Variations for Zero-shot LLM-based Rankerscited by 7
  12. 2025CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrievalcited by 5
18 more
  1. 2024What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworkscited by 4
  2. 2025LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrievalcited by 4
  3. 2026Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoningcited by 4
  4. 2026One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendationcited by 4
  5. 2026Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Rankingcited by 2
  6. 2026Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoderscited by 1
  7. 2026Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluationcited by 1
  8. 2025Enhancing Retrieval-Augmented Generation with Entity Linking for Educational Platformscited by 0
  9. 2026BlitzRank: Principled Zero-shot Ranking Agents with Tournament Graphscited by 0
  10. 2026Diagnosing LLM Reranker Behavior Under Fixed Evidence Poolscited by 0
  11. 2026Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Modelscited by 0
  12. 2026Position Bias Correction is Insufficient for One-Pass Attention Sortingcited by 0
  13. 2026Decomposer: Learning to Decompile Symbolic Music to Programscited by 0
  14. 2026Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorerscited by 0
  15. 2026MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compressioncited by 0
  16. 2026Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoringcited by 0
  17. 2026Uncovering Ordinal-Matching Bias in Audio-Visual LLMscited by 0
  18. 2026From Discovery to Decision: Finite-Budget Recoverability in LLM Votingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Boosting Self-Consistency with Ranking
  2. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
  3. 2025CoRanking: Collaborative Ranking with Small and Large Ranking Agents
  4. 2024CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation
  5. 2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
  6. 2023Self-Consistency of Large Language Models under Ambiguity
  7. 2025Aligning What LLMs Do and Say: Towards Self-Consistent Explanations
  8. 2025Optimal Self-Consistency for Efficient Reasoning with Large Language Models