mlsanityData to 14 Oct 2026

RoToR: Towards More Reliable Responses for Order-Invariant Inputs

arXiv 2502.08662 ↗First posted 10 Feb 2025cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Mitigating positional bias of language models (LMs) for listwise inputs is a well-known and important problem (e.g., lost-in-the-middle). While zero-shot order-invariant LMs have been proposed to solve this issue, their success on practical listwise problems has been limited. In this work, as a first contribution, we identify and overcome two limitations to make zero-shot invariant LMs more practical: (1) training and inference distribution mismatch arising from modifying positional ID assignments to enforce invariance, and (2) failure to adapt to mixture of order-invariant and sensitive inputs in practical listwise problems. Then, to overcome these issues we propose (1) RoToR, a zero-shot invariant LM for genuinely order-invariant inputs with minimal modifications of positional IDs, and (2) Selective Routing, an adaptive framework that handles both order-invariant and order-sensitive inputs in listwise tasks. On the Lost in the middle (LitM), Knowledge Graph QA (KGQA), and MMLU benchmarks, we show that RoToR with Selective Routing can effectively handle practical listwise input tasks in a zero-shot manner (https://github.com/soyoung97/RoToR)

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

30 of 43 references.

References · 30

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  7. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  9. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  10. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  11. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  12. 2019Relational Pooling for Graph Representationscited by 150
18 more
  1. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  2. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  3. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  4. 2018Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputscited by 98
  5. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 91
  6. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  7. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  8. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 72
  9. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  10. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  11. 2023RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbationscited by 43
  12. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  13. 2024Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Biascited by 28
  14. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  15. 2024FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answeringcited by 16
  16. 2023Scaling In-Context Demonstrations with Structured Attentioncited by 8
  17. 2024Inference Scaling for Bridging Retrieval and Augmented Generationcited by 7
  18. 2023Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deteriorationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPOcited by 3
  2. 2026The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Eracited by 0
  3. 2026Consider, Then Choose: Order-Invariant Decisions in System One Modelscited by 0