mlsanityData to 14 Oct 2026

Eliminating Position Bias of Language Models: A Mechanistic Approach

arXiv 2407.01100 ↗First posted 1 Jul 2024cs.CL cs.LGcited by 65

Not on the map. Why?

Abstract (arXiv)

Position bias has proven to be a prevalent issue of modern language models (LMs), where the models prioritize content based on its position within the given context. This bias often leads to unexpected model failures and hurts performance, robustness, and reliability across various applications. Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings. Based on the analyses, we propose to eliminate position bias (e.g., different retrieved documents' orders in QA affect performance) with a training-free zero-shot approach. Our method changes the causal attention to bidirectional attention between documents and utilizes model attention values to decide the relative orders of documents instead of using the order provided in input prompts, therefore enabling Position-INvariant inferencE (PINE) at the document level. By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning. Notably, PINE is especially useful when adapting LMs for evaluating reasoning pairs: it consistently provides 8 to 10 percentage points performance gains, making Llama-3-70B-Instruct perform even better than GPT-4-0125-preview and GPT-4o-2024-08-06 on the RewardBench reasoning set.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

40 of 56 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
28 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2021E(n) Equivariant Graph Neural Networkscited by 965
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  8. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  9. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  10. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  11. 2022UL2: Unifying Language Learning Paradigmscited by 295
  12. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 191
  13. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  14. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  15. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  16. 2022Parallel Context Windows for Large Language Modelscited by 81
  17. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  18. 2023Questioning the Survey Responses of Large Language Modelscited by 74
  19. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 72
  20. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 54
  21. 2023Primacy Effect of ChatGPTcited by 27
  22. 2024Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learningcited by 20
  23. 2023Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verificationcited by 19
  24. 2024Discovering Bias in Latent Space: An Unsupervised Debiasing Approachcited by 15
  25. 20243D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encodingcited by 13
  26. 2024In-Context Example Ordering Guided by Label Distributionscited by 12
  27. 2023Scaling In-Context Demonstrations with Structured Attentioncited by 8
  28. 2024Attention Instruction: Amplifying Attention in the Middle via Promptingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 30; 2026: 29.

65 papers.

Sort:

By area:

  1. 2025On the Emergence of Position Bias in Transformerscited by 59
  2. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 53
  3. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  4. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  5. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 20
  6. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 14
  7. 2025Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generationcited by 12
  8. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  9. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 11
  10. 2025BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Modelscited by 11
  11. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
  12. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 9
38 more
  1. 2025Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMscited by 9
  2. 2024Inference Scaling for Bridging Retrieval and Augmented Generationcited by 7
  3. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  4. 2024Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Bothcited by 5
  5. 2025Unstructured Evidence Attribution for Long Context Query Focused Summarizationcited by 5
  6. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  7. 2025Exploiting Primacy Effect To Improve Large Language Modelscited by 5
  8. 2025CTR-Sink: Attention Sink for Language Models in Click-Through Rate Predictioncited by 4
  9. 2024Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Taskscited by 3
  10. 2024EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-betacited by 3
  11. 2025RoToR: Towards More Reliable Responses for Order-Invariant Inputscited by 3
  12. 2025Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understandingcited by 3
  13. 2025DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answeringcited by 3
  14. 2025LLMs as In-Context Meta-Learners for Model and Hyperparameter Selectioncited by 3
  15. 2025Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environmentscited by 3
  16. 2026Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneckcited by 3
  17. 2026Subliminal Signals in Preference Labelscited by 3
  18. 2026Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPOcited by 3
  19. 2026AesRM: Improving Video Aesthetics with Expert-Level Feedbackcited by 3
  20. 2025On the Reasoning Capacity of AI Models and How to Quantify Itcited by 2
  21. 2025Computation Mechanism Behind LLM Position Generalizationcited by 2
  22. 2025Reranking-based Generation for Unbiased Perspective Summarizationcited by 2
  23. 2025Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Modelscited by 2
  24. 2025The Chameleon Nature of LLMs: Quantifying Multi-Turn Stance Instability in Search-Enabled Language Modelscited by 2
  25. 2025Mitigating Coordinate Prediction Bias from Positional Encoding Failurescited by 2
  26. 2026Understanding the Dynamics of Demonstration Conflict in In-Context Learningcited by 2
  27. 2026The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMscited by 2
  28. 2026How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalizationcited by 2
  29. 2024Membership Inference Attack against Long-Context Large Language Modelscited by 1
  30. 2025Positional Bias in Binary Question Answering: How Uncertainty Shapes Model Preferencescited by 1
  31. 2025Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understandingcited by 1
  32. 2025Adaptive Repetition for Mitigating Position Bias in LLM-Based Rankingcited by 1
  33. 2025GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoningcited by 1
  34. 2026Layer-wise Positional Bias in Short-Context Language Modelingcited by 1
  35. 2026Scaling Reward Modeling without Human Supervisioncited by 1
  36. 2026Self-Describing Structured Data with Dual-Layer Guidance: A Lightweight Alternative to RAG for Precision Retrieval in Large-Scale LLM Knowledge Navigationcited by 1
  37. 2026The Power of Order: Fooling LLMs with Adversarial Table Permutationscited by 1
  38. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction
  2. 2025Positional Biases Shift as Inputs Approach Context Window Limits
  3. 2026Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation
  4. 2024Position Engineering: Boosting Large Language Models through Positional Information Manipulation
  5. 2024Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems
  6. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding
  7. 2026A Better Start for Language Models: Domain-Conditional Position Offsets
  8. 2026The Position Curse: LLMs Struggle to Locate the Last Few Items in a List