Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation

arXiv 2508.15709 ↗First posted 21 Aug 2025cs.CLcited by 9

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Positional bias (PB), manifesting as non-uniform sensitivity across different contextual locations, significantly impairs long-context comprehension and processing capabilities. Previous studies have addressed PB either by modifying the underlying architectures or by employing extensive contextual awareness training. However, the former approach fails to effectively eliminate the substantial performance disparities, while the latter imposes significant data and computational overhead. To address PB effectively, we introduce Pos2Distill, a position to position knowledge distillation framework. Pos2Distill transfers the superior capabilities from advantageous positions to less favorable ones, thereby reducing the huge performance gaps. The conceptual principle is to leverage the inherent, position-induced disparity to counteract the PB itself. We identify distinct manifestations of PB under textbftextscretrieval and textbftextscreasoning paradigms, thereby designing two specialized instantiations: emphPos2Distill-Rtextsuperscript1 and emphPos2Distill-Rtextsuperscript2 respectively, both grounded in this core principle. By employing the Pos2Distill approach, we achieve enhanced uniformity and significant performance gains across all contextual positions in long-context retrieval and reasoning tasks. Crucially, both specialized systems exhibit strong cross-task generalization mutually, while achieving superior performance on their respective tasks.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 49 of its 63 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

49 of 63 references.

References · 49

2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
2023Qwen Technical Reportcited by 3,344
2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282

37 more

2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
2023Large Language Models are not Fair Evaluatorscited by 1,085
2016Sequence-Level Knowledge Distillationcited by 801
2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
2024Make Your LLM Fully Utilize the Contextcited by 119
2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 72
2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
2024Length Generalization of Causal Transformers without Position Encodingcited by 47
2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 42
2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 36
2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 28
2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
2024Making Long-Context Language Models Better Multi-Hop Reasonerscited by 26
2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 24
2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
2024Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neuronscited by 16
2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
2024I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responsescited by 15
2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QAcited by 13
2024Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMscited by 7
2025Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoningcited by 6
2024PEAR: A Robust and Flexible Automation Framework for Ptychography Enabled by Multiple Large Language Model Agentscited by 5
2025CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capabilitycited by 4
2024PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overheadcited by 1
2025Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexitycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 44
2025SWE-QA: Can Language Models Answer Repository-level Code Questions?cited by 38
2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
2026FASA: Frequency-aware Sparse Attentioncited by 8
2026Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generationcited by 5
2025GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extractioncited by 2
2025Beyond Function-Level Search: Repository-Aware Dual-Encoder Code Retrieval with Adversarial Verificationcited by 1
2026Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarkscited by 1
2026Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Modelscited by 0

Similar, not cited

Close in wording, no citation either way