# Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering

arXiv: https://arxiv.org/abs/2210.14541 · HTML: https://mlsanity.com/paper/2210.14541/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 19 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://mlsanity.com/paper/1908.10763/) (2019)
- [Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance](https://mlsanity.com/paper/2005.00315/) (2020)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [What Makes Reading Comprehension Questions Easier?](https://mlsanity.com/paper/1808.09384/) (2018)
- [Making Neural QA as Simple as Possible but not Simpler](https://mlsanity.com/paper/1703.04816/) (2017)
- [Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap](https://mlsanity.com/paper/2109.11256/) (2021)

## Built on by (most cited first, 7 of 7)

- [Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggers](https://mlsanity.com/paper/2310.18360/) (2023)
- [A Frustratingly Easy Improvement for Position Embeddings via Random Padding](https://mlsanity.com/paper/2305.04859/) (2023)
- [Principles from Clinical Research for NLP Model Generalization](https://mlsanity.com/paper/2311.03663/) (2023)
- [Adaptive loose optimization for robust question answering](https://mlsanity.com/paper/2305.03971/) (2023)
- [HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark](https://mlsanity.com/paper/2508.01812/) (2025)
- [Self-Supervised Position Debiasing for Large Language Models](https://mlsanity.com/paper/2401.01218/) (2024)
- [Debiasing Reward Models via Causally Motivated Inference-Time Intervention](https://mlsanity.com/paper/2604.27495/) (2026)

## Abstract (arXiv)

Extractive question answering (QA) models tend to exploit spurious correlations to make predictions when a training set has unintended biases. This tendency results in models not being generalizable to examples where the correlations do not hold. Determining the spurious correlations QA models can exploit is crucial in building generalizable QA models in real-world applications; moreover, a method needs to be developed that prevents these models from learning the spurious correlations even when a training set is biased. In this study, we discovered that the relative position of an answer, which is defined as the relative distance from an answer span to the closest question-context overlap word, can be exploited by QA models as superficial cues for making predictions. Specifically, we find that when the relative positions in a training set are biased, the performance on examples with relative positions unseen during training is significantly degraded. To mitigate the performance degradation for unseen relative positions, we propose an ensemble-based debiasing method that does not require prior knowledge about the distribution of relative positions. We demonstrate that the proposed method mitigates the models' reliance on relative positions using the biased and full SQuAD dataset. We hope that this study can help enhance the generalization ability of QA models in real-world applications.