# Which Shortcut Solution Do Question Answering Models Prefer to Learn?

arXiv: https://arxiv.org/abs/2211.16220 · HTML: https://mlsanity.com/paper/2211.16220/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://mlsanity.com/paper/2002.04326/) (2020)
- [Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI](https://mlsanity.com/paper/2010.07487/) (2020)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Towards Debiasing NLU Models from Unknown Biases](https://mlsanity.com/paper/2009.12303/) (2020)
- [Adversarial Training for Large Neural Language Models](https://mlsanity.com/paper/2004.08994/) (2020)
- [InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective](https://mlsanity.com/paper/2010.02329/) (2020)
- [Competency Problems: On Finding and Removing Artifacts in Language Data](https://mlsanity.com/paper/2104.08646/) (2021)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA](https://mlsanity.com/paper/1906.07132/) (2019)
- [What Makes Reading Comprehension Questions Easier?](https://mlsanity.com/paper/1808.09384/) (2018)
- [The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks](https://mlsanity.com/paper/2006.14599/) (2020)
- [Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective](https://mlsanity.com/paper/2110.03095/) (2021)
- [Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets](https://mlsanity.com/paper/1911.09241/) (2019)
- [Why Machine Reading Comprehension Models Learn Shortcuts?](https://mlsanity.com/paper/2106.01024/) (2021)
- [Making Neural QA as Simple as Possible but not Simpler](https://mlsanity.com/paper/1703.04816/) (2017)
- [Semi-Supervised QA with Generative Domain-Adaptive Nets](https://mlsanity.com/paper/1702.02206/) (2017)
- [Rissanen Data Analysis: Examining Dataset Characteristics via Description Length](https://mlsanity.com/paper/2103.03872/) (2021)
- [Improving QA Generalization by Concurrent Modeling of Multiple Biases](https://mlsanity.com/paper/2010.03338/) (2020)

## Built on by (most cited first, 9 of 9)

- [Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations](https://mlsanity.com/paper/2305.13299/) (2023)
- [Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapse](https://mlsanity.com/paper/2405.05587/) (2024)
- [Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggers](https://mlsanity.com/paper/2310.18360/) (2023)
- [ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations](https://mlsanity.com/paper/2308.10103/) (2023)
- [Mitigating Shortcut Learning with InterpoLated Learning](https://mlsanity.com/paper/2507.05527/) (2025)
- [Single-Sentence Reader: A Novel Approach for Addressing Answer Position Bias](https://mlsanity.com/paper/2308.04566/) (2023)
- [Profiling German Text Simplification with Interpretable Model-Fingerprints](https://mlsanity.com/paper/2601.13050/) (2026)
- [Debiasing Reward Models via Causally Motivated Inference-Time Intervention](https://mlsanity.com/paper/2604.27495/) (2026)
- [Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models](https://mlsanity.com/paper/2609.17112/) (2026)

## Abstract (arXiv)

Question answering (QA) models for reading comprehension tend to learn shortcut solutions rather than the solutions intended by QA datasets. QA models that have learned shortcut solutions can achieve human-level performance in shortcut examples where shortcuts are valid, but these same behaviors degrade generalization potential on anti-shortcut examples where shortcuts are invalid. Various methods have been proposed to mitigate this problem, but they do not fully take the characteristics of shortcuts themselves into account. We assume that the learnability of shortcuts, i.e., how easy it is to learn a shortcut, is useful to mitigate the problem. Thus, we first examine the learnability of the representative shortcuts on extractive and multiple-choice QA datasets. Behavioral tests using biased training sets reveal that shortcuts that exploit answer positions and word-label correlations are preferentially learned for extractive and multiple-choice QA, respectively. We find that the more learnable a shortcut is, the flatter and deeper the loss landscape is around the shortcut solution in the parameter space. We also find that the availability of the preferred shortcuts tends to make the task easier to perform from an information-theoretic viewpoint. Lastly, we experimentally show that the learnability of shortcuts can be utilized to construct an effective QA training set; the more learnable a shortcut is, the smaller the proportion of anti-shortcut examples required to achieve comparable performance on shortcut and anti-shortcut examples. We claim that the learnability of shortcuts should be considered when designing mitigation methods.