mlsanityData to 7 Oct 2026

A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension

arXiv 2209.01824 ↗First posted 5 Sep 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The issue of shortcut learning is widely known in NLP and has been an important research focus in recent years. Unintended correlations in the data enable models to easily solve tasks that were meant to exhibit advanced language understanding and reasoning capabilities. In this survey paper, we focus on the field of machine reading comprehension (MRC), an important task for showcasing high-level language understanding that also suffers from a range of shortcuts. We summarize the available techniques for measuring and mitigating shortcuts and conclude with suggestions for further progress in shortcut research. Importantly, we highlight two concerns for shortcut mitigation in MRC: (1) the lack of public challenge sets, a necessary component for effective and reusable evaluation, and (2) the lack of certain mitigation techniques that are prominent in other areas.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 89 of its 128 references that are in the map. First posted on arXiv 5 Sep 2022.

Builds on

89 of 128 references.

References · 89

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
38 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,039
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  3. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  8. 2018CoQA: A Conversational Question Answering Challengecited by 674
  9. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  10. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  11. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  12. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  13. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  14. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  15. 2018QuAC : Question Answering in Contextcited by 379
  16. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  17. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  18. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  19. 2016Rationalizing Neural Predictionscited by 281
  20. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  21. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  22. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  23. 2020Adversarial Filters of Dataset Biasescited by 159
  24. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  25. 2021Explaining Answers with Entailment Treescited by 148
  26. 2019Towards Debiasing Fact Verification Modelscited by 143
  27. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  28. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  29. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  30. 2019Multi-hop Reading Comprehension through Question Decomposition and Rescoringcited by 119
  31. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  32. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  33. 2020Adversarial Training for Large Neural Language Modelscited by 109
  34. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  35. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  36. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  37. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
  38. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024Cofca: A Step-Wise Counterfactual Multi-hop QA benchmarkcited by 26
  2. 2024Navigating Shortcuts, Spurious Correlations, and Confounders: From Origins via Detection to Mitigationcited by 22
  3. 2023KoRC: Knowledge oriented Reading Comprehension Benchmark for Deep Text Understandingcited by 7
  4. 2025Detecting and Pruning Prominent but Detrimental Neurons in Large Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach
  2. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?
  3. 2025Read Before You Think: Mitigating LLM Comprehension Failures with Step-by-Step Reading
  4. 2024Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models
  5. 2024MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models
  6. 2023NLPBench: Evaluating Large Language Models on Solving NLP Problems
  7. 2024MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs
  8. 2023MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models