# Learning with Instance Bundles for Reading Comprehension

arXiv: https://arxiv.org/abs/2104.08735 · HTML: https://mlsanity.com/paper/2104.08735/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 30 of 31)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Explaining NLP Models via Minimal Contrastive Editing (MiCE)](https://mlsanity.com/paper/2012.13985/) (2020)
- [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](https://mlsanity.com/paper/1908.05803/) (2019)
- [Reasoning Over Paragraph Effects in Situations](https://mlsanity.com/paper/1908.05852/) (2019)
- [Logic-Guided Data Augmentation and Regularization for Consistent Question Answering](https://mlsanity.com/paper/2004.10157/) (2020)
- [Contrastive Explanations for Model Interpretability](https://mlsanity.com/paper/2103.01378/) (2021)
- [Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision](https://mlsanity.com/paper/2004.09034/) (2020)
- [Explaining The Efficacy of Counterfactually Augmented Data](https://mlsanity.com/paper/2010.02114/) (2020)
- [A Logic-Driven Framework for Consistency of Neural Models](https://mlsanity.com/paper/1909.00126/) (2019)
- [Contrastive Learning for Image Captioning](https://mlsanity.com/paper/1710.02534/) (2017)
- [Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge](https://mlsanity.com/paper/1808.08609/) (2018)
- [Group-wise Contrastive Learning for Neural Dialogue Generation](https://mlsanity.com/paper/2009.07543/) (2020)
- [An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search](https://mlsanity.com/paper/1904.06834/) (2019)
- [QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization](https://mlsanity.com/paper/1909.00215/) (2019)
- [RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering](https://mlsanity.com/paper/2010.10757/) (2020)
- [Multi-Step Inference for Reasoning Over Paragraphs](https://mlsanity.com/paper/2004.02995/) (2020)
- [An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models](https://mlsanity.com/paper/2004.02451/) (2020)
- [Cascaded Text Generation with Markov Transformers](https://mlsanity.com/paper/2006.01112/) (2020)
- [Paired Examples as Indirect Supervision in Latent Decision Models](https://mlsanity.com/paper/2104.01759/) (2021)

## Built on by (most cited first, 8 of 8)

- [Successive Prompting for Decomposing Complex Questions](https://mlsanity.com/paper/2212.04092/) (2022)
- [Retrieval-guided Counterfactual Generation for QA](https://mlsanity.com/paper/2110.07596/) (2021)
- [CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation](https://mlsanity.com/paper/2210.04873/) (2022)
- [Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition](https://mlsanity.com/paper/2107.13935/) (2021)
- [Hallucination Augmented Recitations for Language Models](https://mlsanity.com/paper/2311.07424/) (2023)
- [Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformation](https://mlsanity.com/paper/2210.12902/) (2022)
- [How Much Consistency Is Your Accuracy Worth?](https://mlsanity.com/paper/2310.13781/) (2023)
- [Improving QA Model Performance with Cartographic Inoculation](https://mlsanity.com/paper/2401.17498/) (2024)

## Abstract (arXiv)

When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other. However, closely related questions and their corresponding answers are not independent, and leveraging these relationships could provide a strong supervision signal to a model. Drawing on ideas from contrastive estimation, we introduce several new supervision techniques that compare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers, adding another cross entropy loss term that is used in addition to traditional maximum likelihood estimation. Our techniques require bundles of related question-answer pairs, which we can either mine from within existing data or create using various automated heuristics. We empirically demonstrate the effectiveness of training with instance bundles on two datasets -- HotpotQA and ROPES -- showing up to 11% absolute gains in accuracy.