# A Generative Approach for Mitigating Structural Biases in Natural Language Inference

arXiv: https://arxiv.org/abs/2108.14006 · HTML: https://mlsanity.com/paper/2108.14006/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [Towards Debiasing Fact Verification Models](https://mlsanity.com/paper/1908.05267/) (2019)
- [Deep Learning for Sentiment Analysis : A Survey](https://mlsanity.com/paper/1801.07883/) (2018)
- [End-to-End Bias Mitigation by Modelling Biases in Corpora](https://mlsanity.com/paper/1909.06321/) (2019)
- [Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://mlsanity.com/paper/1908.10763/) (2019)
- [Towards Debiasing NLU Models from Unknown Biases](https://mlsanity.com/paper/2009.12303/) (2020)
- [The Fact Extraction and VERification (FEVER) Shared Task](https://mlsanity.com/paper/1811.10971/) (2018)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://mlsanity.com/paper/1808.04926/) (2018)
- [Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance](https://mlsanity.com/paper/2005.00315/) (2020)
- [Learning from others' mistakes: Avoiding dataset biases without modeling them](https://mlsanity.com/paper/2012.01300/) (2020)

## Built on by (most cited first, 3 of 3)

- [Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models](https://mlsanity.com/paper/2305.06841/) (2023)
- [Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patterns](https://mlsanity.com/paper/2201.07614/) (2022)
- [Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference](https://mlsanity.com/paper/2211.03862/) (2022)

## Abstract (arXiv)

Many natural language inference (NLI) datasets contain biases that allow models to perform well by only using a biased subset of the input, without considering the remainder features. For instance, models are able to make a classification decision by only using the hypothesis, without learning the true relationship between it and the premise. These structural biases lead discriminative models to learn unintended superficial features and to generalize poorly out of the training distribution. In this work, we reformulate the NLI task as a generative task, where a model is conditioned on the biased subset of the input and the label and generates the remaining subset of the input. We show that by imposing a uniform prior, we obtain a provably unbiased model. Through synthetic experiments, we find that this approach is highly robust to large amounts of bias. We then demonstrate empirically on two types of natural bias that this approach leads to fully unbiased models in practice. However, we find that generative models are difficult to train and they generally perform worse than discriminative baselines. We highlight the difficulty of the generative modeling task in the context of NLI as a cause for this worse performance. Finally, by fine-tuning the generative model with a discriminative objective, we reduce the performance gap between the generative model and the discriminative baseline, while allowing for a small amount of bias.