# Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

arXiv: https://arxiv.org/abs/1907.04380 · HTML: https://mlsanity.com/paper/1907.04380/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2019.

## Built on by (most cited first, 30 of 49)

- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [DataPerf: Benchmarks for Data-Centric AI Development](https://mlsanity.com/paper/2207.10062/) (2022)
- [Distilling Model Failures as Directions in Latent Space](https://mlsanity.com/paper/2206.14754/) (2022)
- [Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?](https://mlsanity.com/paper/2402.12483/) (2024)
- [Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics](https://mlsanity.com/paper/2110.01518/) (2021)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://mlsanity.com/paper/2203.12942/) (2022)
- [Variational Information Bottleneck for Effective Low-Resource Fine-Tuning](https://mlsanity.com/paper/2106.05469/) (2021)
- [An Investigation of the (In)effectiveness of Counterfactually Augmented Data](https://mlsanity.com/paper/2107.00753/) (2021)
- [Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA](https://mlsanity.com/paper/2210.04692/) (2022)
- [DISCO: Distilling Counterfactuals with Large Language Models](https://mlsanity.com/paper/2212.10534/) (2022)
- [Supervising Model Attention with Human Explanations for Robust Natural Language Inference](https://mlsanity.com/paper/2104.08142/) (2021)
- [Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering](https://mlsanity.com/paper/2110.01013/) (2021)
- [Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding](https://mlsanity.com/paper/2109.04947/) (2021)
- [Check It Again: Progressive Visual Question Answering via Visual Entailment](https://mlsanity.com/paper/2106.04605/) (2021)
- [Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models](https://mlsanity.com/paper/2205.11432/) (2022)
- [Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers](https://mlsanity.com/paper/2105.12628/) (2021)
- [Learning to Split for Automatic Bias Detection](https://mlsanity.com/paper/2204.13749/) (2022)
- [A synthetic data approach for domain generalization of NLI models](https://mlsanity.com/paper/2402.12368/) (2024)
- [Mitigating the Position Bias of Transformer Models in Passage Re-Ranking](https://mlsanity.com/paper/2101.06980/) (2021)
- [Feature-Level Debiased Natural Language Understanding](https://mlsanity.com/paper/2212.05421/) (2022)
- [Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning](https://mlsanity.com/paper/2210.04563/) (2022)
- [Behind the Scenes: An Exploration of Trigger Biases Problem in Few-Shot Event Classification](https://mlsanity.com/paper/2108.12844/) (2021)
- [PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outliers](https://mlsanity.com/paper/2112.09237/) (2021)
- [Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation](https://mlsanity.com/paper/2305.13067/) (2023)
- [Learning Stable Classifiers by Transferring Unstable Features](https://mlsanity.com/paper/2106.07847/) (2021)
- [SILT: Efficient transformer training for inter-lingual inference](https://mlsanity.com/paper/2103.09635/) (2021)
- [Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering](https://mlsanity.com/paper/2205.10363/) (2022)
- [Backdoor Learning for NLP: Recent Advances, Challenges, and Future Research Directions](https://mlsanity.com/paper/2302.06801/) (2023)

## Abstract (arXiv)

Natural Language Inference (NLI) datasets often contain hypothesis-only biases---artifacts that allow models to achieve non-trivial performance without learning whether a premise entails a hypothesis. We propose two probabilistic methods to build models that are more robust to such biases and better transfer across datasets. In contrast to standard approaches to NLI, our methods predict the probability of a premise given a hypothesis and NLI label, discouraging models from ignoring the premise. We evaluate our methods on synthetic and existing NLI datasets by training on datasets containing biases and testing on datasets containing no (or different) hypothesis-only biases. Our results indicate that these methods can make NLI models more robust to dataset-specific artifacts, transferring better than a baseline architecture in 9 out of 12 NLI datasets. Additionally, we provide an extensive analysis of the interplay of our methods with known biases in NLI datasets, as well as the effects of encouraging models to ignore biases and fine-tuning on target datasets.