# What If We Simply Swap the Two Text Fragments? A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks

arXiv: https://arxiv.org/abs/1809.02719 · HTML: https://mlsanity.com/paper/1809.02719/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2018.

## Built on by (most cited first, 14 of 14)

- [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://mlsanity.com/paper/2205.11822/) (2022)
- [Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://mlsanity.com/paper/2203.12942/) (2022)
- [Consistency Analysis of ChatGPT](https://mlsanity.com/paper/2303.06273/) (2023)
- [DISCO: Distilling Counterfactuals with Large Language Models](https://mlsanity.com/paper/2212.10534/) (2022)
- [Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses](https://mlsanity.com/paper/2305.11662/) (2023)
- [Logical Consistency of Large Language Models in Fact-checking](https://mlsanity.com/paper/2412.16100/) (2024)
- [From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency](https://mlsanity.com/paper/2404.12145/) (2024)
- [NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance](https://mlsanity.com/paper/2104.04751/) (2021)
- [Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary](https://mlsanity.com/paper/2310.15541/) (2023)
- [SMoA: Sparse Mixture of Adapters to Mitigate Multiple Dataset Biases](https://mlsanity.com/paper/2302.14413/) (2023)
- [Unsupervised Pre-training with Structured Knowledge for Improving Natural Language Inference](https://mlsanity.com/paper/2109.03941/) (2021)
- [Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference](https://mlsanity.com/paper/2101.07397/) (2021)
- [MRCLens: an MRC Dataset Bias Detection Toolkit](https://mlsanity.com/paper/2207.08943/) (2022)
- [Reverse-engineering NLI: A study of the meta-inferential properties of Natural Language Inference](https://mlsanity.com/paper/2601.05170/) (2026)

## Abstract (arXiv)

Nature language inference (NLI) task is a predictive task of determining the inference relationship of a pair of natural language sentences. With the increasing popularity of NLI, many state-of-the-art predictive models have been proposed with impressive performances. However, several works have noticed the statistical irregularities in the collected NLI data set that may result in an over-estimated performance of these models and proposed remedies. In this paper, we further investigate the statistical irregularities, what we refer as confounding factors, of the NLI data sets. With the belief that some NLI labels should preserve under swapping operations, we propose a simple yet effective way (swapping the two text fragments) of evaluating the NLI predictive models that naturally mitigate the observed problems. Further, we continue to train the predictive models with our swapping manner and propose to use the deviation of the model's evaluation performances under different percentages of training text fragments to be swapped to describe the robustness of a predictive model. Our evaluation metrics leads to some interesting understandings of recent published NLI methods. Finally, we also apply the swapping operation on NLI models to see the effectiveness of this straightforward method in mitigating the confounding factor problems in training generic sentence embeddings for other NLP transfer tasks.