# An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference

arXiv: https://arxiv.org/abs/2010.03777 · HTML: https://mlsanity.com/paper/2010.03777/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- First posted on arXiv 8 Oct 2020.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Built on by (most cited first, 16 of 16)

- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://mlsanity.com/paper/2203.12942/) (2022)
- [DISCO: Distilling Counterfactuals with Large Language Models](https://mlsanity.com/paper/2212.10534/) (2022)
- [Supervising Model Attention with Human Explanations for Robust Natural Language Inference](https://mlsanity.com/paper/2104.08142/) (2021)
- [A synthetic data approach for domain generalization of NLI models](https://mlsanity.com/paper/2402.12368/) (2024)
- [AGRO: Adversarial Discovery of Error-prone groups for Robust Optimization](https://mlsanity.com/paper/2212.00921/) (2022)
- [Behind the Scenes: An Exploration of Trigger Biases Problem in Few-Shot Event Classification](https://mlsanity.com/paper/2108.12844/) (2021)
- [DialogVCS: Robust Natural Language Understanding in Dialogue System Upgrade](https://mlsanity.com/paper/2305.14751/) (2023)
- [Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection](https://mlsanity.com/paper/2505.20209/) (2025)
- [Explanation based Bias Decoupling Regularization for Natural Language Inference](https://mlsanity.com/paper/2404.13390/) (2024)
- [SMoA: Sparse Mixture of Adapters to Mitigate Multiple Dataset Biases](https://mlsanity.com/paper/2302.14413/) (2023)
- [Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization](https://mlsanity.com/paper/2305.12123/) (2023)
- [IBADR: an Iterative Bias-Aware Dataset Refinement Framework for Debiasing NLU models](https://mlsanity.com/paper/2311.00292/) (2023)
- [Using Natural Language Explanations to Improve Robustness of In-context Learning](https://mlsanity.com/paper/2311.07556/) (2023)
- [Exploring Explanations Improves the Robustness of In-Context Learning](https://mlsanity.com/paper/2506.02378/) (2025)
- [Evaluating the Scalability and Adversarial Generalization of GRPO-Trained NLI Models](https://mlsanity.com/paper/2504.18376/) (2025)

## Abstract (arXiv)

The prior work on natural language inference (NLI) debiasing mainly targets at one or few known biases while not necessarily making the models more robust. In this paper, we focus on the model-agnostic debiasing strategies and explore how to (or is it possible to) make the NLI models robust to multiple distinct adversarial attacks while keeping or even strengthening the models' generalization power. We firstly benchmark prevailing neural NLI models including pretrained ones on various adversarial datasets. We then try to combat distinct known biases by modifying a mixture of experts (MoE) ensemble method and show that it's nontrivial to mitigate multiple NLI biases at the same time, and that model-level ensemble method outperforms MoE ensemble method. We also perform data augmentation including text swap, word substitution and paraphrase and prove its efficiency in combating various (though not all) adversarial attacks at the same time. Finally, we investigate several methods to merge heterogeneous training data (1.35M) and perform model ensembling, which are straightforward but effective to strengthen NLI models.