# Decorrelate Irrelevant, Purify Relevant: Overcome Textual Spurious Correlations from a Feature Perspective

arXiv: https://arxiv.org/abs/2202.08048 · HTML: https://mlsanity.com/paper/2202.08048/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Deep Learning: A Critical Appraisal](https://mlsanity.com/paper/1801.00631/) (2018)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [Deep Stable Learning for Out-Of-Distribution Generalization](https://mlsanity.com/paper/2104.07876/) (2021)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [Towards Debiasing Fact Verification Models](https://mlsanity.com/paper/1908.05267/) (2019)
- [Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://mlsanity.com/paper/1908.10763/) (2019)
- [Towards Debiasing NLU Models from Unknown Biases](https://mlsanity.com/paper/2009.12303/) (2020)
- [The Fact Extraction and VERification (FEVER) Shared Task](https://mlsanity.com/paper/1811.10971/) (2018)
- [InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective](https://mlsanity.com/paper/2010.02329/) (2020)
- [Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance](https://mlsanity.com/paper/2005.00315/) (2020)
- [Learning from others' mistakes: Avoiding dataset biases without modeling them](https://mlsanity.com/paper/2012.01300/) (2020)
- [Stable Learning via Sample Reweighting](https://mlsanity.com/paper/1911.12580/) (2019)
- [Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles](https://mlsanity.com/paper/2011.03856/) (2020)
- [Analyzing Compositionality-Sensitivity of NLI Models](https://mlsanity.com/paper/1811.07033/) (2018)
- [Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms](https://mlsanity.com/paper/1610.05683/) (2016)
- [Uncertainty Calibration for Ensemble-Based Debiasing Methods](https://mlsanity.com/paper/2111.04104/) (2021)
- [Deep Networks with Adaptive Nystr"om Approximation](https://mlsanity.com/paper/1911.13036/) (2019)

## Built on by (most cited first, 4 of 4)

- [Kernel-Whitening: Overcome Dataset Bias with Isotropic Sentence Embedding](https://mlsanity.com/paper/2210.07547/) (2022)
- [Distribution Shift Aware Neural Tabular Learning](https://mlsanity.com/paper/2508.19486/) (2025)
- [Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understanding](https://mlsanity.com/paper/2406.12060/) (2024)
- [Rule Learning for Knowledge Graph Reasoning under Agnostic Distribution Shift](https://mlsanity.com/paper/2507.05110/) (2025)

## Abstract (arXiv)

Natural language understanding (NLU) models tend to rely on spurious correlations (i.e., dataset bias) to achieve high performance on in-distribution datasets but poor performance on out-of-distribution ones. Most of the existing debiasing methods often identify and weaken these samples with biased features (i.e., superficial surface features that cause such spurious correlations). However, down-weighting these samples obstructs the model in learning from the non-biased parts of these samples. To tackle this challenge, in this paper, we propose to eliminate spurious correlations in a fine-grained manner from a feature space perspective. Specifically, we introduce Random Fourier Features and weighted re-sampling to decorrelate the dependencies between features to mitigate spurious correlations. After obtaining decorrelated features, we further design a mutual-information-based method to purify them, which forces the model to learn features that are more relevant to tasks. Extensive experiments on two well-studied NLU tasks demonstrate that our method is superior to other comparative approaches.