# BiaSwap: Removing dataset bias with bias-tailored swapping augmentation

arXiv: https://arxiv.org/abs/2108.10008 · HTML: https://mlsanity.com/paper/2108.10008/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 23 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Swapping Autoencoder for Deep Image Manipulation](https://mlsanity.com/paper/2007.00653/) (2020)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [REPAIR: Removing Representation Bias by Dataset Resampling](https://mlsanity.com/paper/1904.07911/) (2019)
- [Learning from Failure: Training Debiased Classifier from Biased Classifier](https://mlsanity.com/paper/2007.02561/) (2020)
- [Learning Robust Representations by Projecting Superficial Statistics Out](https://mlsanity.com/paper/1903.06256/) (2019)
- [Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing](https://mlsanity.com/paper/1912.07538/) (2019)
- [Cycle-Consistency for Robust Visual Question Answering](https://mlsanity.com/paper/1902.05660/) (2019)
- [Model Patching: Closing the Subgroup Performance Gap with Data Augmentation](https://mlsanity.com/paper/2008.06775/) (2020)
- [Not Using the Car to See the Sidewalk: Quantifying and Controlling the Effects of Context in Classification and Segmentation](https://mlsanity.com/paper/1812.06707/) (2018)
- [Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation](https://mlsanity.com/paper/1909.04696/) (2019)
- [Latent Adversarial Debiasing: Mitigating Collider Bias in Deep Neural Networks](https://mlsanity.com/paper/2011.11486/) (2020)

## Built on by (most cited first, 30 of 70)

- [A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others](https://mlsanity.com/paper/2212.04825/) (2022)
- [Towards Last-layer Retraining for Group Robustness with Fewer Annotations](https://mlsanity.com/paper/2309.08534/) (2023)
- [Learning Debiased Classifier with Biased Committee](https://mlsanity.com/paper/2206.10843/) (2022)
- [Discover and Mitigate Unknown Biases with Debiasing Alternate Networks](https://mlsanity.com/paper/2207.10077/) (2022)
- [Avoiding spurious correlations via logit correction](https://mlsanity.com/paper/2212.01433/) (2022)
- [Learning Fair Representation via Distributional Contrastive Disentanglement](https://mlsanity.com/paper/2206.08743/) (2022)
- [SelecMix: Debiased Learning by Contradicting-pair Sampling](https://mlsanity.com/paper/2211.02291/) (2022)
- [Overcoming Simplicity Bias in Deep Networks using a Feature Sieve](https://mlsanity.com/paper/2301.13293/) (2023)
- [Mitigating Dataset Bias by Using Per-sample Gradient](https://mlsanity.com/paper/2205.15704/) (2022)
- [On Modality Bias Recognition and Reduction](https://mlsanity.com/paper/2202.12690/) (2022)
- [Revisiting the Importance of Amplifying Bias for Debiasing](https://mlsanity.com/paper/2205.14594/) (2022)
- [Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation](https://mlsanity.com/paper/2402.18919/) (2024)
- [Training Debiased Subnetworks with Contrastive Weight Pruning](https://mlsanity.com/paper/2210.05247/) (2022)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlations](https://mlsanity.com/paper/2403.03375/) (2024)
- [OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses](https://mlsanity.com/paper/2204.02426/) (2022)
- [Weakly Supervised Open-Vocabulary Object Detection](https://mlsanity.com/paper/2312.12437/) (2023)
- [Unsupervised Concept Discovery Mitigates Spurious Correlations](https://mlsanity.com/paper/2402.13368/) (2024)
- [CAD-VAE: Leveraging Correlation-Aware Latents for Comprehensive Fair Disentanglement](https://mlsanity.com/paper/2503.07938/) (2025)
- [Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement](https://mlsanity.com/paper/2311.15303/) (2023)
- [Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation](https://mlsanity.com/paper/2112.01021/) (2021)
- [Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness](https://mlsanity.com/paper/2503.09487/) (2025)
- [Combating Unknown Bias with Effective Bias-Conflicting Scoring and Gradient Alignment](https://mlsanity.com/paper/2111.13108/) (2021)
- [Self-supervised debiasing using low rank regularization](https://mlsanity.com/paper/2210.05248/) (2022)
- [General Greedy De-bias Learning](https://mlsanity.com/paper/2112.10572/) (2021)
- [Fairness-aware Vision Transformer via Debiased Self-Attention](https://mlsanity.com/paper/2301.13803/) (2023)
- [DeNetDM: Debiasing by Network Depth Modulation](https://mlsanity.com/paper/2403.19863/) (2024)
- [Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization](https://mlsanity.com/paper/2409.03303/) (2024)
- [The Box Size Confidence Bias Harms Your Object Detector](https://mlsanity.com/paper/2112.01901/) (2021)
- [Using Positive Matching Contrastive Loss with Facial Action Units to mitigate bias in Facial Expression Recognition](https://mlsanity.com/paper/2303.04896/) (2023)

## Abstract (arXiv)

Deep neural networks often make decisions based on the spurious correlations inherent in the dataset, failing to generalize in an unbiased data distribution. Although previous approaches pre-define the type of dataset bias to prevent the network from learning it, recognizing the bias type in the real dataset is often prohibitive. This paper proposes a novel bias-tailored augmentation-based approach, BiaSwap, for learning debiased representation without requiring supervision on the bias type. Assuming that the bias corresponds to the easy-to-learn attributes, we sort the training images based on how much a biased classifier can exploits them as shortcut and divide them into bias-guiding and bias-contrary samples in an unsupervised manner. Afterwards, we integrate the style-transferring module of the image translation model with the class activation maps of such biased classifier, which enables to primarily transfer the bias attributes learned by the classifier. Therefore, given the pair of bias-guiding and bias-contrary, BiaSwap generates the bias-swapped image which contains the bias attributes from the bias-contrary images, while preserving bias-irrelevant ones in the bias-guiding images. Given such augmented images, BiaSwap demonstrates the superiority in debiasing against the existing baselines over both synthetic and real-world datasets. Even without careful supervision on the bias, BiaSwap achieves a remarkable performance on both unbiased and bias-guiding samples, implying the improved generalization capability of the model.