# SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization

arXiv: https://arxiv.org/abs/2006.01791 · HTML: https://mlsanity.com/paper/2006.01791/

## Key facts

- Cited by 149 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2020.

## Built on by (most cited first, 30 of 149)

- [Mix3D: Out-of-Context Data Augmentation for 3D Scenes](https://mlsanity.com/paper/2110.02210/) (2021)
- [TransMix: Attend to Mix for Vision Transformers](https://mlsanity.com/paper/2111.09833/) (2021)
- [DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models](https://mlsanity.com/paper/2405.14881/) (2024)
- [AutoMix: Unveiling the Power of Mixup for Stronger Classifiers](https://mlsanity.com/paper/2103.13027/) (2021)
- [MixGen: A New Multi-Modal Data Augmentation](https://mlsanity.com/paper/2206.08358/) (2022)
- [C-Mixup: Improving Generalization in Regression](https://mlsanity.com/paper/2210.05775/) (2022)
- [Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspective](https://mlsanity.com/paper/2303.13434/) (2023)
- [TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers](https://mlsanity.com/paper/2207.08409/) (2022)
- [Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module](https://mlsanity.com/paper/2209.02604/) (2022)
- [SSMix: Saliency-Based Span Mixup for Text Classification](https://mlsanity.com/paper/2106.08062/) (2021)
- [Object-aware Contrastive Learning for Debiased Scene Representation](https://mlsanity.com/paper/2108.00049/) (2021)
- [ExcelFormer: A neural network surpassing GBDTs on tabular data](https://mlsanity.com/paper/2301.02819/) (2023)
- [AlignMixup: Improving Representations By Interpolating Aligned Features](https://mlsanity.com/paper/2103.15375/) (2021)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [A Simple Data Mixing Prior for Improving Self-Supervised Learning](https://mlsanity.com/paper/2206.07692/) (2022)
- [A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective](https://mlsanity.com/paper/2208.09913/) (2022)
- [Survey: Image Mixing and Deleting for Data Augmentation](https://mlsanity.com/paper/2106.07085/) (2021)
- [TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers](https://mlsanity.com/paper/2210.07562/) (2022)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [Subclass-balancing Contrastive Learning for Long-tailed Recognition](https://mlsanity.com/paper/2306.15925/) (2023)
- [Prompt Learning via Meta-Regularization](https://mlsanity.com/paper/2404.00851/) (2024)
- [It Takes Two to Tango: Mixup for Deep Metric Learning](https://mlsanity.com/paper/2106.04990/) (2021)
- [Image Data Augmentation Approaches: A Comprehensive Survey and Future directions](https://mlsanity.com/paper/2301.02830/) (2023)
- [GuidedMixup: An Efficient Mixup Strategy Guided by Saliency Maps](https://mlsanity.com/paper/2306.16612/) (2023)
- [Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup](https://mlsanity.com/paper/2111.15454/) (2021)
- [ScoreNet: Learning Non-Uniform Attention and Augmentation for Transformer-Based Histopathological Image Classification](https://mlsanity.com/paper/2202.07570/) (2022)
- [OpenMixup: Open Mixup Toolbox and Benchmark for Visual Representation Learning](https://mlsanity.com/paper/2209.04851/) (2022)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [RankMixup: Ranking-Based Mixup Training for Network Calibration](https://mlsanity.com/paper/2308.11990/) (2023)

## Abstract (arXiv)

Advanced data augmentation strategies have widely been studied to improve the generalization ability of deep learning models. Regional dropout is one of the popular solutions that guides the model to focus on less discriminative parts by randomly removing image regions, resulting in improved regularization. However, such information removal is undesirable. On the other hand, recent strategies suggest to randomly cut and mix patches and their labels among training images, to enjoy the advantages of regional dropout without having any pointless pixel in the augmented images. We argue that such random selection strategies of the patches may not necessarily represent sufficient information about the corresponding object and thereby mixing the labels according to that uninformative patch enables the model to learn unexpected feature representation. Therefore, we propose SaliencyMix that carefully selects a representative image patch with the help of a saliency map and mixes this indicative patch with the target image, thus leading the model to learn more appropriate feature representation. SaliencyMix achieves the best known top-1 error of 21.26% and 20.09% for ResNet-50 and ResNet-101 architectures on ImageNet classification, respectively, and also improves the model robustness against adversarial perturbations. Furthermore, models that are trained with SaliencyMix help to improve the object detection performance. Source code is available at https://github.com/SaliencyMix/SaliencyMix.