# Use the Detection Transformer as a Data Augmenter

arXiv: https://arxiv.org/abs/2304.04554 · HTML: https://mlsanity.com/paper/2304.04554/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization](https://mlsanity.com/paper/2006.01791/) (2020)
- [Learning Data Augmentation Strategies for Object Detection](https://mlsanity.com/paper/1906.11172/) (2019)
- [Data augmentation using learned transformations for one-shot medical image segmentation](https://mlsanity.com/paper/1902.09383/) (2019)
- [Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness](https://mlsanity.com/paper/2010.08001/) (2020)
- [SnapMix: Semantically Proportional Mixing for Augmenting Fine-grained Data](https://mlsanity.com/paper/2012.04846/) (2020)
- [KeepAugment: A Simple Information-Preserving Data Augmentation Approach](https://mlsanity.com/paper/2011.11778/) (2020)
- [Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation](https://mlsanity.com/paper/1805.09707/) (2018)
- [HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization](https://mlsanity.com/paper/2106.00149/) (2021)
- [Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder](https://mlsanity.com/paper/2105.00026/) (2021)
- [Semi-supervised Task-driven Data Augmentation for Medical Image Segmentation](https://mlsanity.com/paper/2007.05363/) (2020)
- [Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifiers](https://mlsanity.com/paper/2103.14453/) (2021)
- [Automatic Data Augmentation for 3D Medical Image Segmentation](https://mlsanity.com/paper/2010.11695/) (2020)
- [DADA: Deep Adversarial Data Augmentation for Extremely Low Data Regime Classification](https://mlsanity.com/paper/1809.00981/) (2018)

## Abstract (arXiv)

Detection Transformer (DETR) is a Transformer architecture based object detection model. In this paper, we demonstrate that it can also be used as a data augmenter. We term our approach as DETR assisted CutMix, or DeMix for short. DeMix builds on CutMix, a simple yet highly effective data augmentation technique that has gained popularity in recent years. CutMix improves model performance by cutting and pasting a patch from one image onto another, yielding a new image. The corresponding label for this new example is specified as the weighted average of the original labels, where the weight is proportional to the area of the patch. CutMix selects a random patch to be cut. In contrast, DeMix elaborately selects a semantically rich patch, located by a pre-trained DETR. The label of the new image is specified in the same way as in CutMix. Experimental results on benchmark datasets for image classification demonstrate that DeMix significantly outperforms prior art data augmentation methods including CutMix. Oue code is available at https://github.com/ZJLAB-AMMI/DeMix.