# SeA: Semantic Adversarial Augmentation for Last Layer Features from Unsupervised Representation Learning

arXiv: https://arxiv.org/abs/2408.13351 · HTML: https://mlsanity.com/paper/2408.13351/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://mlsanity.com/paper/1909.05235/) (2019)
- [Regularizing Deep Networks with Semantic Data Augmentation](https://mlsanity.com/paper/2007.10538/) (2020)
- [Adversarial Feature Augmentation for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1711.08561/) (2017)
- [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021)
- [Unsupervised Visual Representation Learning by Online Constrained K-Means](https://mlsanity.com/paper/2105.11527/) (2021)
- [Weakly Supervised Representation Learning with Coarse Labels](https://mlsanity.com/paper/2005.09681/) (2020)
- [Adversarial Feature Augmentation and Normalization for Visual Recognition](https://mlsanity.com/paper/2103.12171/) (2021)
- [Hierarchically Robust Representation Learning](https://mlsanity.com/paper/1911.04047/) (2019)

## Abstract (arXiv)

Deep features extracted from certain layers of a pre-trained deep model show superior performance over the conventional hand-crafted features. Compared with fine-tuning or linear probing that can explore diverse augmentations, eg, random crop/flipping, in the original input space, the appropriate augmentations for learning with fixed deep features are more challenging and have been less investigated, which degenerates the performance. To unleash the potential of fixed deep features, we propose a novel semantic adversarial augmentation (SeA) in the feature space for optimization. Concretely, the adversarial direction implied by the gradient will be projected to a subspace spanned by other examples to preserve the semantic information. Then, deep features will be perturbed with the semantic direction, and augmented features will be applied to learn the classifier. Experiments are conducted on 11 benchmark downstream classification tasks with 4 popular pre-trained models. Our method is 2% better than the deep features without SeA on average. Moreover, compared to the expensive fine-tuning that is expected to give good performance, SeA shows a comparable performance on 6 out of 11 tasks, demonstrating the effectiveness of our proposal in addition to its efficiency. Code is available at urlhttps://github.com/idstcv/SeA.