# Adversarial Masking for Self-Supervised Learning

arXiv: https://arxiv.org/abs/2201.13100 · HTML: https://mlsanity.com/paper/2201.13100/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [Faster AutoAugment: Learning Augmentation Strategies using Backpropagation](https://mlsanity.com/paper/1911.06987/) (2019)
- [Viewmaker Networks: Learning Views for Unsupervised Representation Learning](https://mlsanity.com/paper/2010.07432/) (2020)
- [Augmenting Convolutional networks with attention-based aggregation](https://mlsanity.com/paper/2112.13692/) (2021)

## Built on by (most cited first, 30 of 78)

- [SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders](https://mlsanity.com/paper/2206.10207/) (2022)
- [Masked Frequency Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2206.07706/) (2022)
- [Hard Patches Mining for Masked Image Modeling](https://mlsanity.com/paper/2304.05919/) (2023)
- [Understanding Masked Image Modeling via Learning Occlusion Invariant Feature](https://mlsanity.com/paper/2208.04164/) (2022)
- [Understanding Masked Autoencoders via Hierarchical Latent Variable Models](https://mlsanity.com/paper/2306.04898/) (2023)
- [Mixed Autoencoder for Self-supervised Visual Representation Learning](https://mlsanity.com/paper/2303.17152/) (2023)
- [Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysis](https://mlsanity.com/paper/2302.05615/) (2023)
- [Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works](https://mlsanity.com/paper/2211.08129/) (2022)
- [FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning](https://mlsanity.com/paper/2412.12032/) (2024)
- [CL-MAE: Curriculum-Learned Masked Autoencoders](https://mlsanity.com/paper/2308.16572/) (2023)
- [MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding](https://mlsanity.com/paper/2402.14391/) (2024)
- [Extreme Masking for Learning Instance and Distributed Visual Representations](https://mlsanity.com/paper/2206.04667/) (2022)
- [Improving Masked Autoencoders by Learning Where to Mask](https://mlsanity.com/paper/2303.06583/) (2023)
- [Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning](https://mlsanity.com/paper/2410.19560/) (2024)
- [Swin MAE: Masked Autoencoders for Small Datasets](https://mlsanity.com/paper/2212.13805/) (2022)
- [Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems](https://mlsanity.com/paper/2303.01669/) (2023)
- [Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training](https://mlsanity.com/paper/2404.08327/) (2024)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [Efficient Vision-Language Pre-training by Cluster Masking](https://mlsanity.com/paper/2405.08815/) (2024)
- [Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking](https://mlsanity.com/paper/2303.05475/) (2023)
- [Harnessing Massive Satellite Imagery with Efficient Masked Image Modeling](https://mlsanity.com/paper/2406.11933/) (2024)
- [Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency](https://mlsanity.com/paper/2206.08222/) (2022)
- [Pre-training with Random Orthogonal Projection Image Modeling](https://mlsanity.com/paper/2310.18737/) (2023)
- [Ignorance is Bliss: Robust Control via Information Gating](https://mlsanity.com/paper/2303.06121/) (2023)
- [AnatoMask: Enhancing Medical Image Segmentation with Reconstruction-guided Self-masking](https://mlsanity.com/paper/2407.06468/) (2024)
- [DMT-JEPA: Discriminative Masked Targets for Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2405.17995/) (2024)
- [ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders](https://mlsanity.com/paper/2407.13036/) (2024)
- [SIGMA-PPG: Statistical-prior Informed Generative Masking Architecture for PPG Foundation Model](https://mlsanity.com/paper/2601.21031/) (2026)
- [UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering](https://mlsanity.com/paper/2212.10729/) (2022)
- [GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning](https://mlsanity.com/paper/2212.13425/) (2022)

## Abstract (arXiv)

We propose ADIOS, a masked image model (MIM) framework for self-supervised learning, which simultaneously learns a masking function and an image encoder using an adversarial objective. The image encoder is trained to minimise the distance between representations of the original and that of a masked image. The masking function, conversely, aims at maximising this distance. ADIOS consistently improves on state-of-the-art self-supervised learning (SSL) methods on a variety of tasks and datasets -- including classification on ImageNet100 and STL10, transfer learning on CIFAR10/100, Flowers102 and iNaturalist, as well as robustness evaluated on the backgrounds challenge (Xiao et al., 2021) -- while generating semantically meaningful masks. Unlike modern MIM models such as MAE, BEiT and iBOT, ADIOS does not rely on the image-patch tokenisation construction of Vision Transformers, and can be implemented with convolutional backbones. We further demonstrate that the masks learned by ADIOS are more effective in improving representation learning of SSL methods than masking schemes used in popular MIM models. Code is available at https://github.com/YugeTen/adios.