# Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domains

arXiv: https://arxiv.org/abs/2205.04771 · HTML: https://mlsanity.com/paper/2205.04771/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 10 May 2022.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [A Fourier-based Framework for Domain Generalization](https://mlsanity.com/paper/2105.11120/) (2021)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Episodic Training for Domain Generalization](https://mlsanity.com/paper/1902.00113/) (2019)
- [Deep Domain-Adversarial Image Generation for Domain Generalisation](https://mlsanity.com/paper/2003.06054/) (2020)
- [How Well Do Self-Supervised Models Transfer?](https://mlsanity.com/paper/2011.13377/) (2020)

## Built on by (most cited first, 10 of 10)

- [A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond](https://mlsanity.com/paper/2208.00173/) (2022)
- [Domain-Generalizable Multiple-Domain Clustering](https://mlsanity.com/paper/2301.13530/) (2023)
- [Towards domain-invariant Self-Supervised Learning with Batch Styles Standardization](https://mlsanity.com/paper/2303.06088/) (2023)
- [DPStyler: Dynamic PromptStyler for Source-Free Domain Generalization](https://mlsanity.com/paper/2403.16697/) (2024)
- [Disentangling Masked Autoencoders for Unsupervised Domain Generalization](https://mlsanity.com/paper/2407.07544/) (2024)
- [EnfoMax: Domain Entropy and Mutual Information Maximization for Domain Generalized Face Anti-spoofing](https://mlsanity.com/paper/2302.08674/) (2023)
- [Disentangling Content from Style to Overcome Shortcut Learning: A Hybrid Generative-Discriminative Learning Framework](https://mlsanity.com/paper/2509.11598/) (2025)
- [Minimal Semantic Sufficiency Meets Unsupervised Domain Generalization](https://mlsanity.com/paper/2509.15791/) (2025)
- [ALUM: Adversarial Data Uncertainty Modeling from Latent Model Uncertainty Compensation](https://mlsanity.com/paper/2303.16866/) (2023)
- [Adapting Self-Supervised Representations to Multi-Domain Setups](https://mlsanity.com/paper/2309.03999/) (2023)

## Abstract (arXiv)

Generalizing learned representations across significantly different visual domains is a fundamental yet crucial ability of the human visual system. While recent self-supervised learning methods have achieved good performances with evaluation set on the same domain as the training set, they will have an undesirable performance decrease when tested on a different domain. Therefore, the self-supervised learning from multiple domains task is proposed to learn domain-invariant features that are not only suitable for evaluation on the same domain as the training set but also can be generalized to unseen domains. In this paper, we propose a Domain-invariant Masked AutoEncoder (DiMAE) for self-supervised learning from multi-domains, which designs a new pretext task, i.e., the cross-domain reconstruction task, to learn domain-invariant features. The core idea is to augment the input image with style noise from different domains and then reconstruct the image from the embedding of the augmented image, regularizing the encoder to learn domain-invariant features. To accomplish the idea, DiMAE contains two critical designs, 1) content-preserved style mix, which adds style information from other domains to input while persevering the content in a parameter-free manner, and 2) multiple domain-specific decoders, which recovers the corresponding domain style of input to the encoded domain-invariant features for reconstruction. Experiments on PACS and DomainNet illustrate that DiMAE achieves considerable gains compared with recent state-of-the-art methods.