# From Pixels to Components: Eigenvector Masking for Visual Representation Learning

arXiv: https://arxiv.org/abs/2502.06314 · HTML: https://mlsanity.com/paper/2502.06314/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 30 of 34)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [Context Autoencoder for Self-Supervised Representation Learning](https://mlsanity.com/paper/2202.03026/) (2022)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers](https://mlsanity.com/paper/2111.12710/) (2021)
- [What to Hide from Your Students: Attention-Guided Masked Image Modeling](https://mlsanity.com/paper/2203.12719/) (2022)
- [MST: Masked Self-Supervised Transformer for Visual Representation](https://mlsanity.com/paper/2106.05656/) (2021)
- [SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders](https://mlsanity.com/paper/2206.10207/) (2022)
- [Deconstructing Denoising Diffusion Models for Self-Supervised Learning](https://mlsanity.com/paper/2401.14404/) (2024)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [Adversarial Masking for Self-Supervised Learning](https://mlsanity.com/paper/2201.13100/) (2022)
- [Hard Patches Mining for Masked Image Modeling](https://mlsanity.com/paper/2304.05919/) (2023)
- [Learning by Reconstruction Produces Uninformative Features For Perception](https://mlsanity.com/paper/2402.11337/) (2024)

## Built on by (most cited first, 2 of 2)

- [Can Masked Autoencoders Also Listen to Birds?](https://mlsanity.com/paper/2504.12880/) (2025)
- [Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning](https://mlsanity.com/paper/2609.12035/) (2026)

## Abstract (arXiv)

Predicting masked from visible parts of an image is a powerful self-supervised approach for visual representation learning. However, the common practice of masking random patches of pixels exhibits certain failure modes, which can prevent learning meaningful high-level features, as required for downstream tasks. We propose an alternative masking strategy that operates on a suitable transformation of the data rather than on the raw pixels. Specifically, we perform principal component analysis and then randomly mask a subset of components, which accounts for a fixed ratio of the data variance. The learning task then amounts to reconstructing the masked components from the visible ones. Compared to local patches of pixels, the principal components of images carry more global information. We thus posit that predicting masked from visible components involves more high-level features, allowing our masking strategy to extract more useful representations. This is corroborated by our empirical findings which demonstrate improved image classification performance for component over pixel masking. Our method thus constitutes a simple and robust data-driven alternative to traditional masked image modeling approaches.