# PixMIM: Rethinking Pixel Reconstruction in Masked Image Modeling

arXiv: https://arxiv.org/abs/2303.02416 · HTML: https://mlsanity.com/paper/2303.02416/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 65 references that are in the map.
- First posted on arXiv 4 Mar 2023.

## Builds on (most cited first, 30 of 49)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)

## Built on by (most cited first, 28 of 28)

- [Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06659/) (2024)
- [Predicting Gradient is Better: Exploring Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2311.15153/) (2023)
- [SARATR-X: Toward Building A Foundation Model for SAR Target Recognition](https://mlsanity.com/paper/2405.09365/) (2024)
- [Improving Pixel-based MIM by Reducing Wasted Modeling Capability](https://mlsanity.com/paper/2308.00261/) (2023)
- [Masked Angle-Aware Autoencoder for Remote Sensing Images](https://mlsanity.com/paper/2408.01946/) (2024)
- [A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification](https://mlsanity.com/paper/2407.12210/) (2024)
- [T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency](https://mlsanity.com/paper/2312.01529/) (2023)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training](https://mlsanity.com/paper/2312.01522/) (2023)
- [QPT V2: Masked Image Modeling Advances Visual Scoring](https://mlsanity.com/paper/2407.16541/) (2024)
- [Probing the Mid-level Vision Capabilities of Self-Supervised Learning](https://mlsanity.com/paper/2411.17474/) (2024)
- [SuPreME: A Supervised Pre-training Framework for Multimodal ECG Representation Learning](https://mlsanity.com/paper/2502.19668/) (2025)
- [Rethinking Overlooked Aspects in Vision-Language Models](https://mlsanity.com/paper/2405.11850/) (2024)
- [CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework](https://mlsanity.com/paper/2504.12576/) (2025)
- [POINTS-GUI-G: GUI-Grounding Journey](https://mlsanity.com/paper/2602.06391/) (2026)
- [Learning Mask Invariant Mutual Information for Masked Image Modeling](https://mlsanity.com/paper/2502.19718/) (2025)
- [Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation](https://mlsanity.com/paper/2503.00782/) (2025)
- [Self-Guided Masked Autoencoder](https://mlsanity.com/paper/2507.19773/) (2025)
- [Explanations of Classifiers Enhance Medical Image Segmentation via End-to-end Pre-training](https://mlsanity.com/paper/2401.08469/) (2024)
- [SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representation](https://mlsanity.com/paper/2406.10673/) (2024)
- [Self-Supervised Learning for Real-World Object Detection: a Survey](https://mlsanity.com/paper/2410.07442/) (2024)
- [Dataset Ownership Verification for Pre-trained Masked Models](https://mlsanity.com/paper/2507.12022/) (2025)
- [NERULA: A Dual-Pathway Self-Supervised Learning Framework for Electrocardiogram Signal Analysis](https://mlsanity.com/paper/2405.19348/) (2024)
- [Disentangling Content from Style to Overcome Shortcut Learning: A Hybrid Generative-Discriminative Learning Framework](https://mlsanity.com/paper/2509.11598/) (2025)
- [PiLaMIM: Toward Richer Visual Representations by Integrating Pixel and Latent Masked Image Modeling](https://mlsanity.com/paper/2501.03005/) (2025)
- [MaskAnyNet: Rethinking Masked Image Regions as Valuable Information in Supervised Learning](https://mlsanity.com/paper/2511.12480/) (2025)
- [Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance](https://mlsanity.com/paper/2604.16086/) (2026)
- [Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning](https://mlsanity.com/paper/2609.38278/) (2026)

## Abstract (arXiv)

Masked Image Modeling (MIM) has achieved promising progress with the advent of Masked Autoencoders (MAE) and BEiT. However, subsequent works have complicated the framework with new auxiliary tasks or extra pre-trained models, inevitably increasing computational overhead. This paper undertakes a fundamental analysis of MIM from the perspective of pixel reconstruction, which examines the input image patches and reconstruction target, and highlights two critical but previously overlooked bottlenecks. Based on this analysis, we propose a remarkably simple and effective method, ourmethod, that entails two strategies: 1) filtering the high-frequency components from the reconstruction target to de-emphasize the network's focus on texture-rich details and 2) adopting a conservative data transform strategy to alleviate the problem of missing foreground in MIM training. ourmethod can be easily integrated into most existing pixel-based MIM approaches (ie, using raw images as reconstruction target) with negligible additional computation. Without bells and whistles, our method consistently improves three MIM approaches, MAE, ConvMAE, and LSMAE, across various downstream tasks. We believe this effective plug-and-play method will serve as a strong baseline for self-supervised learning and provide insights for future improvements of the MIM framework. Code and models are available at urlhttps://github.com/open-mmlab/mmselfsup/tree/dev-1.x/configs/selfsup/pixmim.