# Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology

arXiv: https://arxiv.org/abs/2404.10242 · HTML: https://mlsanity.com/paper/2404.10242/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 71 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 30 of 33)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Multimodal Masked Autoencoders Learn Transferable Representations](https://mlsanity.com/paper/2205.14204/) (2022)
- [Three things everyone should know about Vision Transformers](https://mlsanity.com/paper/2203.09795/) (2022)
- [Masked Frequency Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2206.07706/) (2022)
- [Dilated Neighborhood Attention Transformer](https://mlsanity.com/paper/2209.15001/) (2022)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond](https://mlsanity.com/paper/2208.00173/) (2022)
- [RxRx1: A Dataset for Evaluating Experimental Batch Correction Methods](https://mlsanity.com/paper/2301.05768/) (2023)
- [Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning](https://mlsanity.com/paper/1511.05286/) (2015)
- [Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak Labels](https://mlsanity.com/paper/2209.07819/) (2022)

## Built on by (most cited first, 30 of 52)

- [How to Build the Virtual Cell with Artificial Intelligence: Priorities and Opportunities](https://mlsanity.com/paper/2409.11654/) (2024)
- [On the Scalability of GNNs for Molecular Graphs](https://mlsanity.com/paper/2404.11568/) (2024)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all](https://mlsanity.com/paper/2410.13956/) (2024)
- [Flow Matching Meets Biology and Life Science: A Survey](https://mlsanity.com/paper/2507.17731/) (2025)
- [ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy](https://mlsanity.com/paper/2411.02572/) (2024)
- [RxRx3-core: Benchmarking drug-target interactions in High-Content Microscopy](https://mlsanity.com/paper/2503.20158/) (2025)
- [TxPert: Leveraging Biochemical Relationships for Out-of-Distribution Transcriptomic Perturbation Prediction](https://mlsanity.com/paper/2505.14919/) (2025)
- [UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training](https://mlsanity.com/paper/2502.02307/) (2025)
- [CellCLIP -- Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning](https://mlsanity.com/paper/2506.06290/) (2025)
- [THOR: A Versatile Foundation Model for Earth Observation Climate and Society Applications](https://mlsanity.com/paper/2601.16011/) (2026)
- [How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval](https://mlsanity.com/paper/2409.08302/) (2024)
- [AI-powered virtual tissues from spatial proteomics for clinical diagnostics and biomedical discovery](https://mlsanity.com/paper/2501.06039/) (2025)
- [ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning](https://mlsanity.com/paper/2503.19331/) (2025)
- [Statistical and structural identifiability in representation learning](https://mlsanity.com/paper/2603.11970/) (2026)
- [Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation models](https://mlsanity.com/paper/2412.16247/) (2024)
- [SatelliteCalculator: A Multi-Task Vision Foundation Model for Quantitative Remote Sensing Inversion](https://mlsanity.com/paper/2504.13442/) (2025)
- [MorphGen: Controllable and Morphologically Plausible Generative Cell-Imaging](https://mlsanity.com/paper/2510.01298/) (2025)
- [Integrating chemical structures as treatments improves representations of microscopy images for morphological profiling](https://mlsanity.com/paper/2504.09544/) (2025)
- [C3R: Channel Conditioned Cell Representations for unified evaluation in microscopy imaging](https://mlsanity.com/paper/2505.18745/) (2025)
- [A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features](https://mlsanity.com/paper/2505.21317/) (2025)
- [Efficient Cell Painting Image Representation Learning via Cross-Well Aligned Masked Siamese Network](https://mlsanity.com/paper/2509.19896/) (2025)
- [GrapHist: Graph Self-Supervised Learning for Histopathology](https://mlsanity.com/paper/2603.00143/) (2026)
- [PhenoProfiler: Advancing Phenotypic Learning for Image-based Drug Discovery](https://mlsanity.com/paper/2502.19568/) (2025)
- [Learning from Random Subspace Exploration: Generalized Test-Time Augmentation with Self-supervised Distillation](https://mlsanity.com/paper/2507.01347/) (2025)
- [Compressing Biology: Evaluating the Stable Diffusion VAE for Phenotypic Drug Discovery](https://mlsanity.com/paper/2510.19887/) (2025)
- [CHAMMI-75: Pre-training multi-channel models with heterogeneous microscopy images](https://mlsanity.com/paper/2512.20833/) (2025)
- [Generalization of Self-Supervised Vision Transformers for Protein Localization Across Microscopy Domains](https://mlsanity.com/paper/2602.05527/) (2026)
- [Elucidating the Design Space of Flow Matching for Cellular Microscopy](https://mlsanity.com/paper/2603.26790/) (2026)

## Abstract (arXiv)

Featurizing microscopy images for use in biological research remains a significant challenge, especially for large-scale experiments spanning millions of images. This work explores the scaling properties of weakly supervised classifiers and self-supervised masked autoencoders (MAEs) when training with increasingly larger model backbones and microscopy datasets. Our results show that ViT-based MAEs outperform weakly supervised classifiers on a variety of tasks, achieving as much as a 11.5% relative improvement when recalling known biological relationships curated from public databases. Additionally, we develop a new channel-agnostic MAE architecture (CA-MAE) that allows for inputting images of different numbers and orders of channels at inference time. We demonstrate that CA-MAEs effectively generalize by inferring and evaluating on a microscopy image dataset (JUMP-CP) generated under different experimental conditions with a different channel structure than our pretraining data (RPI-93M). Our findings motivate continued research into scaling self-supervised learning on microscopy data in order to create powerful foundation models of cellular biology that have the potential to catalyze advancements in drug discovery and beyond.