# Masked Autoencoders are Scalable Learners of Cellular Morphology

arXiv: https://arxiv.org/abs/2309.16064 · HTML: https://mlsanity.com/paper/2309.16064/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 27 Sep 2023.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Three things everyone should know about Vision Transformers](https://mlsanity.com/paper/2203.09795/) (2022)
- [Masked Frequency Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2206.07706/) (2022)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [RxRx1: A Dataset for Evaluating Experimental Batch Correction Methods](https://mlsanity.com/paper/2301.05768/) (2023)
- [Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning](https://mlsanity.com/paper/1511.05286/) (2015)
- [Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak Labels](https://mlsanity.com/paper/2209.07819/) (2022)
- [Metadata-guided Consistency Learning for High Content Images](https://mlsanity.com/paper/2212.11595/) (2022)
- [DEEMD: Drug Efficacy Estimation against SARS-CoV-2 based on cell Morphology with Deep multiple instance learning](https://mlsanity.com/paper/2105.05758/) (2021)

## Built on by (most cited first, 9 of 9)

- [Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology](https://mlsanity.com/paper/2404.10242/) (2024)
- [ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy](https://mlsanity.com/paper/2411.02572/) (2024)
- [Weakly Supervised Set-Consistency Learning Improves Morphological Profiling of Single-Cell Images](https://mlsanity.com/paper/2406.05308/) (2024)
- [Automated Discovery of Pairwise Interactions from Unstructured Data](https://mlsanity.com/paper/2409.07594/) (2024)
- [Propensity Score Alignment of Unpaired Multimodal Data](https://mlsanity.com/paper/2404.01595/) (2024)
- [DynaCLR: Contrastive Learning of Cellular Dynamics with Temporal Regularization](https://mlsanity.com/paper/2410.11281/) (2024)
- [Membership Inference Attacks with False Discovery Rate Control](https://mlsanity.com/paper/2508.07066/) (2025)
- [PhenoProfiler: Advancing Phenotypic Learning for Image-based Drug Discovery](https://mlsanity.com/paper/2502.19568/) (2025)
- [Gene-Level Representation Learning via Interventional Style Transfer in Optical Pooled Screening](https://mlsanity.com/paper/2406.07763/) (2024)

## Abstract (arXiv)

Inferring biological relationships from cellular phenotypes in high-content microscopy screens provides significant opportunity and challenge in biological research. Prior results have shown that deep vision models can capture biological signal better than hand-crafted features. This work explores how self-supervised deep learning approaches scale when training larger models on larger microscopy datasets. Our results show that both CNN- and ViT-based masked autoencoders significantly outperform weakly supervised baselines. At the high-end of our scale, a ViT-L/8 trained on over 3.5-billion unique crops sampled from 93-million microscopy images achieves relative improvements as high as 28% over our best weakly supervised baseline at inferring known biological relationships curated from public databases. Relevant code and select models released with this work can be found at: https://github.com/recursionpharma/maes_microscopy.