# ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy

arXiv: https://arxiv.org/abs/2411.02572 · HTML: https://mlsanity.com/paper/2411.02572/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 69 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [MultiGrain: a unified image embedding for classes and instances](https://mlsanity.com/paper/1902.05509/) (2019)
- [Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning](https://mlsanity.com/paper/2201.03529/) (2022)
- [The effectiveness of MAE pre-pretraining for billion-scale pretraining](https://mlsanity.com/paper/2303.13496/) (2023)
- [Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology](https://mlsanity.com/paper/2404.10242/) (2024)
- [RxRx1: A Dataset for Evaluating Experimental Batch Correction Methods](https://mlsanity.com/paper/2301.05768/) (2023)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [Learning Super-Features for Image Retrieval](https://mlsanity.com/paper/2201.13182/) (2022)
- [Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words](https://mlsanity.com/paper/2309.16108/) (2023)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures](https://mlsanity.com/paper/1808.09011/) (2018)
- [MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representations](https://mlsanity.com/paper/2402.10093/) (2024)
- [Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget](https://mlsanity.com/paper/2304.10520/) (2023)
- [ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images](https://mlsanity.com/paper/2311.15264/) (2023)
- [Masked Autoencoders are Scalable Learners of Cellular Morphology](https://mlsanity.com/paper/2309.16064/) (2023)

## Built on by (most cited first, 13 of 13)

- [RxRx3-core: Benchmarking drug-target interactions in High-Content Microscopy](https://mlsanity.com/paper/2503.20158/) (2025)
- [AI-powered virtual tissues from spatial proteomics for clinical diagnostics and biomedical discovery](https://mlsanity.com/paper/2501.06039/) (2025)
- [ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning](https://mlsanity.com/paper/2503.19331/) (2025)
- [Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation models](https://mlsanity.com/paper/2412.16247/) (2024)
- [C3R: Channel Conditioned Cell Representations for unified evaluation in microscopy imaging](https://mlsanity.com/paper/2505.18745/) (2025)
- [A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features](https://mlsanity.com/paper/2505.21317/) (2025)
- [Efficient Cell Painting Image Representation Learning via Cross-Well Aligned Masked Siamese Network](https://mlsanity.com/paper/2509.19896/) (2025)
- [CHAMMI-75: Pre-training multi-channel models with heterogeneous microscopy images](https://mlsanity.com/paper/2512.20833/) (2025)
- [Elucidating the Design Space of Flow Matching for Cellular Microscopy](https://mlsanity.com/paper/2603.26790/) (2026)
- [Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing](https://mlsanity.com/paper/2502.00594/) (2025)
- [Deep Learning for BioImaging: What Are We Really Learning?](https://mlsanity.com/paper/2603.13377/) (2026)
- [Intervention-Aware Multiscale Representation Learning from Imaging Phenomics and Perturbation Transcriptomics](https://mlsanity.com/paper/2604.22832/) (2026)
- [Effective Biological Representation Learning by Masking Gene Expression](https://mlsanity.com/paper/2605.31562/) (2026)

## Abstract (arXiv)

Large-scale cell microscopy screens are used in drug discovery and molecular biology research to study the effects of millions of chemical and genetic perturbations on cells. To use these images in downstream analysis, we need models that can map each image into a feature space that represents diverse biological phenotypes consistently, in the sense that perturbations with similar biological effects have similar representations. In this work, we present the largest foundation model for cell microscopy data to date, a new 1.9 billion-parameter ViT-G/8 MAE trained on over 8 billion microscopy image crops. Compared to a previous published ViT-L/8 MAE, our new model achieves a 60% improvement in linear separability of genetic perturbations and obtains the best overall performance on whole-genome biological relationship recall and replicate consistency benchmarks. Beyond scaling, we developed two key methods that improve performance: (1) training on a curated and diverse dataset; and, (2) using biologically motivated linear probing tasks to search across each transformer block for the best candidate representation of whole-genome screens. We find that many self-supervised vision transformers, pretrained on either natural or microscopy images, yield significantly more biologically meaningful representations of microscopy images in their intermediate blocks than in their typically used final blocks. More broadly, our approach and results provide insights toward a general strategy for successfully building foundation models for large-scale biological data.