# CellCLIP -- Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning

arXiv: https://arxiv.org/abs/2506.06290 · HTML: https://mlsanity.com/paper/2506.06290/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 54 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP](https://mlsanity.com/paper/2110.11316/) (2021)
- [Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology](https://mlsanity.com/paper/2404.10242/) (2024)
- [On the Scalability of GNNs for Molecular Graphs](https://mlsanity.com/paper/2404.11568/) (2024)
- [Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words](https://mlsanity.com/paper/2309.16108/) (2023)
- [GPS++: An Optimised Hybrid MPNN/Transformer for Molecular Property Prediction](https://mlsanity.com/paper/2212.02229/) (2022)
- [CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss](https://mlsanity.com/paper/2309.14580/) (2023)
- [MAFA: Managing False Negatives for Vision-Language Pre-training](https://mlsanity.com/paper/2312.06112/) (2023)
- [RxRx3-core: Benchmarking drug-target interactions in High-Content Microscopy](https://mlsanity.com/paper/2503.20158/) (2025)
- [Can Large Language Models Understand Molecules?](https://mlsanity.com/paper/2402.00024/) (2024)
- [How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval](https://mlsanity.com/paper/2409.08302/) (2024)

## Built on by (most cited first, 7 of 7)

- [Efficient Cell Painting Image Representation Learning via Cross-Well Aligned Masked Siamese Network](https://mlsanity.com/paper/2509.19896/) (2025)
- [Progressive Multi-Agent Reasoning for Biological Perturbation Prediction](https://mlsanity.com/paper/2602.07408/) (2026)
- [Can AI Scientist Agents Learn from Lab-in-the-Loop Feedback? Evidence from Iterative Perturbation Discovery](https://mlsanity.com/paper/2603.26177/) (2026)
- [CP-Agent: Context-Aware Multimodal Reasoning for Cellular Morphological Profiling under Chemical Perturbations](https://mlsanity.com/paper/2606.03435/) (2026)
- [Rethinking Text-Based Image Retrieval in Specific Domain](https://mlsanity.com/paper/2608.10524/) (2026)
- [MorphoCLIP: Text-Supervised Contrastive Learning for Perturbation Matching in Cell Painting Images](https://mlsanity.com/paper/2608.22690/) (2026)
- [MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining](https://mlsanity.com/paper/2610.17866/) (2026)

## Abstract (arXiv)

High-content screening (HCS) assays based on high-throughput microscopy techniques such as Cell Painting have enabled the interrogation of cells' morphological responses to perturbations at an unprecedented scale. The collection of such data promises to facilitate a better understanding of the relationships between different perturbations and their effects on cellular state. Towards achieving this goal, recent advances in cross-modal contrastive learning could, in theory, be leveraged to learn a unified latent space that aligns perturbations with their corresponding morphological effects. However, the application of such methods to HCS data is not straightforward due to substantial differences in the semantics of Cell Painting images compared to natural images, and the difficulty of representing different classes of perturbations (e.g., small molecule vs CRISPR gene knockout) in a single latent space. In response to these challenges, here we introduce CellCLIP, a cross-modal contrastive learning framework for HCS data. CellCLIP leverages pre-trained image encoders coupled with a novel channel encoding scheme to better capture relationships between different microscopy channels in image embeddings, along with natural language encoders for representing perturbations. Our framework outperforms current open-source models, demonstrating the best performance in both cross-modal retrieval and biologically meaningful downstream tasks while also achieving significant reductions in computation time.