# How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval

arXiv: https://arxiv.org/abs/2409.08302 · HTML: https://mlsanity.com/paper/2409.08302/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 10 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Pre-training via Denoising for Molecular Property Prediction](https://mlsanity.com/paper/2206.00133/) (2022)
- [CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP](https://mlsanity.com/paper/2110.11316/) (2021)
- [Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology](https://mlsanity.com/paper/2404.10242/) (2024)
- [RxRx1: A Dataset for Evaluating Experimental Batch Correction Methods](https://mlsanity.com/paper/2301.05768/) (2023)
- [How to Understand Masked Autoencoders](https://mlsanity.com/paper/2202.03670/) (2022)
- [Is Self-Supervised Learning More Robust Than Supervised Learning?](https://mlsanity.com/paper/2206.05259/) (2022)
- [CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss](https://mlsanity.com/paper/2309.14580/) (2023)
- [Molecule-Morphology Contrastive Pretraining for Transferable Molecular Representation](https://mlsanity.com/paper/2305.09790/) (2023)
- [GPS++: Reviving the Art of Message Passing for Molecular Property Prediction](https://mlsanity.com/paper/2302.02947/) (2023)

## Built on by (most cited first, 6 of 6)

- [CellCLIP -- Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning](https://mlsanity.com/paper/2506.06290/) (2025)
- [Integrating chemical structures as treatments improves representations of microscopy images for morphological profiling](https://mlsanity.com/paper/2504.09544/) (2025)
- [A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features](https://mlsanity.com/paper/2505.21317/) (2025)
- [Intervention-Aware Multiscale Representation Learning from Imaging Phenomics and Perturbation Transcriptomics](https://mlsanity.com/paper/2604.22832/) (2026)
- [Conditional Neural Optimal Transport for Predicting Cellular Phenotypes from Molecular Structure](https://mlsanity.com/paper/2608.14293/) (2026)
- [MorphoCLIP: Text-Supervised Contrastive Learning for Perturbation Matching in Cell Painting Images](https://mlsanity.com/paper/2608.22690/) (2026)

## Abstract (arXiv)

Predicting molecular impact on cellular function is a core challenge in therapeutic design. Phenomic experiments, designed to capture cellular morphology, utilize microscopy based techniques and demonstrate a high throughput solution for uncovering molecular impact on the cell. In this work, we learn a joint latent space between molecular structures and microscopy phenomic experiments, aligning paired samples with contrastive learning. Specifically, we study the problem ofContrastive PhenoMolecular Retrieval, which consists of zero-shot molecular structure identification conditioned on phenomic experiments. We assess challenges in multi-modal learning of phenomics and molecular modalities such as experimental batch effect, inactive molecule perturbations, and encoding perturbation concentration. We demonstrate improved multi-modal learner retrieval through (1) a uni-modal pre-trained phenomics model, (2) a novel inter sample similarity aware loss, and (3) models conditioned on a representation of molecular concentration. Following this recipe, we propose MolPhenix, a molecular phenomics model. MolPhenix leverages a pre-trained phenomics model to demonstrate significant performance gains across perturbation concentrations, molecular scaffolds, and activity thresholds. In particular, we demonstrate an 8.1x improvement in zero shot molecular retrieval of active molecules over the previous state-of-the-art, reaching 77.33% in top-1% accuracy. These results open the door for machine learning to be applied in virtual phenomics screening, which can significantly benefit drug discovery applications.