# Integrating chemical structures as treatments improves representations of microscopy images for morphological profiling

arXiv: https://arxiv.org/abs/2504.09544 · HTML: https://mlsanity.com/paper/2504.09544/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 44 references that are in the map.
- First posted on arXiv 13 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology](https://mlsanity.com/paper/2404.10242/) (2024)
- [Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning](https://mlsanity.com/paper/1511.05286/) (2015)
- [CHAMMI: A benchmark for channel-adaptive models in microscopy imaging](https://mlsanity.com/paper/2310.19224/) (2023)
- [Contrastive Learning of Single-Cell Phenotypic Representations for Treatment Classification](https://mlsanity.com/paper/2103.16670/) (2021)
- [Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak Labels](https://mlsanity.com/paper/2209.07819/) (2022)
- [Weakly Supervised Set-Consistency Learning Improves Morphological Profiling of Single-Cell Images](https://mlsanity.com/paper/2406.05308/) (2024)
- [How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval](https://mlsanity.com/paper/2409.08302/) (2024)
- [Morphological Profiling for Drug Discovery in the Era of Deep Learning](https://mlsanity.com/paper/2312.07899/) (2023)

## Built on by (most cited first, 2 of 2)

- [Efficient Cell Painting Image Representation Learning via Cross-Well Aligned Masked Siamese Network](https://mlsanity.com/paper/2509.19896/) (2025)
- [Self-supervised Representation Learning with Local Aggregation for Image-based Profiling](https://mlsanity.com/paper/2506.14265/) (2025)

## Abstract (arXiv)

Recent advances in self-supervised deep learning have improved our ability to quantify cellular morphological changes in high-throughput microscopy screens, a process known as morphological profiling. However, most current methods only learn from images, despite many screens being inherently multimodal, as they involve both a chemical or genetic perturbation as well as an image-based readout. We hypothesized that incorporating chemical compound structures during self-supervised pre-training could improve learned representations of images from high-throughput microscopy screens. We introduce a representation learning framework, MICON (Molecular-Image Contrastive Learning), that models chemical compounds as treatments that induce transformations of cell phenotypes. MICON significantly outperforms classical hand-crafted features such as CellProfiler and existing deep-learning-based representation learning methods in challenging evaluation settings where models must identify reproducible effects of drugs across independent replicates and data-generating centers. We demonstrate that incorporating chemical compound information into the learning process provides small, but consistent improvements in performance and that modeling compounds specifically as treatments outperforms approaches that directly align images and compounds in a single representation space. Our findings point to a new direction for representation learning in morphological profiling, suggesting that methods should explicitly account for the multimodal nature of microscopy screening data.