# Weakly supervised cross-modal learning in high-content screening

arXiv: https://arxiv.org/abs/2311.04678 · HTML: https://mlsanity.com/paper/2311.04678/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Principal Neighbourhood Aggregation for Graph Nets](https://mlsanity.com/paper/2004.05718/) (2020)
- [How Powerful are Spectral Graph Neural Networks](https://mlsanity.com/paper/2205.11172/) (2022)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [RxRx1: A Dataset for Evaluating Experimental Batch Correction Methods](https://mlsanity.com/paper/2301.05768/) (2023)
- [Molecule-Morphology Contrastive Pretraining for Transferable Molecular Representation](https://mlsanity.com/paper/2305.09790/) (2023)
- [No Free Lunch in Self Supervised Representation Learning](https://mlsanity.com/paper/2304.11718/) (2023)
- [MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations](https://mlsanity.com/paper/2303.10839/) (2023)

## Built on by (most cited first, 4 of 4)

- [ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images](https://mlsanity.com/paper/2311.15264/) (2023)
- [A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features](https://mlsanity.com/paper/2505.21317/) (2025)
- [Out-of-distribution evaluations of channel agnostic masked autoencoders in fluorescence microscopy](https://mlsanity.com/paper/2503.19149/) (2025)
- [Deep Learning for BioImaging: What Are We Really Learning?](https://mlsanity.com/paper/2603.13377/) (2026)

## Abstract (arXiv)

With the surge in available data from various modalities, there is a growing need to bridge the gap between different data types. In this work, we introduce a novel approach to learn cross-modal representations between image data and molecular representations for drug discovery. We propose EMM and IMM, two innovative loss functions built on top of CLIP that leverage weak supervision and cross sites replicates in High-Content Screening. Evaluating our model against known baseline on cross-modal retrieval, we show that our proposed approach allows to learn better representations and mitigate batch effect. In addition, we also present a preprocessing method for the JUMP-CP dataset that effectively reduce the required space from 85Tb to a mere usable 7Tb size, still retaining all perturbations and most of the information content.