# A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features

arXiv: https://arxiv.org/abs/2505.21317 · HTML: https://mlsanity.com/paper/2505.21317/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 69 references that are in the map.
- First posted on arXiv 27 May 2025.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [DiffDock: Diffusion Steps, Twists, and Turns for Molecular Docking](https://mlsanity.com/paper/2210.01776/) (2022)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [Dataset Augmentation in Feature Space](https://mlsanity.com/paper/1702.05538/) (2017)
- [TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance](https://mlsanity.com/paper/2309.12314/) (2023)
- [CLIP-KD: An Empirical Study of CLIP Model Distillation](https://mlsanity.com/paper/2307.12732/) (2023)
- [Compressing Visual-linguistic Model via Knowledge Distillation](https://mlsanity.com/paper/2104.02096/) (2021)
- [Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning](https://mlsanity.com/paper/2306.01859/) (2023)
- [Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data](https://mlsanity.com/paper/2303.14080/) (2023)
- [Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology](https://mlsanity.com/paper/2404.10242/) (2024)
- [XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning](https://mlsanity.com/paper/2211.13929/) (2022)
- [Multimodal Knowledge Expansion](https://mlsanity.com/paper/2103.14431/) (2021)
- [Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks](https://mlsanity.com/paper/2204.10496/) (2022)
- [Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all](https://mlsanity.com/paper/2410.13956/) (2024)
- [You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning](https://mlsanity.com/paper/2406.09294/) (2024)
- [Rotation-Agnostic Image Representation Learning for Digital Pathology](https://mlsanity.com/paper/2311.08359/) (2023)
- [ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images](https://mlsanity.com/paper/2311.15264/) (2023)
- [ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy](https://mlsanity.com/paper/2411.02572/) (2024)
- [Cross-Modality Distillation: A case for Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1807.07682/) (2018)
- [Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers](https://mlsanity.com/paper/2405.16419/) (2024)
- [Weakly Supervised Set-Consistency Learning Improves Morphological Profiling of Single-Cell Images](https://mlsanity.com/paper/2406.05308/) (2024)
- [Rotation Invariance and Extensive Data Augmentation: a strategy for the Mitosis Domain Generalization (MIDOG) Challenge](https://mlsanity.com/paper/2109.00823/) (2021)
- [Propensity Score Alignment of Unpaired Multimodal Data](https://mlsanity.com/paper/2404.01595/) (2024)
- [How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval](https://mlsanity.com/paper/2409.08302/) (2024)
- [Weakly supervised cross-modal learning in high-content screening](https://mlsanity.com/paper/2311.04678/) (2023)

## Built on by (most cited first, 2 of 2)

- [Intervention-Aware Multiscale Representation Learning from Imaging Phenomics and Perturbation Transcriptomics](https://mlsanity.com/paper/2604.22832/) (2026)
- [Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm](https://mlsanity.com/paper/2606.10504/) (2026)

## Abstract (arXiv)

Understanding cellular responses to stimuli is crucial for biological discovery and drug development. Transcriptomics provides interpretable, gene-level insights, while microscopy imaging offers rich predictive features but is harder to interpret. Weakly paired datasets, where samples share biological states, enable multimodal learning but are scarce, limiting their utility for training and multimodal inference. We propose a framework to enhance transcriptomics by distilling knowledge from microscopy images. Using weakly paired data, our method aligns and binds modalities, enriching gene expression representations with morphological information. To address data scarcity, we introduce (1) Semi-Clipped, an adaptation of CLIP for cross-modal distillation using pretrained foundation models, achieving state-of-the-art results, and (2) PEA (Perturbation Embedding Augmentation), a novel augmentation technique that enhances transcriptomics data while preserving inherent biological information. These strategies improve the predictive power and retain the interpretability of transcriptomics, enabling rich unimodal representations for complex biological tasks.