# ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images

arXiv: https://arxiv.org/abs/2311.15264 · HTML: https://mlsanity.com/paper/2311.15264/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 66 references that are in the map.
- First posted on arXiv 26 Nov 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [LambdaNetworks: Modeling Long-Range Interactions Without Attention](https://mlsanity.com/paper/2102.08602/) (2021)
- [Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition](https://mlsanity.com/paper/2106.12368/) (2021)
- [CytoImageNet: A large-scale pretraining dataset for bioimage transfer learning](https://mlsanity.com/paper/2111.11646/) (2021)
- [Unpaired Image-to-Image Translation with Limited Data to Reveal Subtle Phenotypes](https://mlsanity.com/paper/2302.08503/) (2023)
- [Weakly supervised cross-modal learning in high-content screening](https://mlsanity.com/paper/2311.04678/) (2023)
- [Comparison of semi-supervised learning methods for High Content Screening quality control](https://mlsanity.com/paper/2208.04592/) (2022)
- [aura-net : robust segmentation of phase-contrast microscopy images with few annotations](https://mlsanity.com/paper/2102.01389/) (2021)

## Built on by (most cited first, 15 of 15)

- [ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy](https://mlsanity.com/paper/2411.02572/) (2024)
- [Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers](https://mlsanity.com/paper/2405.16419/) (2024)
- [ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning](https://mlsanity.com/paper/2503.19331/) (2025)
- [Isolated Channel Vision Transformers: From Single-Channel Pretraining to Multi-Channel Finetuning](https://mlsanity.com/paper/2503.09826/) (2025)
- [C3R: Channel Conditioned Cell Representations for unified evaluation in microscopy imaging](https://mlsanity.com/paper/2505.18745/) (2025)
- [A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features](https://mlsanity.com/paper/2505.21317/) (2025)
- [CHAMMI-75: Pre-training multi-channel models with heterogeneous microscopy images](https://mlsanity.com/paper/2512.20833/) (2025)
- [DC-ViT: Modulating Spatial and Channel Interactions for Multi-Channel Images](https://mlsanity.com/paper/2603.14281/) (2026)
- [JUMP-lite: Compact, reproducible benchmarking of cell representations](https://mlsanity.com/paper/2608.07632/) (2026)
- [Out-of-distribution evaluations of channel agnostic masked autoencoders in fluorescence microscopy](https://mlsanity.com/paper/2503.19149/) (2025)
- [Sparse Mixture-of-Experts for Multi-Channel Imaging: Are All Channel Interactions Required?](https://mlsanity.com/paper/2511.17400/) (2025)
- [mViSE: A Visual Search Engine for Analyzing Multiplex IHC Brain Tissue Images](https://mlsanity.com/paper/2512.11745/) (2025)
- [Channel-Aware Probing for Multi-Channel Imaging](https://mlsanity.com/paper/2602.12696/) (2026)
- [3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy](https://mlsanity.com/paper/2606.23964/) (2026)
- [HASSL: Hierarchy-Aware Self-Supervised Learning Framework for Single Cell Microscopy](https://mlsanity.com/paper/2607.04353/) (2026)

## Abstract (arXiv)

Unlike color photography images, which are consistently encoded into RGB channels, biological images encompass various modalities, where the type of microscopy and the meaning of each channel varies with each experiment. Importantly, the number of channels can range from one to a dozen and their correlation is often comparatively much lower than RGB, as each of them brings specific information content. This aspect is largely overlooked by methods designed out of the bioimage field, and current solutions mostly focus on intra-channel spatial attention, often ignoring the relationship between channels, yet crucial in most biological applications. Importantly, the variable channel type and count prevent the projection of several experiments to a unified representation for large scale pre-training. In this study, we propose ChAda-ViT, a novel Channel Adaptive Vision Transformer architecture employing an Inter-Channel Attention mechanism on images with an arbitrary number, order and type of channels. We also introduce IDRCell100k, a bioimage dataset with a rich set of 79 experiments covering 7 microscope modalities, with a multitude of channel types, and counts varying from 1 to 10 per experiment. Our architecture, trained in a self-supervised manner, outperforms existing approaches in several biologically relevant downstream tasks. Additionally, it can be used to bridge the gap for the first time between assays with different microscopes, channel numbers or types by embedding various image and experimental modalities into a unified biological image representation. The latter should facilitate interdisciplinary studies and pave the way for better adoption of deep learning in biological image-based analyses. Code and Data available at https://github.com/nicoboou/chadavit.