# CytoCLIP: Learning Cytoarchitectural Characteristics in Developing Human Brain Using Contrastive Language Image Pre-Training

arXiv: https://arxiv.org/abs/2601.12282 · HTML: https://mlsanity.com/paper/2601.12282/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 18 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Images](https://mlsanity.com/paper/2111.04740/) (2021)
- [Machine learning methods for histopathological image analysis](https://mlsanity.com/paper/1709.00786/) (2017)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Improving Cytoarchitectonic Segmentation of Human Brain Areas with Self-supervised Siamese Networks](https://mlsanity.com/paper/1806.05104/) (2018)
- [Contrastive Representation Learning for Whole Brain Cytoarchitectonic Mapping in Histological Human Brain Sections](https://mlsanity.com/paper/2011.12865/) (2020)
- [A State-of-the-art Survey of Artificial Neural Networks for Whole-slide Image Analysis:from Popular Convolutional Neural Networks to Potential Visual Transformers](https://mlsanity.com/paper/2104.06243/) (2021)
- [Parcellation of Visual Cortex on high-resolution histological Brain Sections using Convolutional Neural Networks](https://mlsanity.com/paper/1705.10545/) (2017)
- [Convolutional Neural Networks for cytoarchitectonic brain mapping at large scale](https://mlsanity.com/paper/2011.12857/) (2020)
- [Histopathology Image Classification using Deep Manifold Contrastive Learning](https://mlsanity.com/paper/2306.14459/) (2023)
- [Language Augmentation in CLIP for Improved Anatomy Detection on Multi-modal Medical Images](https://mlsanity.com/paper/2405.20735/) (2024)
- [PosDiffAE: Position-aware Diffusion Auto-encoder For High-Resolution Brain Tissue Classification Incorporating Artifact Restoration](https://mlsanity.com/paper/2507.02405/) (2025)

## Abstract (arXiv)

The functions of different regions of the human brain are closely linked to their distinct cytoarchitecture, which is defined by the spatial arrangement and morphology of the cells. Identifying brain regions by their cytoarchitecture enables various scientific analyses of the brain. However, delineating these areas manually in brain histological sections is time-consuming and requires specialized knowledge. An automated approach is necessary to minimize the effort needed from human experts. To address this, we propose CytoCLIP, a suite of vision-language models derived from pre-trained Contrastive Language-Image Pre-Training (CLIP) frameworks to learn joint visual-text representations of brain cytoarchitecture. CytoCLIP comprises two model variants: one is trained using low-resolution whole-region images to understand the overall cytoarchitectural pattern of an area, and the other is trained on high-resolution image tiles for detailed cellular-level representation. The training dataset is created from NISSL-stained histological sections of developing fetal brains of different gestational weeks. It includes 86 distinct regions for low-resolution images and 379 brain regions for high-resolution tiles. We evaluate the model's understanding of the cytoarchitecture and generalization ability using region classification and cross-modal retrieval tasks. Multiple experiments are performed under various data setups, including data from samples of different ages and sectioning planes. Experimental results demonstrate that CytoCLIP outperforms existing methods. It achieves a weighted F1 score of 0.87 for whole-region classification and 0.91 for high-resolution image tile classification.