# Cytoarchitecture in Words: Weakly Supervised Vision-Language Modeling for Human Brain Microscopy

arXiv: https://arxiv.org/abs/2602.23088 · HTML: https://mlsanity.com/paper/2602.23088/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 15 of 15)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [Ministral 3](https://mlsanity.com/paper/2601.08584/) (2026)
- [Text-Only Training for Image Captioning using Noise-Injected CLIP](https://mlsanity.com/paper/2211.00575/) (2022)
- [Unsupervised Image Captioning](https://mlsanity.com/paper/1811.10787/) (2018)
- [UltraMedical: Building Specialized Generalists in Biomedicine](https://mlsanity.com/paper/2406.03949/) (2024)
- [Contour Proposal Networks for Biomedical Instance Segmentation](https://mlsanity.com/paper/2104.03393/) (2021)
- [CytoNet: A Foundation Model for the Human Cerebral Cortex at Cellular Resolution](https://mlsanity.com/paper/2511.01870/) (2025)

## Built on by (most cited first, 1 of 1)

- [MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining](https://mlsanity.com/paper/2610.17866/) (2026)

## Abstract (arXiv)

Vision foundation models increasingly support interactive scientific workflows, but natural-language interaction requires coupling visual representations to language. Curated image-text pairs for this coupling are scarce in many biomedical domains. This is the case for cell-body-stained histological sections of the human brain, where microscopic image patches encode cytoarchitecture: cellular density, morphology, laminar structure, and areal organization. We propose retrieve-and-enrich supervision, a weakly supervised scheme for training image-conditioned language models without curated image-text pairs. The method retrieves label-level text from the literature via a shared anatomical label, then enriches it with image-specific properties such as cortical layer thickness and cell density. Labels provide training targets only, not model inputs. We use this scheme to couple CytoNet, a cytoarchitectonic vision foundation model, to an open-weight large language model via a lightweight Flamingo-style adapter. Across 57 brain areas, the resulting model produces plausible cytoarchitectonic descriptions, supports open-set use by rejecting out-of-scope areas, and predicts the correct area for in-scope patches with 90.6% accuracy. Removing explicit area names from generated text still leaves descriptions sufficient for a language model to recover the area in an 8-way test with 68.6% accuracy. A second instruction-tuned model trained with image-specific targets recovers cortical layer thickness and cell-densities from individual patches, allowing the model to read out information beyond canonical area descriptions. Our results show that retrieve-and-enrich supervision offers a practical route to vision-language training in specialized imaging domains where labels and expert text exist but image-level captions do not.