mlsanityData to 7 Oct 2026

HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis

arXiv 2406.16192 ↗First posted 23 Jun 2024cs.CVcited by 104

Not on the map. Why?

Abstract (arXiv)

Spatial transcriptomics enables interrogating the molecular composition of tissue with ever-increasing resolution and sensitivity. However, costs, rapidly evolving technology, and lack of standards have constrained computational methods in ST to narrow tasks and small cohorts. In addition, the underlying tissue morphology, as reflected by H&E-stained whole slide images (WSIs), encodes rich information often overlooked in ST studies. Here, we introduce HEST-1k, a collection of 1,229 spatial transcriptomic profiles, each linked to a WSI and extensive metadata. HEST-1k was assembled from 153 public and internal cohorts encompassing 26 organs, two species (Homo Sapiens and Mus Musculus), and 367 cancer samples from 25 cancer types. HEST-1k processing enabled the identification of 2.1 million expression--morphology pairs and over 76 million nuclei. To support its development, we additionally introduce the HEST-Library, a Python package designed to perform a range of actions with HEST samples. We test HEST-1k and Library on three use cases: (1) benchmarking foundation models for pathology (HEST-Benchmark), (2) biomarker exploration, and (3) multimodal representation learning. HEST-1k, HEST-Library, and HEST-Benchmark can be freely accessed at https://github.com/mahmoodlab/hest.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Builds on 45 of its 176 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

45 of 176 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  9. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
33 more
  1. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  2. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  3. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  4. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  5. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  6. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  7. 2020Self supervised contrastive learning for digital histopathologycited by 146
  8. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  9. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  10. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
  11. 2021BRACS: A Dataset for BReAst Carcinoma Subtyping in H&E Histology Imagescited by 130
  12. 2020PanNuke Dataset Extension, Insights and Baselinescited by 113
  13. 2023CellViT: Vision Transformers for Precise Cell Segmentation and Classificationcited by 108
  14. 2023Artificial Intelligence for Digital and Computational Pathologycited by 106
  15. 2020Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primarycited by 89
  16. 2020Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotationscited by 89
  17. 2023Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learningcited by 77
  18. 2021Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detectioncited by 66
  19. 2021A Petri Dish for Histopathology Image Analysiscited by 63
  20. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  21. 2021Fine-Tuning and Training of DenseNet for Histopathology Image Representation Using TCGA Diagnostic Slidescited by 52
  22. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  23. 2024Accurate Spatial Gene Expression Prediction by integrating Multi-resolution featurescited by 52
  24. 2024Transcriptomics-guided Slide Representation Learning in Computational Pathologycited by 50
  25. 2021Predicting Axillary Lymph Node Metastasis in Early Breast Cancer Using Deep Learning on Primary Tumor Biopsy Slidescited by 43
  26. 2021UniToPatho, a labeled histopathological dataset for colorectal polyps classification and adenoma dysplasia gradingcited by 33
  27. 2023Regression-based Deep-Learning predicts molecular biomarkers from pathology slidescited by 15
  28. 2022Learning to Predict RNA Sequence Expressions from Whole Slide Images with Applications for Search and Classificationcited by 14
  29. 2021DiagSet: a dataset for prostate cancer histopathological image classificationcited by 13
  30. 2023hist2RNA: An efficient deep learning architecture to predict gene expression from breast cancer histopathology imagescited by 10
  31. 2022Unsupervised Visual Representation Learning via Mutual Information Regularized Assignmentcited by 7
  32. 2024M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Imagescited by 7
  33. 2023Breast Cancer Histopathology Image based Gene Expression Prediction using Spatial Transcriptomics data and Deep Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 44; 2026: 53.

104 papers.

Sort:

By area:

  1. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  2. 2025Diffusion Generative Modeling for Spatially Resolved Gene Expression Inference from Histology Imagescited by 35
  3. 2025Distilling foundation models for robust and efficient models in digital pathologycited by 33
  4. 2024Multistain Pretraining for Slide Representation Learning in Pathologycited by 27
  5. 2025Training state-of-the-art pathology foundation models with orders of magnitude less datacited by 19
  6. 2025Scalable Generation of Spatial Transcriptomics from Histology Images via Whole-Slide Flow Matchingcited by 18
  7. 2024MERGE: Multi-faceted Hierarchical Graph-based GNN for Gene Expression Prediction from Whole Slide Histopathology Imagescited by 17
  8. 2025Flow Matching Meets Biology and Life Science: A Surveycited by 14
  9. 2025Atlas: A Novel Pathology Foundation Model by Mayo Clinic, Charit'e, and Aignosticscited by 10
  10. 2025PathOrchestra: A Comprehensive Foundation Model for Computational Pathology with Over 100 Diverse Clinical-Grade Taskscited by 8
  11. 2025SToFM: a Multi-scale Foundation Model for Spatial Transcriptomicscited by 8
  12. 2025A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomicscited by 8
38 more
  1. 2025Spatially Gene Expression Prediction using Dual-Scale Contrastive Learningcited by 7
  2. 2025MeDi: Metadata-Guided Diffusion Models for Mitigating Biases in Tumor Classificationcited by 6
  3. 2025From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Researchcited by 6
  4. 2025MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncologycited by 6
  5. 2026Atlas 2 -- Foundation models for clinical deploymentcited by 6
  6. 2024Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomicscited by 5
  7. 2025Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challengescited by 5
  8. 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilotscited by 5
  9. 2026HistoPrism: Unlocking Functional Pathway Analysis from Pan-Cancer Histology via Gene Expression Predictioncited by 5
  10. 2024RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistencycited by 4
  11. 2025PH2ST:ST-Prompt Guided Histological Hypergraph Learning for Spatial Gene Expression Predictioncited by 3
  12. 2025Teaching pathology foundation models to accurately predict gene expression with parameter efficient knowledge transfercited by 3
  13. 2025Completing Spatial Transcriptomics Data for Gene Expression Prediction Benchmarkingcited by 3
  14. 2025Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learningcited by 3
  15. 2025SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Spacecited by 3
  16. 2025GenAR: Next-Scale Autoregressive Generation for Spatial Gene Expression Predictioncited by 3
  17. 2025Learning Relative Gene Expression Trends from Pathology Images in Spatial Transcriptomicscited by 3
  18. 2025PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantizationcited by 2
  19. 2025PEaRL: Pathway-Enhanced Representation Learning for Gene and Pathway Expression Prediction from Histologycited by 2
  20. 2025GeneFlow: Translation of Single-cell Gene Expression to Histopathological Images via Rectified Flowcited by 2
  21. 2025PLUTO-4: Frontier Pathology Foundation Modelscited by 2
  22. 2026LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathologycited by 2
  23. 2026NuNext: Reframing Nucleus Detection as Next-Point Detectioncited by 2
  24. 2026Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathologycited by 2
  25. 2026Adapting a Pre-trained Single-Cell Foundation Model to Spatial Gene Expression Generation from Histology Imagescited by 2
  26. 2026Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Stainingcited by 2
  27. 2024From Pixels to Gigapixels: Bridging Local Inductive Bias and Long-Range Dependencies with Pixel-Mambacited by 1
  28. 2025Unifying Multiple Foundation Models for Advanced Computational Pathologycited by 1
  29. 2025DELST: Dual Entailment Learning for Hyperbolic Image-Gene Pretraining in Spatial Transcriptomicscited by 1
  30. 2025Towards Spatial Transcriptomics-guided Pathological Image Recognition with Batch-Agnostic Encodercited by 1
  31. 2025Gene-DML: Dual-Pathway Multi-Level Discrimination for Gene Expression Prediction from Histopathology Imagescited by 1
  32. 2025MVHybrid: Improving Spatial Transcriptomics Prediction with Hybrid State Space-Vision Transformer Backbone in Pathology Vision Foundation Modelscited by 1
  33. 2025Towards Cellular-Scale Interpretability in Pathology Foundation Models for Biomarker Assessmentcited by 1
  34. 2025SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptomecited by 1
  35. 2025Auxiliary Gene Learning: Spatial Gene Expression Estimation by Auxiliary Gene Selectioncited by 1
  36. 2025Democratising Pathology Co-Pilots: An Open Pipeline and Dataset for Whole-Slide Vision-Language Modellingcited by 1
  37. 2026PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classificationcited by 1
  38. 2026MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Modelscited by 1

50 most cited of 104 All 104 (JSON)