mlsanityData to 7 Oct 2026

Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation

arXiv 2605.01144 ↗First posted 1 May 2026cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Whole-slide pathology report generation requires models to integrate localized histomorphology, global tissue context, and diagnostically relevant semantic information, yet existing approaches typically rely on fixed pretrained visual representations and may fail to represent key diagnostic concepts. Here we present SCOUT: Semantic Context-aware mOdality fUsion Transformer, a concept-grounded multimodal framework that integrates local histological patterns, whole-slide context, and expert-curated diagnostic concepts. SCOUT maintains an evolving visual representation together with recursively updated slide- and concept-conditioned representations, allowing visual and contextual information to iteratively co-evolve across encoder depth. During decoding, separate attention pathways attend to these complementary streams before an adaptive multimodal fusion module combines them for each generated token. We evaluated SCOUT on TCGA-BRCA, HistAI, and REG-2025, comprising more than 20,900 WSI-report pairs across more than seven cancer types. Using common CONCHv1.5 features and evaluation protocols, SCOUT outperformed WSI-Caption, HistGen, and Bi-Gen, improving BLEU scores by 9.6% on average, METEOR by 11.0%, and ROUGE-L by 3.6% relative to the strongest competing method. On REG-2025, SCOUT additionally improved the Clinical Report Quality Score by 5.6%, indicating better preservation of clinically relevant report attributes beyond lexical similarity. Ablations showed complementary contributions from iterative context refinement and adaptive fusion, while learned gates provided inspectable token-level estimates of modality use. Our results suggest that progressive contextual conditioning is effective across heterogeneous pathology report generation settings and provides a flexible framework for integrating domain knowledge into vision-language models for computational pathology.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

21 of 36 references.

References · 21

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  4. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  5. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  6. 2024Multimodal Whole Slide Foundation Model for Pathologycited by 156
  7. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  8. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 93
  9. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  10. 2025Accelerating Data Processing and Benchmarking of AI Models for Pathologycited by 58
  11. 2020A Generalized Deep Learning Framework for Whole-Slide Image Segmentation and Analysiscited by 45
  12. 2024HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interactioncited by 43
9 more
  1. 2023WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Imagescited by 31
  2. 2023SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathologycited by 25
  3. 2025Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generationcited by 11
  4. 2023Automatic Report Generation for Histopathology images using pre-trained Vision Transformerscited by 9
  5. 2025GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathologycited by 8
  6. 2025Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generationcited by 7
  7. 2025Pathology Report Generation and Multimodal Representation Learning for Cutaneous Melanocytic Lesionscited by 6
  8. 2025LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learningcited by 4
  9. 2025S2D-ALIGN: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PathReportEval: A Systematic Benchmark for Pathology Report Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning
  2. 2025PolyPath: Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation
  3. 2025MPath: Multimodal Pathology Report Generation from Whole Slide Images
  4. 2026A Generative Foundation Model for Multimodal Histopathology
  5. 2026Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation
  6. 2024PathAlign: A vision-language model for whole slide images in histopathology
  7. 2026Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generation
  8. 2026MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding