Regional Attention-Enhanced Swin Transformer for Clinically Relevant Medical Image Captioning
Not on the map. Why?
Abstract (arXiv)
Automated medical image captioning translates complex radiological images into diagnostic narratives that can support reporting workflows. We present a Swin-BART encoder-decoder system with a lightweight regional attention module that amplifies diagnostically salient regions before cross-attention. Trained and evaluated on ROCO, our model achieves state-of-the-art semantic fidelity while remaining compact and interpretable. We report results as mean±std over three seeds and include 95% confidence intervals. Compared with baselines, our approach improves ROUGE (proposed 0.603, ResNet-CNN 0.356, BLIP2-OPT 0.255) and BERTScore (proposed 0.807, BLIP2-OPT 0.645, ResNet-CNN 0.623), with competitive BLEU, CIDEr, and METEOR. We further provide ablations (regional attention on/off and token-count sweep), per-modality analysis (CT/MRI/X-ray), paired significance tests, and qualitative heatmaps that visualize the regions driving each description. Decoding uses beam search (beam size =4), length penalty =1.1, norepeatngramsize =3, and max length =128. The proposed design yields accurate, clinically phrased captions and transparent regional attributions, supporting safe research use with a human in the loop.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 20 references that are in the map. First posted on arXiv 13 Nov 2025.
Builds on
10 of 20 references.
References · 10
- 2017Attention Is All You Needcited by 60,361
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
- 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
- 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Textscited by 51
- 2021Advancing Autonomous Driving: DepthSense with Radar and Spatial Attentioncited by 5
- 2024DS@BioMed at ImageCLEFmedical Caption 2024: Enhanced Attention Mechanisms in Medical Caption Generation through Concept Detection Integrationcited by 1
Built on by
Not cited yet.