# Lesion Guided Explainable Few Weak-shot Medical Report Generation

arXiv: https://arxiv.org/abs/2211.08732 · HTML: https://mlsanity.com/paper/2211.08732/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Clinically Accurate Chest X-Ray Report Generation](https://mlsanity.com/paper/1904.02633/) (2019)
- [Contrastive Embedding for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2103.16173/) (2021)
- [Transferable Contrastive Network for Generalized Zero-Shot Learning](https://mlsanity.com/paper/1908.05832/) (2019)
- [Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports](https://mlsanity.com/paper/2004.12274/) (2020)
- [More Grounded Image Captioning by Distilling Image-Text Matching Model](https://mlsanity.com/paper/2004.00390/) (2020)
- [Zero-Shot Detection](https://mlsanity.com/paper/1803.07113/) (2018)
- [Weak-shot Fine-grained Classification via Similarity Transfer](https://mlsanity.com/paper/2009.09197/) (2020)
- [Weak-shot Semantic Segmentation by Transferring Semantic Affinity and Boundary](https://mlsanity.com/paper/2110.01519/) (2021)

## Built on by (most cited first, 6 of 6)

- [ChEX: Interactive Localization and Region Description in Chest X-rays](https://mlsanity.com/paper/2404.15770/) (2024)
- [Cross-Modal Causal Intervention for Medical Report Generation](https://mlsanity.com/paper/2303.09117/) (2023)
- [A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Data](https://mlsanity.com/paper/2405.12833/) (2024)
- [A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Models](https://mlsanity.com/paper/2508.03734/) (2025)
- [Predicting Diabetic Macular Edema Treatment Responses Using OCT: Dataset and Methods of APTOS Competition](https://mlsanity.com/paper/2505.05768/) (2025)
- [Visual Question Answering in Ophthalmology: A Progressive and Practical Perspective](https://mlsanity.com/paper/2410.16662/) (2024)

## Abstract (arXiv)

Medical images are widely used in clinical practice for diagnosis. Automatically generating interpretable medical reports can reduce radiologists' burden and facilitate timely care. However, most existing approaches to automatic report generation require sufficient labeled data for training. In addition, the learned model can only generate reports for the training classes, lacking the ability to adapt to previously unseen novel diseases. To this end, we propose a lesion guided explainable few weak-shot medical report generation framework that learns correlation between seen and novel classes through visual and semantic feature alignment, aiming to generate medical reports for diseases not observed in training. It integrates a lesion-centric feature extractor and a Transformer-based report generation module. Concretely, the lesion-centric feature extractor detects the abnormal regions and learns correlations between seen and novel classes with multi-view (visual and lexical) embeddings. Then, features of the detected regions and corresponding embeddings are concatenated as multi-view input to the report generation module for explainable report generation, including text descriptions and corresponding abnormal regions detected in the images. We conduct experiments on FFA-IR, a dataset providing explainable annotations, showing that our framework outperforms others on report generation for novel diseases.