# TWLR: Text-Guided Weakly-Supervised Lesion Localization and Severity Regression for Explainable Diabetic Retinopathy Grading

arXiv: https://arxiv.org/abs/2512.13008 · HTML: https://mlsanity.com/paper/2512.13008/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.00962/) (2022)
- [A Benchmark for Studying Diabetic Retinopathy: Segmentation, Grading, and Transferability](https://mlsanity.com/paper/2008.09772/) (2020)
- [CANet: Cross-disease Attention Network for Joint Diabetic Retinopathy and Diabetic Macular Edema Grading](https://mlsanity.com/paper/1911.01376/) (2019)
- [CANet: An Unsupervised Intrusion Detection System for High Dimensional CAN Bus Data](https://mlsanity.com/paper/1906.02492/) (2019)
- [Lesion-based Contrastive Learning for Diabetic Retinopathy Grading from Fundus Images](https://mlsanity.com/paper/2107.08274/) (2021)
- [Weakly-supervised localization of diabetic retinopathy lesions in retinal fundus images](https://mlsanity.com/paper/1706.09634/) (2017)
- [CLIP-DR: Textual Knowledge-Guided Diabetic Retinopathy Grading with Ranking-aware Prompting](https://mlsanity.com/paper/2407.04068/) (2024)
- [Weakly-supervised Medical Image Segmentation with Gaze Annotations](https://mlsanity.com/paper/2407.07406/) (2024)
- [WeakMedSAM: Weakly-Supervised Medical Image Segmentation via SAM with Sub-Class Exploration and Prompt Affinity Mining](https://mlsanity.com/paper/2503.04106/) (2025)
- [Iterative Augmentation of Visual Evidence for Weakly-Supervised Lesion Localization in Deep Interpretability Frameworks: Application to Color Fundus Images](https://mlsanity.com/paper/1910.07373/) (2019)
- [Weakly-supervised detection of AMD-related lesions in color fundus images using explainable deep learning](https://mlsanity.com/paper/2212.00565/) (2022)
- [Hierarchical Salient Patch Identification for Interpretable Fundus Disease Localization](https://mlsanity.com/paper/2405.14334/) (2024)

## Abstract (arXiv)

Accurate medical image analysis can greatly assist clinical diagnosis, but its effectiveness relies on high-quality expert annotations Obtaining pixel-level labels for medical images, particularly fundus images, remains costly and time-consuming. Meanwhile, despite the success of deep learning in medical imaging, the lack of interpretability limits its clinical adoption. To address these challenges, we propose TWLR, a two-stage framework for interpretable diabetic retinopathy (DR) assessment. In the first stage, a vision-language model integrates domain-specific ophthalmological knowledge into text embeddings to jointly perform DR grading and lesion classification, effectively linking semantic medical concepts with visual features. The second stage introduces an iterative severity regression framework based on weakly-supervised semantic segmentation. Lesion saliency maps generated through iterative refinement direct a progressive inpainting mechanism that systematically eliminates pathological features, effectively downgrading disease severity toward healthier fundus appearances. Critically, this severity regression approach achieves dual benefits: accurate lesion localization without pixel-level supervision and providing an interpretable visualization of disease-to-healthy transformations. Experimental results on the FGADR, DDR, and a private dataset demonstrate that TWLR achieves competitive performance in both DR classification and lesion segmentation, offering a more explainable and annotation-efficient solution for automated retinal image analysis.