# Towards Interpretable Foundation Models for Retinal Fundus Images

arXiv: https://arxiv.org/abs/2603.18846 · HTML: https://mlsanity.com/paper/2603.18846/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Builds on (most cited first, 9 of 9)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation](https://mlsanity.com/paper/2206.03208/) (2022)
- [RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports](https://mlsanity.com/paper/2405.14137/) (2024)
- [Unsupervised visualization of image datasets using contrastive learning](https://mlsanity.com/paper/2210.09879/) (2022)
- [Training a high-performance retinal foundation model with half-the-data and 400 times less compute](https://mlsanity.com/paper/2405.00117/) (2024)

## Abstract (arXiv)

Foundation models are used to extract transferable representations from large amounts of unlabeled data, typically via self-supervised learning (SSL). However, many of these models rely on architectures that offer limited interpretability, a critical issue in high-stakes domains such as medical imaging. We propose DualIFM, a foundation model that is interpretable-by-design via a BagNet backbone whose small receptive fields generate class evidence maps that are faithful to the model's decision-making process. Additionally, DualIFM incorporates a 2D projection layer during pretraining that enables direct visualization of the representation space, providing a dataset-level view of the learned structure including meaningful clinical clusters as well as potential spurious correlations. We trained DualIFM on over 800,000 color fundus photographs from various sources to learn generalizable representations for different downstream tasks. Our model achieves performance comparable to RETFound, which has 16× more parameters, while providing interpretable predictions on out-of-distribution data. These results suggest that large-scale SSL pretraining paired with inherent interpretability can lead to robust representations for retinal imaging. Code and pretrained models are available at github.com/berenslab/interpretableFM.