# Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms

arXiv: https://arxiv.org/abs/2604.19350 · HTML: https://mlsanity.com/paper/2604.19350/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography](https://mlsanity.com/paper/2203.11205/) (2022)
- [Are Natural Domain Foundation Models Useful for Medical Image Classification?](https://mlsanity.com/paper/2310.19522/) (2023)
- [Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography](https://mlsanity.com/paper/2405.12255/) (2024)
- [MedVAE: Efficient Automated Interpretation of Medical Images with Large-Scale Generalizable Autoencoders](https://mlsanity.com/paper/2502.14753/) (2025)
- [XFMamba: Cross-Fusion Mamba for Multi-View Medical Image Classification](https://mlsanity.com/paper/2503.02619/) (2025)
- [Boltzmann Attention Sampling for Image Analysis with Small Objects](https://mlsanity.com/paper/2503.02841/) (2025)
- [TransReg: Cross-transformer as auto-registration module for multi-view mammogram mass detection](https://mlsanity.com/paper/2311.05192/) (2023)

## Built on by (most cited first, 1 of 1)

- [M3D-Net: Hierarchical Coordination of Spatial Context, Feature Reuse, and Differential Attention for Mammography Classification](https://mlsanity.com/paper/2609.27523/) (2026)

## Abstract (arXiv)

Vision Transformers (textttViT) have become the architecture of choice for many computer vision tasks, yet their performance in computer-aided diagnostics remains limited. Focusing on breast cancer detection from mammograms, we identify two main causes for this shortfall. First, medical images are high-resolution with small abnormalities, leading to an excessive number of tokens and making it difficult for the softmax-based attention to localize and attend to relevant regions. Second, medical image classification is inherently fine-grained, with low inter-class and high intra-class variability, where standard cross-entropy training is insufficient. To overcome these challenges, we propose a framework with three key components: (1) Region of interest (textttRoI) based token reduction using an object detection model to guide attention; (2) contrastive learning between selected textttRoI to enhance fine-grained discrimination through hard-negative based training; and (3) a textttDINOv2 pretrained textttViT that captures localization-aware, fine-grained features instead of global textttCLIP representations. Experiments on public mammography datasets demonstrate that our method achieves superior performance over existing baselines, establishing its effectiveness and potential clinical utility for large-scale breast cancer screening. Our code is available for reproducibility here: https://aih-iitd.github.io/publications/attend-what-matters