# Efficient Multi-scale Masked Autoencoders with Hybrid-Attention Mechanism for Breast Lesion Classification

arXiv: https://arxiv.org/abs/2503.07157 · HTML: https://mlsanity.com/paper/2503.07157/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [End-to-end Training for Whole Image Breast Cancer Diagnosis using An All Convolutional Design](https://mlsanity.com/paper/1711.05775/) (2017)
- [Detecting and classifying lesions in mammograms with Deep Learning](https://mlsanity.com/paper/1707.08401/) (2017)
- [CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer](https://mlsanity.com/paper/2112.01330/) (2021)
- [Adding Seemingly Uninformative Labels Helps in Low Data Regimes](https://mlsanity.com/paper/2008.00807/) (2020)

## Abstract (arXiv)

Self-supervised learning (SSL) with Vision Transformers (ViT) has shown immense potential in medical image analysis. However, the quadratic complexity (O(N2)) of standard self-attention poses a severe barrier for high-resolution biomedical tasks, effectively excluding resource-constrained research labs from utilizing state-of-the-art models. To address this computational bottleneck without sacrificing diagnostic accuracy, we propose MIRAM, a Multi-scale Masked Autoencoder that leverages a hybrid-attention mechanism. Our architecture uniquely decouples semantic learning from detail reconstruction using a dual-decoder design: a standard transformer decoder captures global semantics at low resolution, while a linear-complexity decoder (comparing Linformer, Performer, and Nystr"omformer) handles the computationally expensive high-resolution reconstruction. This reduces the complexity of the upscaling stage from quadratic to linear (O(N)), enabling high-fidelity training on consumer-grade GPUs. We validate our approach on the CBIS-DDSM mammography dataset. Remarkably, our Nystr"omformer-based variant achieves a classification accuracy of 61.0%, outperforming both standard MAE (58.9%) and MoCo-v3 (60.2%) while requiring significantly less memory. These results demonstrate that hybrid-attention architectures can democratize high-resolution medical AI, making powerful SSL accessible to researchers with limited hardware resources.