# CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classification arXiv: https://arxiv.org/abs/2403.16520 ยท HTML: https://mlsanity.com/paper/2403.16520/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 33 references that are in the map. - First posted on arXiv 25 Mar 2024. ## Builds on (most cited first, 21 of 21) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020) - [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024) - [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024) - [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017) - [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024) - [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022) - [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019) - [SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.13560/) (2024) - [SynthStrip: Skull-Stripping for Any Brain Image](https://mlsanity.com/paper/2203.09974/) (2022) - [Advancing Radiograph Representation Learning with Masked Record Modeling](https://mlsanity.com/paper/2301.13155/) (2023) - [MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts](https://mlsanity.com/paper/2401.04081/) (2024) - [nnMamba: 3D Biomedical Image Segmentation, Classification and Landmark Detection with State Space Model](https://mlsanity.com/paper/2402.03526/) (2024) - [BlackMamba: Mixture of Experts for State-Space Models](https://mlsanity.com/paper/2402.01771/) (2024) - [Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations](https://mlsanity.com/paper/2302.14138/) (2023) - [Multi-Modal Hypergraph Diffusion Network with Dual Prior for Alzheimer Classification](https://mlsanity.com/paper/2204.02399/) (2022) - [Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrieval](https://mlsanity.com/paper/2312.15840/) (2023) ## Built on by (most cited first, 15 of 15) - [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024) - [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024) - [Vision Mamba: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2405.04404/) (2024) - [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024) - [Visual Mamba: A Survey and New Outlooks](https://mlsanity.com/paper/2404.18861/) (2024) - [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024) - [Mamba in Vision: A Comprehensive Survey of Techniques and Applications](https://mlsanity.com/paper/2410.03105/) (2024) - [A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond](https://mlsanity.com/paper/2410.02362/) (2024) - [MMR-Mamba: Multi-Modal MRI Reconstruction with Mamba and Spatial-Frequency Information Fusion](https://mlsanity.com/paper/2406.18950/) (2024) - [ADAgent: LLM Agent for Alzheimer's Disease Analysis with Collaborative Coordinator](https://mlsanity.com/paper/2506.11150/) (2025) - [SkinMamba: A Precision Skin Lesion Segmentation Architecture with Cross-Scale Global State Modeling and Frequency Boundary Guidance](https://mlsanity.com/paper/2409.10890/) (2024) - [Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models](https://mlsanity.com/paper/2503.11224/) (2025) - [Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba](https://mlsanity.com/paper/2406.16722/) (2024) - [Focal Cortical Dysplasia Type II Detection Using Cross Modality Transfer Learning and Grad-CAM in 3D-CNNs for MRI Analysis](https://mlsanity.com/paper/2504.07775/) (2025) - [MoCA: Multi-modal Cross-masked Autoencoder for Time Series in Digital Health](https://mlsanity.com/paper/2506.02260/) (2025) ## Abstract (arXiv) Alzheimer's disease (AD) is an incurable neurodegenerative condition leading to cognitive and functional deterioration. Given the lack of a cure, prompt and precise AD diagnosis is vital, a complex process dependent on multiple factors and multi-modal data. While successful efforts have been made to integrate multi-modal representation learning into medical datasets, scant attention has been given to 3D medical images. In this paper, we propose Contrastive Masked Vim Autoencoder (CMViM), the first efficient representation learning method tailored for 3D multi-modal data. Our proposed framework is built on a masked Vim autoencoder to learn a unified multi-modal representation and long-dependencies contained in 3D medical images. We also introduce an intra-modal contrastive learning module to enhance the capability of the multi-modal Vim encoder for modeling the discriminative features in the same modality, and an inter-modal contrastive learning module to alleviate misaligned representation among modalities. Our framework consists of two main steps: 1) incorporate the Vision Mamba (Vim) into the mask autoencoder to reconstruct 3D masked multi-modal data efficiently. 2) align the multi-modal representations with contrastive learning mechanisms from both intra-modal and inter-modal aspects. Our framework is pre-trained and validated ADNI2 dataset and validated on the downstream task for AD classification. The proposed CMViM yields 2.7% AUC performance improvement compared with other state-of-the-art methods.