# 4D Multimodal Co-attention Fusion Network with Latent Contrastive Alignment for Alzheimer's Diagnosis

arXiv: https://arxiv.org/abs/2504.16798 · HTML: https://mlsanity.com/paper/2504.16798/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation](https://mlsanity.com/paper/2206.02425/) (2022)
- [SwiFT: Swin 4D fMRI Transformer](https://mlsanity.com/paper/2307.05916/) (2023)
- [OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All](https://mlsanity.com/paper/2405.16108/) (2024)
- [A Deep Network for Explainable Prediction of Non-Imaging Phenotypes using Anatomical Multi-View Data](https://mlsanity.com/paper/2401.04579/) (2024)
- [Hierarchical Spatio-Temporal State-Space Modeling for fMRI Analysis](https://mlsanity.com/paper/2408.13074/) (2024)
- [Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment](https://mlsanity.com/paper/2312.15064/) (2023)
- [Multimodal MRI Accurately Identifies Amyloid Status in Unbalanced Cohorts in Alzheimer's Disease Continuum](https://mlsanity.com/paper/2406.13305/) (2024)
- [Turbo your multi-modal classification with contrastive learning](https://mlsanity.com/paper/2409.09282/) (2024)
- [Learning 3D Medical Image Models From Brain Functional Connectivity Network Supervision For Mental Disorder Diagnosis](https://mlsanity.com/paper/2503.04205/) (2025)
- [A deep spatio-temporal attention model of dynamic functional network connectivity shows sensitivity to Alzheimer's in asymptomatic individuals](https://mlsanity.com/paper/2408.00378/) (2024)

## Built on by (most cited first, 3 of 3)

- [fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding](https://mlsanity.com/paper/2511.21760/) (2025)
- [BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics](https://mlsanity.com/paper/2606.17742/) (2026)
- [BrainMAP: Multimodal Graph Learning For Efficient Brain Disease Localization](https://mlsanity.com/paper/2506.11178/) (2025)

## Abstract (arXiv)

Multimodal neuroimaging provides complementary structural and functional insights into both human brain organization and disease-related dynamics. Recent studies demonstrate enhanced diagnostic sensitivity for Alzheimer's disease (AD) through synergistic integration of neuroimaging data (e.g., sMRI, fMRI) with behavioral cognitive scores tabular data biomarkers. However, the intrinsic heterogeneity across modalities (e.g., 4D spatiotemporal fMRI dynamics vs. 3D anatomical sMRI structure) presents critical challenges for discriminative feature fusion. To bridge this gap, we propose M2M-AlignNet: a geometry-aware multimodal co-attention network with latent alignment for early AD diagnosis using sMRI and fMRI. At the core of our approach is a multi-patch-to-multi-patch (M2M) contrastive loss function that quantifies and reduces representational discrepancies via geometry-weighted patch correspondence, explicitly aligning fMRI components across brain regions with their sMRI structural substrates without one-to-one constraints. Additionally, we propose a latent-as-query co-attention module to autonomously discover fusion patterns, circumventing modality prioritization biases while minimizing feature redundancy. We conduct extensive experiments to confirm the effectiveness of our method and highlight the correspondance between fMRI and sMRI as AD biomarkers.