mlsanityData to 7 Oct 2026

PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology

arXiv 2608.25970 ↗First posted 26 Aug 2026cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Multimodal medical prediction often faces incomplete pairing: auxiliary modalities with complementary signal are available for only a subset of subjects (or none) and cannot be assumed at deployment. We introduce PANDA (Prototype Anchored Data Alignment), a two-stage framework that transfers auxiliary information to a primary-modality model without auxiliary inputs at inference. Stage 1 learns a shared embedding from the paired subset and estimates class prototypes from auxiliary modalities; Stage 2 trains the primary encoder on all subjects using cross-entropy plus alignment to the frozen prototypes. Because supervision is defined at the class-prototype level, PANDA accommodates arbitrary pairing rates, including zero subject overlap. We evaluate PANDA on two applications. On a 1,021-subject multi-scanner ADNI cohort, we perform AD/CN classification with three auxiliary modalities at distinct pairing rates: tabular scores (44.8%), FDG-PET (18.7%), and external handwriting kinematics (0% overlap). Relative to the same-backbone MRI-only baseline, PANDA attains AUC 0.868 +-0.020 (+7.9pp) and reduces 1.5T CN false positives by 24.3pp; on a fully trainable Conv5-FC3 backbone it reaches AUC 0.893 (best overall). A pairing-rate ablation shows that the joint anchor remains within seed noise from 75% to 5% pairing. On TCGA-Lung survival prediction from whole-slide images with RNA-seq as auxiliary data, PANDA improves over WSI-only on 2-year OS (AUC +3.5pp) and Cox PH (C-index +9.0pts) and outperforms full-fusion training, which underperforms WSI-only, while requiring no RNA at inference; wide confidence intervals on this smaller cohort keep the gains below conventional significance. Overall, PANDA provides a deployment-oriented mechanism for leveraging incomplete auxiliary modalities to improve primary-modality prediction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 47 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

25 of 47 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  7. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  8. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  9. 2019On Variational Bounds of Mutual Informationcited by 550
  10. 2022MONAI: An open-source framework for deep learning in healthcarecited by 438
  11. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 282
  12. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
13 more
  1. 2019Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptationcited by 235
  2. 2020Self-Supervision with Superpixels: Training Few-shot Medical Image Segmentation without Annotationcited by 128
  3. 2019Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluationcited by 106
  4. 2016HeMIS: Hetero-Modal Image Segmentationcited by 102
  5. 2020A unifying mutual information view of metric learning: cross-entropy vs. pairwise lossescited by 100
  6. 2019Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Networkcited by 98
  7. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  8. 2019Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effectscited by 40
  9. 2024MoME: Mixture of Multimodal Experts for Cancer Survival Predictioncited by 23
  10. 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
  11. 2024HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modelingcited by 15
  12. 2024DiaMond: Dementia Diagnosis with Multi-Modal Vision Transformers Using MRI and PETcited by 6
  13. 2024A Progressive Single-Modality to Multi-Modality Classification Framework for Alzheimer's Disease Sub-type Diagnosiscited by 3

Built on by

Not cited yet.