mlsanityData to 14 Oct 2026

UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment

arXiv 2608.10316 ↗First posted 10 Aug 2026cs.CV cs.LG cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-modal learning combining medical images and clinical text is promising for disease diagnosis. However, standard multi-modal training leads to shortcut learning: models exploit the easier modality (e.g., diagnostic cues in text) while neglecting harder-to-learn features (e.g., subtle visual patterns). We propose UniMod, a framework that mitigates shortcut learning by requiring each modality to predict the diagnosis on its own. It supervises image-only, text-only, and multi-modal classification simultaneously, so each modality must extract diagnostic features. We add cross-modality alignment for knowledge transfer and within-modality supervised contrastive alignment over same-diagnosis patients. On Harvard-Glaucoma, UniMod reaches 0.850 AUC, outperforming OGM-GE and Gradient Blending by 1.6-1.8%; on CheXpert Plus, it reaches 0.966 AUC, surpassing them by over 5%. UniMod also extends to 5-class multi-label diagnosis without architectural change, improving mean AUC by 0.097 over CGGM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 70 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

44 of 70 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
32 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  11. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  13. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  14. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  15. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  16. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  17. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  18. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  19. 2017Deep Mutual Learningcited by 628
  20. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  21. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 461
  22. 2022Multimodal Learning with Transformers: A Surveycited by 400
  23. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  24. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  25. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  26. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 106
  27. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  28. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-trainingcited by 82
  29. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 35
  30. 2025Learning to Estimate Package Delivery Time in Mixed Imbalanced Delivery and Pickup Logistics Servicescited by 5
  31. 2025DeepSTA: A Spatial-Temporal Attention Network for Logistics Delivery Timely Rate Prediction in Anomaly Conditionscited by 4
  32. 2026TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Modelscited by 2

Built on by

Not cited yet.