mlsanityData to 7 Oct 2026

Multi-modal Deep Learning

arXiv 2403.03385 ↗First posted 6 Mar 2024eess.IV cs.AI cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

This article investigates deep learning methodologies for single-modality clinical data analysis, as a crucial precursor to multi-modal medical research. Building on Guo JingYuan's work, the study refines clinical data processing through Compact Convolutional Transformer (CCT), Patch Up, and the innovative CamCenterLoss technique, establishing a foundation for future multimodal investigations. The proposed methodology demonstrates improved prediction accuracy and at tentiveness to critically ill patients compared to Guo JingYuan's ResNet and StageNet approaches. Novelty that using image-pretrained vision transformer backbone to perform transfer learning time-series clinical data.The study highlights the potential of CCT, Patch Up, and novel CamCenterLoss in processing single modality clinical data within deep learning frameworks, paving the way for future multimodal medical research and promoting precision and personalized healthcare

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

15 of 30 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  6. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 398
  7. 2017Multitask learning and benchmarking with clinical time series datacited by 287
  8. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  9. 2021Task-Oriented Multi-User Semantic Communicationscited by 72
  10. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  11. 2021Is it Time to Replace CNNs with Transformers for Medical Images?cited by 60
  12. 2020StageNet: Stage-Aware Neural Networks for Health Risk Predictioncited by 47
3 more
  1. 2021COVID-VIT: Classification of COVID-19 from CT chest images based on vision transformer modelscited by 27
  2. 2021Multi-domain Integrative Swin Transformer network for Sparse-View Tomographic Reconstructioncited by 15
  3. 2021Vision Transformer for COVID-19 CXR Diagnosis using Chest X-ray Feature Corpuscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024What to align in multimodal contrastive learning?cited by 41
  2. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  3. 2025Learning Musical Representations for Music Performance Question Answeringcited by 17
  4. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
  5. 2025Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Expertscited by 4
  6. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Modelscited by 3
  7. 2026Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classificationcited by 3
  8. 2024Hybrid Quantum-inspired Resnet and Densenet for Pattern Recognitioncited by 1
  9. 2025Automatic Detection of Intro and Credits in Video using CLIP and Multihead Attentioncited by 1
  10. 2025ICYM2I: The illusion of multimodal informativeness under missingnesscited by 1
  11. 2026The Mouth is Not the Brain: Bridging Energy-Based World Models and Language Generationcited by 1
  12. 2025MODIS: Multi-Omics Data Integration for Small and unpaired datasetscited by 0
3 more
  1. 2025PREMISE: Matching-based Prediction for Accurate Review Recommendationcited by 0
  2. 2026Face and Voice Cross-modal Association with Learning Convex Feature Embeddingcited by 0
  3. 2026Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classificationcited by 0