# Application of Multimodal Fusion Deep Learning Model in Disease Recognition

arXiv: https://arxiv.org/abs/2406.18546 · HTML: https://mlsanity.com/paper/2406.18546/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 15 of 15)

- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Survival Prediction Across Diverse Cancer Types Using Neural Networks](https://mlsanity.com/paper/2404.08713/) (2024)
- [Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks](https://mlsanity.com/paper/2311.10944/) (2023)
- [Convolutional neural network classification of cancer cytopathology images: taking breast cancer as an example](https://mlsanity.com/paper/2404.08279/) (2024)
- [Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation](https://mlsanity.com/paper/2306.04811/) (2023)
- [Exploring Diverse Methods in Visual Question Answering](https://mlsanity.com/paper/2404.13565/) (2024)
- [Improving Depth Gradient Continuity in Transformers: A Comparative Study on Monocular Depth Estimation with CNN](https://mlsanity.com/paper/2308.08333/) (2023)
- [Regional Style and Color Transfer](https://mlsanity.com/paper/2404.13880/) (2024)
- [3D-Aware Neural Body Fitting for Occlusion Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2308.10123/) (2023)
- [Visualizing and Understanding Patch Interactions in Vision Transformer](https://mlsanity.com/paper/2203.05922/) (2022)
- [An Optimization-Based Meta-Learning Model for MRI Reconstruction with Diverse Dataset](https://mlsanity.com/paper/2110.00715/) (2021)
- [Constrained Attractor Selection Using Deep Reinforcement Learning](https://mlsanity.com/paper/1909.10500/) (2019)
- [Attractor Selection in Nonlinear Energy Harvesting Using Deep Reinforcement Learning](https://mlsanity.com/paper/2010.01255/) (2020)
- [DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era](https://mlsanity.com/paper/2404.12278/) (2024)

## Built on by (most cited first, 12 of 12)

- [Identification of Prognostic Biomarkers for Stage III Non-Small Cell Lung Carcinoma in Female Nonsmokers Using Machine Learning](https://mlsanity.com/paper/2408.16068/) (2024)
- [Attention Mechanism and Context Modeling System for Text Mining Machine Translation](https://mlsanity.com/paper/2408.04216/) (2024)
- [Dynamic Hypergraph-Enhanced Prediction of Sequential Medical Visits](https://mlsanity.com/paper/2408.07084/) (2024)
- [A Lightweight GAN-Based Image Fusion Algorithm for Visible and Infrared Images](https://mlsanity.com/paper/2409.15332/) (2024)
- [Advanced Payment Security System:XGBoost, LightGBM and SMOTE Integrated](https://mlsanity.com/paper/2406.04658/) (2024)
- [Text classification optimization algorithm based on graph neural network](https://mlsanity.com/paper/2408.15257/) (2024)
- [Convolutional Neural Networks for Predictive Modeling of Lung Disease](https://mlsanity.com/paper/2408.12605/) (2024)
- [Reducing Bias in Deep Learning Optimization: The RSGDM Approach](https://mlsanity.com/paper/2409.15314/) (2024)
- [Comprehensive Overview of Artificial Intelligence Applications in Modern Industries](https://mlsanity.com/paper/2409.13059/) (2024)
- [Deep Learning-Based Channel Squeeze U-Structure for Lung Nodule Detection and Segmentation](https://mlsanity.com/paper/2409.13868/) (2024)
- [Stock Type Prediction Model Based on Hierarchical Graph Neural Network](https://mlsanity.com/paper/2412.06862/) (2024)
- [Enhanced Credit Score Prediction Using Ensemble Deep Learning Model](https://mlsanity.com/paper/2410.00256/) (2024)

## Abstract (arXiv)

This paper introduces an innovative multi-modal fusion deep learning approach to overcome the drawbacks of traditional single-modal recognition techniques. These drawbacks include incomplete information and limited diagnostic accuracy. During the feature extraction stage, cutting-edge deep learning models including convolutional neural networks (CNN), recurrent neural networks (RNN), and transformers are applied to distill advanced features from image-based, temporal, and structured data sources. The fusion strategy component seeks to determine the optimal fusion mode tailored to the specific disease recognition task. In the experimental section, a comparison is made between the performance of the proposed multi-mode fusion model and existing single-mode recognition methods. The findings demonstrate significant advantages of the multimodal fusion model across multiple evaluation metrics.