# DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era

arXiv: https://arxiv.org/abs/2404.12278 · HTML: https://mlsanity.com/paper/2404.12278/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 106 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [Survival Prediction Across Diverse Cancer Types Using Neural Networks](https://mlsanity.com/paper/2404.08713/) (2024)
- [Using satellite imagery to understand and promote sustainable development](https://mlsanity.com/paper/2010.06988/) (2020)
- [Convolutional neural network classification of cancer cytopathology images: taking breast cancer as an example](https://mlsanity.com/paper/2404.08279/) (2024)
- [Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction](https://mlsanity.com/paper/2401.01646/) (2024)
- [A Survey of Multimodal Information Fusion for Smart Healthcare: Mapping the Journey from Data to Wisdom](https://mlsanity.com/paper/2306.11963/) (2023)
- [Data-Centric Engineering: integrating simulation, machine learning and statistics. Challenges and Opportunities](https://mlsanity.com/paper/2111.06223/) (2021)
- [Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision](https://mlsanity.com/paper/2304.00379/) (2023)
- [Google Trends Analysis of COVID-19](https://mlsanity.com/paper/2011.03847/) (2020)

## Built on by (most cited first, 9 of 9)

- [Application of Multimodal Fusion Deep Learning Model in Disease Recognition](https://mlsanity.com/paper/2406.18546/) (2024)
- [A comparative study of generative adversarial networks for image recognition algorithms based on deep learning and traditional methods](https://mlsanity.com/paper/2408.03568/) (2024)
- [Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data](https://mlsanity.com/paper/2406.18547/) (2024)
- [Applying Conditional Generative Adversarial Networks for Imaging Diagnosis](https://mlsanity.com/paper/2408.02074/) (2024)
- [Investigation of Customized Medical Decision Algorithms Utilizing Graph Neural Networks](https://mlsanity.com/paper/2405.17460/) (2024)
- [Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications](https://mlsanity.com/paper/2406.02601/) (2024)
- [Fairness and Robustness of CLIP-Based Models for Chest X-rays](https://mlsanity.com/paper/2507.21291/) (2025)
- [CURENet: Combining Unified Representations for Efficient Chronic Disease Prediction](https://mlsanity.com/paper/2511.11423/) (2025)
- [Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2512.19663/) (2025)

## Abstract (arXiv)

In the big data era, integrating diverse data modalities poses significant challenges, particularly in complex fields like healthcare. This paper introduces a new process model for multimodal Data Fusion for Data Mining, integrating embeddings and the Cross-Industry Standard Process for Data Mining with the existing Data Fusion Information Group model. Our model aims to decrease computational costs, complexity, and bias while improving efficiency and reliability. We also propose "disentangled dense fusion", a novel embedding fusion method designed to optimize mutual information and facilitate dense inter-modality feature interaction, thereby minimizing redundant information. We demonstrate the model's efficacy through three use cases: predicting diabetic retinopathy using retinal images and patient metadata, domestic violence prediction employing satellite imagery, internet, and census data, and identifying clinical and demographic features from radiography images and clinical notes. The model achieved a Macro F1 score of 0.92 in diabetic retinopathy prediction, an R-squared of 0.854 and sMAPE of 24.868 in domestic violence prediction, and a macro AUC of 0.92 and 0.99 for disease prediction and sex classification, respectively, in radiological analysis. These results underscore the Data Fusion for Data Mining model's potential to significantly impact multimodal data processing, promoting its adoption in diverse, resource-constrained settings.