# MultiFIX: An XAI-friendly feature inducing approach to building models from multimodal data

arXiv: https://arxiv.org/abs/2402.12183 · HTML: https://mlsanity.com/paper/2402.12183/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 6 of 6)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [A Patient-Centric Dataset of Images and Metadata for Identifying Melanomas Using Clinical Context](https://mlsanity.com/paper/2008.07360/) (2020)
- [What do we need to build explainable AI systems for the medical domain?](https://mlsanity.com/paper/1712.09923/) (2017)
- [On Explaining Machine Learning Models by Evolving Crucial and Compact Features](https://mlsanity.com/paper/1907.02260/) (2019)
- [Improving the efficiency of GP-GOMEA for higher-arity operators](https://mlsanity.com/paper/2402.09854/) (2024)

## Abstract (arXiv)

In the health domain, decisions are often based on different data modalities. Thus, when creating prediction models, multimodal fusion approaches that can extract and combine relevant features from different data modalities, can be highly beneficial. Furthermore, it is important to understand how each modality impacts the final prediction, especially in high-stake domains, so that these models can be used in a trustworthy and responsible manner. We propose MultiFIX: a new interpretability-focused multimodal data fusion pipeline that explicitly induces separate features from different data types that can subsequently be combined to make a final prediction. An end-to-end deep learning architecture is used to train a predictive model and extract representative features of each modality. Each part of the model is then explained using explainable artificial intelligence techniques. Attention maps are used to highlight important regions in image inputs. Inherently interpretable symbolic expressions, learned with GP-GOMEA, are used to describe the contribution of tabular inputs. The fusion of the extracted features to predict the target label is also replaced by a symbolic expression, learned with GP-GOMEA. Results on synthetic problems demonstrate the strengths and limitations of MultiFIX. Lastly, we apply MultiFIX to a publicly available dataset for the detection of malignant skin lesions.