# Interpretable Tensor Fusion

arXiv: https://arxiv.org/abs/2405.04671 · HTML: https://mlsanity.com/paper/2405.04671/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2005.07310/) (2020)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers](https://mlsanity.com/paper/2109.04448/) (2021)
- [Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!](https://mlsanity.com/paper/2010.06572/) (2020)
- [Do Explanations make VQA Models more Predictable to a Human?](https://mlsanity.com/paper/1810.12366/) (2018)
- [Perceptual Score: What Data Modalities Does Your Model Perceive?](https://mlsanity.com/paper/2110.14375/) (2021)
- [Multimodal Explanations by Predicting Counterfactuality in Videos](https://mlsanity.com/paper/1812.01263/) (2018)
- [Multi-modal Egocentric Activity Recognition using Audio-Visual Features](https://mlsanity.com/paper/1807.00612/) (2018)
- [Classification of glomerular hypercellularity using convolutional features and support vector machine](https://mlsanity.com/paper/1907.00028/) (2019)

## Built on by (most cited first, 2 of 2)

- [AI-based Anomaly Detection for Clinical-Grade Histopathological Diagnostics](https://mlsanity.com/paper/2406.14866/) (2024)
- [SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis](https://mlsanity.com/paper/2609.05174/) (2026)

## Abstract (arXiv)

Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce interpretable tensor fusion (InTense), a multimodal learning method for training neural networks to simultaneously learn multimodal data representations and their interpretable fusion. InTense can separately capture both linear combinations and multiplicative interactions of diverse data types, thereby disentangling higher-order interactions from the individual effects of each modality. InTense provides interpretability out of the box by assigning relevance scores to modalities and their associations. The approach is theoretically grounded and yields meaningful relevance scores on multiple synthetic and real-world datasets. Experiments on six real-world datasets show that InTense outperforms existing state-of-the-art multimodal interpretable approaches in terms of accuracy and interpretability.