# MUTAN: Multimodal Tucker Fusion for Visual Question Answering

arXiv: https://arxiv.org/abs/1705.06676 · HTML: https://mlsanity.com/paper/1705.06676/

## Key facts

- Cited by 137 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2017.

## Built on by (most cited first, 30 of 137)

- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning](https://mlsanity.com/paper/2104.03135/) (2021)
- [CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning](https://mlsanity.com/paper/2208.05358/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [Multi-Modal Answer Validation for Knowledge-Based VQA](https://mlsanity.com/paper/2103.12248/) (2021)
- [Multi-modal Sensor Fusion for Auto Driving Perception: A Survey](https://mlsanity.com/paper/2202.02703/) (2022)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering](https://mlsanity.com/paper/2104.03149/) (2021)
- [Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training](https://mlsanity.com/paper/2106.13488/) (2021)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery](https://mlsanity.com/paper/2304.09974/) (2023)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention](https://mlsanity.com/paper/2303.02995/) (2023)
- [GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer](https://mlsanity.com/paper/2406.01210/) (2024)
- [BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models](https://mlsanity.com/paper/2312.02896/) (2023)
- [From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data](https://mlsanity.com/paper/2205.03147/) (2022)
- [Gender and Racial Bias in Visual Question Answering Datasets](https://mlsanity.com/paper/2205.08148/) (2022)
- [Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery](https://mlsanity.com/paper/2405.10948/) (2024)
- [SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answering](https://mlsanity.com/paper/2204.02285/) (2022)
- [MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2203.09138/) (2022)
- [Classifier-guided Gradient Modulation for Enhanced Multimodal Learning](https://mlsanity.com/paper/2411.01409/) (2024)
- [Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering](https://mlsanity.com/paper/2103.05568/) (2021)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Learning to Answer Visual Questions from Web Videos](https://mlsanity.com/paper/2205.05019/) (2022)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [LocalTrans: A Multiscale Local Transformer Network for Cross-Resolution Homography Estimation](https://mlsanity.com/paper/2106.04067/) (2021)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)

## Abstract (arXiv)

Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks. They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues. We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations. Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank. With MUTAN, we control the complexity of the merging scheme while keeping nice interpretable fusion relations. We show how our MUTAN model generalizes some of the latest VQA architectures, providing state-of-the-art results.