# On the Computational Benefit of Multimodal Learning

arXiv: https://arxiv.org/abs/2309.13782 · HTML: https://mlsanity.com/paper/2309.13782/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Text to 3D Scene Generation with Rich Lexical Grounding](https://mlsanity.com/paper/1505.06289/) (2015)
- [A Theory of Multimodal Learning](https://mlsanity.com/paper/2309.12458/) (2023)
- [From Local Pseudorandom Generators to Hardness of Learning](https://mlsanity.com/paper/2101.08303/) (2021)
- [TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learning](https://mlsanity.com/paper/2007.06793/) (2020)
- [From average case complexity to improper learning complexity](https://mlsanity.com/paper/1311.2272/) (2013)
- [On the Importance of Contrastive Loss in Multimodal Learning](https://mlsanity.com/paper/2304.03717/) (2023)

## Built on by (most cited first, 1 of 1)

- [Towards Modality Generalization: A Benchmark and Prospective Analysis](https://mlsanity.com/paper/2412.18277/) (2024)

## Abstract (arXiv)

Human perception inherently operates in a multimodal manner. Similarly, as machines interpret the empirical world, their learning processes ought to be multimodal. The recent, remarkable successes in empirical multimodal learning underscore the significance of understanding this paradigm. Yet, a solid theoretical foundation for multimodal learning has eluded the field for some time. While a recent study by Lu (2023) has shown the superior sample complexity of multimodal learning compared to its unimodal counterpart, another basic question remains: does multimodal learning also offer computational advantages over unimodal learning? This work initiates a study on the computational benefit of multimodal learning. We demonstrate that, under certain conditions, multimodal learning can outpace unimodal learning exponentially in terms of computation. Specifically, we present a learning task that is NP-hard for unimodal learning but is solvable in polynomial time by a multimodal algorithm. Our construction is based on a novel modification to the intersection of two half-spaces problem.