# On Stronger Computational Separations Between Multimodal and Unimodal Machine Learning

arXiv: https://arxiv.org/abs/2404.02254 · HTML: https://mlsanity.com/paper/2404.02254/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [A Theory of Multimodal Learning](https://mlsanity.com/paper/2309.12458/) (2023)
- [TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learning](https://mlsanity.com/paper/2007.06793/) (2020)
- [Learning Using Privileged Information: SVM+ and Weighted SVM](https://mlsanity.com/paper/1306.3161/) (2013)
- [Distributional PAC-Learning from Nisan's Natural Proofs](https://mlsanity.com/paper/2310.03641/) (2023)

## Abstract (arXiv)

Recently, multimodal machine learning has enjoyed huge empirical success (e.g. GPT-4). Motivated to develop theoretical justification for this empirical success, Lu (NeurIPS '23, ALT '24) introduces a theory of multimodal learning, and considers possible textitseparations between theoretical models of multimodal and unimodal learning. In particular, Lu (ALT '24) shows a computational separation, which is relevant to textitworst-case instances of the learning task. In this paper, we give a stronger textitaverage-case computational separation, where for ``typical'' instances of the learning task, unimodal learning is computationally hard, but multimodal learning is easy. We then question how ``natural'' the average-case separation is. Would it be encountered in practice? To this end, we prove that under basic conditions, any given computational separation between average-case unimodal and multimodal learning tasks implies a corresponding cryptographic key agreement protocol. We suggest to interpret this as evidence that very strong textitcomputational advantages of multimodal learning may arise textitinfrequently in practice, since they exist only for the ``pathological'' case of inherently cryptographic distributions. However, this does not apply to possible (super-polynomial) textitstatistical advantages.