# Mixup Helps Understanding Multimodal Video Better

arXiv: https://arxiv.org/abs/2510.10986 · HTML: https://mlsanity.com/paper/2510.10986/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Grounding of Textual Phrases in Images by Reconstruction](https://mlsanity.com/paper/1511.03745/) (2015)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance](https://mlsanity.com/paper/2405.17730/) (2024)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Improving Multi-Modal Learning with Uni-Modal Teachers](https://mlsanity.com/paper/2106.11059/) (2021)
- [Diagnosing and Re-learning for Balanced Multimodal Learning](https://mlsanity.com/paper/2407.09705/) (2024)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [MetaMixUp: Learning Adaptive Interpolation Policy of MixUp with Meta-Learning](https://mlsanity.com/paper/1908.10059/) (2019)
- [Hierarchical Boundary-Aware Neural Encoder for Video Captioning](https://mlsanity.com/paper/1611.09312/) (2016)
- [Improving Multimodal Learning with Multi-Loss Gradient Modulation](https://mlsanity.com/paper/2405.07930/) (2024)
- [SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities](https://mlsanity.com/paper/2205.00302/) (2022)
- [Improving Multimodal Learning Balance and Sufficiency through Data Remixing](https://mlsanity.com/paper/2506.11550/) (2025)

## Abstract (arXiv)

Multimodal video understanding plays a crucial role in tasks such as action recognition and emotion classification by combining information from different modalities. However, multimodal models are prone to overfitting strong modalities, which can dominate learning and suppress the contributions of weaker ones. To address this challenge, we first propose Multimodal Mixup (MM), which applies the Mixup strategy at the aggregated multimodal feature level to mitigate overfitting by generating virtual feature-label pairs. While MM effectively improves generalization, it treats all modalities uniformly and does not account for modality imbalance during training. Building on MM, we further introduce Balanced Multimodal Mixup (B-MM), which dynamically adjusts the mixing ratios for each modality based on their relative contributions to the learning objective. Extensive experiments on several datasets demonstrate the effectiveness of our methods in improving generalization and multimodal robustness.