# Mutual Modality Learning for Video Action Classification

arXiv: https://arxiv.org/abs/2011.02543 · HTML: https://mlsanity.com/paper/2011.02543/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2020.

## Built on by (most cited first, 3 of 3)

- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecture](https://mlsanity.com/paper/2111.10974/) (2021)
- [Multilingual Communication System with Deaf Individuals Utilizing Natural and Visual Languages](https://mlsanity.com/paper/2212.00305/) (2022)

## Abstract (arXiv)

The construction of models for video action classification progresses rapidly. However, the performance of those models can still be easily improved by ensembling with the same models trained on different modalities (e.g. Optical flow). Unfortunately, it is computationally expensive to use several modalities during inference. Recent works examine the ways to integrate advantages of multi-modality into a single RGB-model. Yet, there is still a room for improvement. In this paper, we explore the various methods to embed the ensemble power into a single model. We show that proper initialization, as well as mutual modality learning, enhances single-modality models. As a result, we achieve state-of-the-art results in the Something-Something-v2 benchmark.