# Students taught by multimodal teachers are superior action recognizers

arXiv: https://arxiv.org/abs/2210.04331 · HTML: https://mlsanity.com/paper/2210.04331/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 20 references that are in the map.
- First posted on arXiv 9 Oct 2022.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 18 of 18)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks](https://mlsanity.com/paper/1912.09930/) (2019)
- [Object-Region Video Transformers](https://mlsanity.com/paper/2110.06915/) (2021)
- [With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition](https://mlsanity.com/paper/2111.01024/) (2021)
- [Is an Object-Centric Video Representation Beneficial for Transfer?](https://mlsanity.com/paper/2207.10075/) (2022)
- [M&M Mix: A Multimodal Multiview Transformer Ensemble](https://mlsanity.com/paper/2206.09852/) (2022)
- [Revisiting spatio-temporal layouts for compositional action recognition](https://mlsanity.com/paper/2111.01936/) (2021)
- [Motion Guided Attention Fusion to Recognize Interactions from Videos](https://mlsanity.com/paper/2104.00646/) (2021)
- [Interactive Fusion of Multi-level Features for Compositional Activity Recognition](https://mlsanity.com/paper/2012.05689/) (2020)

## Built on by (most cited first, 2 of 2)

- [Multimodal Distillation for Egocentric Action Recognition](https://mlsanity.com/paper/2307.07483/) (2023)
- [Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning](https://mlsanity.com/paper/2510.13182/) (2025)

## Abstract (arXiv)

The focal point of egocentric video understanding is modelling hand-object interactions. Standard models -- CNNs, Vision Transformers, etc. -- which receive RGB frames as input perform well, however, their performance improves further by employing additional modalities such as object detections, optical flow, audio, etc. as input. The added complexity of the required modality-specific modules, on the other hand, makes these models impractical for deployment. The goal of this work is to retain the performance of such multimodal approaches, while using only the RGB images as input at inference time. Our approach is based on multimodal knowledge distillation, featuring a multimodal teacher (in the current experiments trained only using object detections, optical flow and RGB frames) and a unimodal student (using only RGB frames as input). We present preliminary results which demonstrate that the resulting model -- distilled from a multimodal teacher -- significantly outperforms the baseline RGB model (trained without knowledge distillation), as well as an omnivorous version of itself (trained on all modalities jointly), in both standard and compositional action recognition.