# Interactive Fusion of Multi-level Features for Compositional Activity Recognition

arXiv: https://arxiv.org/abs/2012.05689 · HTML: https://mlsanity.com/paper/2012.05689/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [Multimodal Distillation for Egocentric Action Recognition](https://mlsanity.com/paper/2307.07483/) (2023)
- [Is an Object-Centric Video Representation Beneficial for Transfer?](https://mlsanity.com/paper/2207.10075/) (2022)
- [Video-Text Pre-training with Learned Regions](https://mlsanity.com/paper/2112.01194/) (2021)
- [Revisiting spatio-temporal layouts for compositional action recognition](https://mlsanity.com/paper/2111.01936/) (2021)
- [COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality](https://mlsanity.com/paper/2112.05892/) (2021)
- [Continual Learning for Image Segmentation with Dynamic Query](https://mlsanity.com/paper/2311.17450/) (2023)
- [SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognition](https://mlsanity.com/paper/2204.04796/) (2022)
- [Students taught by multimodal teachers are superior action recognizers](https://mlsanity.com/paper/2210.04331/) (2022)
- [Modelling Spatio-Temporal Interactions For Compositional Action Recognition](https://mlsanity.com/paper/2305.02673/) (2023)

## Abstract (arXiv)

To understand a complex action, multiple sources of information, including appearance, positional, and semantic features, need to be integrated. However, these features are difficult to be fused since they often differ significantly in modality and dimensionality. In this paper, we present a novel framework that accomplishes this goal by interactive fusion, namely, projecting features across different spaces and guiding it using an auxiliary prediction task. Specifically, we implement the framework in three steps, namely, positional-to-appearance feature extraction, semantic feature interaction, and semantic-to-positional prediction. We evaluate our approach on two action recognition datasets, Something-Something and Charades. Interactive fusion achieves consistent accuracy gain beyond off-the-shelf action recognition algorithms. In particular, on Something-Else, the compositional setting of Something-Something, interactive fusion reports a remarkable gain of 2.9% in terms of top-1 accuracy.