# Three-Stream Fusion Network for First-Person Interaction Recognition

arXiv: https://arxiv.org/abs/2002.08219 · HTML: https://mlsanity.com/paper/2002.08219/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 4 of 4)

- [HTNet: Anchor-free Temporal Action Localization with Hierarchical Transformers](https://mlsanity.com/paper/2207.09662/) (2022)
- [Visual Question Answering based on Local-Scene-Aware Referring Expression Generation](https://mlsanity.com/paper/2101.08978/) (2021)
- [Speeding Up Action Recognition Using Dynamic Accumulation of Residuals in Compressed Domain](https://mlsanity.com/paper/2209.14757/) (2022)
- [Rewarded meta-pruning: Meta Learning with Rewards for Channel Pruning](https://mlsanity.com/paper/2301.11063/) (2023)

## Abstract (arXiv)

First-person interaction recognition is a challenging task because of unstable video conditions resulting from the camera wearer's movement. For human interaction recognition from a first-person viewpoint, this paper proposes a three-stream fusion network with two main parts: three-stream architecture and three-stream correlation fusion. Thre three-stream architecture captures the characteristics of the target appearance, target motion, and camera ego-motion. Meanwhile the three-stream correlation fusion combines the feature map of each of the three streams to consider the correlations among the target appearance, target motion and camera ego-motion. The fused feature vector is robust to the camera movement and compensates for the noise of the camera ego-motion. Short-term intervals are modeled using the fused feature vector, and a long short-term memory(LSTM) model considers the temporal dynamics of the video. We evaluated the proposed method on two-public benchmark datasets to validate the effectiveness of our approach. The experimental results show that the proposed fusion method successfully generated a discriminative feature vector, and our network outperformed all competing activity recognition methods in first-person videos where considerable camera ego-motion occurs.