# An End-to-End Two-Stream Network Based on RGB Flow and Representation Flow for Human Action Recognition

arXiv: https://arxiv.org/abs/2411.18002 · HTML: https://mlsanity.com/paper/2411.18002/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition](https://mlsanity.com/paper/1711.11152/) (2017)
- [Representation Flow for Action Recognition](https://mlsanity.com/paper/1810.01455/) (2018)
- [Going Deeper into First-Person Activity Recognition](https://mlsanity.com/paper/1605.03688/) (2016)
- [End-to-End Learning of Motion Representation for Video Understanding](https://mlsanity.com/paper/1804.00413/) (2018)
- [Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognition](https://mlsanity.com/paper/1807.11794/) (2018)

## Abstract (arXiv)

With the rapid advancements in deep learning, computer vision tasks have seen significant improvements, making two-stream neural networks a popular focus for video based action recognition. Traditional models using RGB and optical flow streams achieve strong performance but at a high computational cost. To address this, we introduce a representation flow algorithm to replace the optical flow branch in the egocentric action recognition model, enabling end-to-end training while reducing computational cost and prediction time. Our model, designed for egocentric action recognition, uses class activation maps (CAMs) to improve accuracy and ConvLSTM for spatio temporal encoding with spatial attention. When evaluated on the GTEA61, EGTEA GAZE+, and HMDB datasets, our model matches the accuracy of the original model on GTEA61 and exceeds it by 0.65% and 0.84% on EGTEA GAZE+ and HMDB, respectively. Prediction runtimes are significantly reduced to 0.1881s, 0.1503s, and 0.1459s, compared to the original model's 101.6795s, 25.3799s, and 203.9958s. Ablation studies were also conducted to study the impact of different parameters on model performance. Keywords: two-stream, egocentric, action recognition, CAM, representation flow, CAM, ConvLSTM