# Two Stream LSTM: A Deep Fusion Framework for Human Action Recognition

arXiv: https://arxiv.org/abs/1704.01194 · HTML: https://mlsanity.com/paper/1704.01194/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2017.

## Built on by (most cited first, 18 of 18)

- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [No frame left behind: Full Video Action Recognition](https://mlsanity.com/paper/2103.15395/) (2021)
- [Continuous Human Action Recognition for Human-Machine Interaction: A Review](https://mlsanity.com/paper/2202.13096/) (2022)
- [FAR: Fourier Aerial Video Recognition](https://mlsanity.com/paper/2203.10694/) (2022)
- [Computer Vision for Clinical Gait Analysis: A Gait Abnormality Video Dataset](https://mlsanity.com/paper/2407.04190/) (2024)
- [Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structure](https://mlsanity.com/paper/2104.05758/) (2021)
- [Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition](https://mlsanity.com/paper/2209.09194/) (2022)
- [Multi-Perspective LSTM for Joint Visual Representation Learning](https://mlsanity.com/paper/2105.02802/) (2021)
- [evTransFER: A Transfer Learning Framework for Event-based Facial Expression Recognition](https://mlsanity.com/paper/2508.03609/) (2025)
- [NeuroView-RNN: It's About Time](https://mlsanity.com/paper/2202.11811/) (2022)
- [Turning old models fashion again: Recycling classical CNN networks using the Lattice Transformation](https://mlsanity.com/paper/2109.13885/) (2021)
- [Exploring Automated Recognition of Instructional Activity and Discourse from Multimodal Classroom Data](https://mlsanity.com/paper/2512.00087/) (2025)
- [Event-based Timestamp Image Encoding Network for Human Action Recognition and Anticipation](https://mlsanity.com/paper/2104.05145/) (2021)
- [Action Shuffling for Weakly Supervised Temporal Localization](https://mlsanity.com/paper/2105.04208/) (2021)
- [ActAR: Actor-Driven Pose Embeddings for Video Action Recognition](https://mlsanity.com/paper/2204.08671/) (2022)
- [Two Stream Scene Understanding on Graph Embedding](https://mlsanity.com/paper/2311.06746/) (2023)
- [A Universal Action Space for General Behavior Analysis](https://mlsanity.com/paper/2602.09518/) (2026)
- [A Heterogeneous Two-Stream Framework for Video Action Recognition with Comparative Fusion Analysis](https://mlsanity.com/paper/2604.23415/) (2026)

## Abstract (arXiv)

In this paper we address the problem of human action recognition from video sequences. Inspired by the exemplary results obtained via automatic feature learning and deep learning approaches in computer vision, we focus our attention towards learning salient spatial features via a convolutional neural network (CNN) and then map their temporal relationship with the aid of Long-Short-Term-Memory (LSTM) networks. Our contribution in this paper is a deep fusion framework that more effectively exploits spatial features from CNNs with temporal features from LSTM models. We also extensively evaluate their strengths and weaknesses. We find that by combining both the sets of features, the fully connected features effectively act as an attention mechanism to direct the LSTM to interesting parts of the convolutional feature sequence. The significance of our fusion method is its simplicity and effectiveness compared to other state-of-the-art methods. The evaluation results demonstrate that this hierarchical multi stream fusion method has higher performance compared to single stream mapping methods allowing it to achieve high accuracy outperforming current state-of-the-art methods in three widely used databases: UCF11, UCFSports, jHMDB.