# Lattice Long Short-Term Memory for Human Action Recognition

arXiv: https://arxiv.org/abs/1708.03958 · HTML: https://mlsanity.com/paper/1708.03958/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2017.

## Built on by (most cited first, 16 of 16)

- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [Hierarchical Self-supervised Representation Learning for Movie Understanding](https://mlsanity.com/paper/2204.03101/) (2022)
- [CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformers](https://mlsanity.com/paper/2305.07840/) (2023)
- [Compressed Vision for Efficient Video Understanding](https://mlsanity.com/paper/2210.02995/) (2022)
- [PGT: A Progressive Method for Training Models on Long Videos](https://mlsanity.com/paper/2103.11313/) (2021)
- [Efficient Action Recognition Using Confidence Distillation](https://mlsanity.com/paper/2109.02137/) (2021)
- [MMTSA: Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognition](https://mlsanity.com/paper/2210.09222/) (2022)
- [Deep Learning Computer Vision Algorithms for Real-time UAVs On-board Camera Image Processing](https://mlsanity.com/paper/2211.01037/) (2022)
- [Real-Time Activity Recognition and Intention Recognition Using a Vision-based Embedded System](https://mlsanity.com/paper/2107.12744/) (2021)
- [Application-Driven AI Paradigm for Human Action Recognition](https://mlsanity.com/paper/2209.15271/) (2022)
- [A Survey on Human Action Recognition](https://mlsanity.com/paper/2301.06082/) (2022)
- [Texture-Based Input Feature Selection for Action Recognition](https://mlsanity.com/paper/2303.00138/) (2023)
- [Cutup and Detect: Human Fall Detection on Cutup Untrimmed Videos Using a Large Foundational Video Understanding Model](https://mlsanity.com/paper/2401.16280/) (2024)
- [HRTR: A Single-stage Transformer for Fine-grained Sub-second Action Segmentation in Stroke Rehabilitation](https://mlsanity.com/paper/2506.02472/) (2025)
- [Effects of Different Attention Mechanisms Applied on 3D Models in Video Classification](https://mlsanity.com/paper/2601.10854/) (2026)

## Abstract (arXiv)

Human actions captured in video sequences are three-dimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short-Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long. In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.