# Memory-Augmented Temporal Dynamic Learning for Action Recognition

arXiv: https://arxiv.org/abs/1904.13080 · HTML: https://mlsanity.com/paper/1904.13080/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2019.

## Built on by (most cited first, 3 of 3)

- [Divide-and-Assemble: Learning Block-wise Memory for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2107.13118/) (2021)
- [Moving Towards Centers: Re-ranking with Attention and Memory for Re-identification](https://mlsanity.com/paper/2105.01447/) (2021)
- [Unsupervised Domain Adaptive Learning via Synthetic Data for Person Re-identification](https://mlsanity.com/paper/2109.05542/) (2021)

## Abstract (arXiv)

Human actions captured in video sequences contain two crucial factors for action recognition, i.e., visual appearance and motion dynamics. To model these two aspects, Convolutional and Recurrent Neural Networks (CNNs and RNNs) are adopted in most existing successful methods for recognizing actions. However, CNN based methods are limited in modeling long-term motion dynamics. RNNs are able to learn temporal motion dynamics but lack effective ways to tackle unsteady dynamics in long-duration motion. In this work, we propose a memory-augmented temporal dynamic learning network, which learns to write the most evident information into an external memory module and ignore irrelevant ones. In particular, we present a differential memory controller to make a discrete decision on whether the external memory module should be updated with current feature. The discrete memory controller takes in the memory history, context embedding and current feature as inputs and controls information flow into the external memory module. Additionally, we train this discrete memory controller using straight-through estimator. We evaluate this end-to-end system on benchmark datasets (UCF101 and HMDB51) of human action recognition. The experimental results show consistent improvements on both datasets over prior works and our baselines.