# Adversarial Memory Networks for Action Prediction

arXiv: https://arxiv.org/abs/2112.09875 · HTML: https://mlsanity.com/paper/2112.09875/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 50 references that are in the map.
- First posted on arXiv 18 Dec 2021.

## Builds on (most cited first, 19 of 19)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [Dynamic Key-Value Memory Networks for Knowledge Tracing](https://mlsanity.com/paper/1611.08108/) (2016)
- [Encouraging LSTMs to Anticipate Actions Very Early](https://mlsanity.com/paper/1703.07023/) (2017)
- [Deep Temporal Linear Encoding Networks](https://mlsanity.com/paper/1611.06678/) (2016)
- [Action Anticipation with RBF Kernelized Feature Mapping RNN](https://mlsanity.com/paper/1911.07806/) (2019)
- [A Temporal Sequence Learning for Action Recognition and Prediction](https://mlsanity.com/paper/1906.06813/) (2019)

## Abstract (arXiv)

Action prediction aims to infer the forthcoming human action with partially-observed videos, which is a challenging task due to the limited information underlying early observations. Existing methods mainly adopt a reconstruction strategy to handle this task, expecting to learn a single mapping function from partial observations to full videos to facilitate the prediction process. In this study, we propose adversarial memory networks (AMemNet) to generate the "full video" feature conditioning on a partial video query from two new aspects. Firstly, a key-value structured memory generator is designed to memorize different partial videos as key memories and dynamically write full videos in value memories with gating mechanism and querying attention. Secondly, we develop a class-aware discriminator to guide the memory generator to deliver not only realistic but also discriminative full video features upon adversarial training. The final prediction result of AMemNet is given by late fusion over RGB and optical flow streams. Extensive experimental results on two benchmark video datasets, UCF-101 and HMDB51, are provided to demonstrate the effectiveness of the proposed AMemNet model over state-of-the-art methods.