# Spatio-temporal video autoencoder with differentiable memory

arXiv: https://arxiv.org/abs/1511.06309 · HTML: https://mlsanity.com/paper/1511.06309/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2015.

## Built on by (most cited first, 30 of 32)

- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [Self-supervised Video Transformer](https://mlsanity.com/paper/2112.01514/) (2021)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Predicting Video with VQVAE](https://mlsanity.com/paper/2103.01950/) (2021)
- [Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals](https://mlsanity.com/paper/2107.12626/) (2021)
- [Masked Motion Encoding for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2210.06096/) (2022)
- [SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2211.12509/) (2022)
- [Domain Adaptive Video Segmentation via Temporal Consistency Regularization](https://mlsanity.com/paper/2107.11004/) (2021)
- [Comparing Correspondences: Video Prediction with Correspondence-wise Losses](https://mlsanity.com/paper/2104.09498/) (2021)
- [MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks](https://mlsanity.com/paper/2203.09303/) (2022)
- [Domain Adaptive Video Segmentation via Temporal Pseudo Supervision](https://mlsanity.com/paper/2207.02372/) (2022)
- [Language-based Action Concept Spaces Improve Video Self-Supervised Learning](https://mlsanity.com/paper/2307.10922/) (2023)
- [Learning to forecast vegetation greenness at fine resolution over Africa with ConvLSTMs](https://mlsanity.com/paper/2210.13648/) (2022)
- [Solving Low-Dose CT Reconstruction via GAN with Local Coherence](https://mlsanity.com/paper/2309.13584/) (2023)
- [TRecViT: A Recurrent Video Transformer](https://mlsanity.com/paper/2412.14294/) (2024)
- [LayerLock: Non-collapsing Representation Learning with Progressive Freezing](https://mlsanity.com/paper/2509.10156/) (2025)
- [TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction](https://mlsanity.com/paper/2110.14392/) (2021)
- [CARNet: A Dynamic Autoencoder for Learning Latent Dynamics in Autonomous Driving Tasks](https://mlsanity.com/paper/2205.08712/) (2022)
- [KARNet: Kalman Filter Augmented Recurrent Neural Network for Learning World Models in Autonomous Driving Tasks](https://mlsanity.com/paper/2305.14644/) (2023)
- [TOTNet: Occlusion-Aware Temporal Tracking for Robust Ball Detection in Sports Videos](https://mlsanity.com/paper/2508.09650/) (2025)
- [Affine-modeled video extraction from a single motion blurred image](https://mlsanity.com/paper/2104.03777/) (2021)
- [Dominant motion identification of multi-particle system using deep learning from video](https://mlsanity.com/paper/2104.12722/) (2021)
- [DriveGuard: Robustification of Automated Driving Systems with Deep Spatio-Temporal Convolutional Autoencoder](https://mlsanity.com/paper/2111.03480/) (2021)
- [HomE: Homography-Equivariant Video Representation Learning](https://mlsanity.com/paper/2306.01623/) (2023)
- [Wide and Narrow: Video Prediction from Context and Motion](https://mlsanity.com/paper/2110.11586/) (2021)
- [Borrowing from yourself: Faster future video segmentation with partial channel update](https://mlsanity.com/paper/2202.05748/) (2022)

## Abstract (arXiv)

We describe a new spatio-temporal video autoencoder, based on a classic spatial image autoencoder and a novel nested temporal autoencoder. The temporal encoder is represented by a differentiable visual memory composed of convolutional long short-term memory (LSTM) cells that integrate changes over time. Here we target motion changes and use as temporal decoder a robust optical flow prediction module together with an image sampler serving as built-in feedback loop. The architecture is end-to-end differentiable. At each time step, the system receives as input a video frame, predicts the optical flow based on the current observation and the LSTM memory state as a dense transformation map, and applies it to the current frame to generate the next frame. By minimising the reconstruction error between the predicted next frame and the corresponding ground truth next frame, we train the whole system to extract features useful for motion estimation without any supervision effort. We present one direct application of the proposed framework in weakly-supervised semantic segmentation of videos through label propagation using optical flow.