# Unsupervised Learning of Long-Term Motion Dynamics for Videos

arXiv: https://arxiv.org/abs/1701.01821 · HTML: https://mlsanity.com/paper/1701.01821/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2017.

## Built on by (most cited first, 30 of 37)

- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [Playable Video Generation](https://mlsanity.com/paper/2101.12195/) (2021)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2108.01959/) (2021)
- [MemFlow: Optical Flow Estimation and Prediction with Memory](https://mlsanity.com/paper/2404.04808/) (2024)
- [Home Action Genome: Cooperative Compositional Action Understanding](https://mlsanity.com/paper/2105.05226/) (2021)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [Contrastive Positive Mining for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2208.03497/) (2022)
- [SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-supervised Skeleton Action Recognition](https://mlsanity.com/paper/2209.02399/) (2022)
- [Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization](https://mlsanity.com/paper/2108.02183/) (2021)
- [Static and Dynamic Concepts for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2207.12795/) (2022)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2111.11051/) (2021)
- [Unsupervised Motion Representation Learning with Capsule Autoencoders](https://mlsanity.com/paper/2110.00529/) (2021)
- [Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorization](https://mlsanity.com/paper/2304.08799/) (2023)
- [EEGFuseNet: Hybrid Unsupervised Deep Feature Characterization and Fusion for High-Dimensional EEG with An Application to Emotion Recognition](https://mlsanity.com/paper/2102.03777/) (2021)
- [Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency](https://mlsanity.com/paper/2204.03017/) (2022)
- [Dual Contrastive Learning for Spatio-temporal Representation](https://mlsanity.com/paper/2207.05340/) (2022)
- [CoCon: Cooperative-Contrastive Learning](https://mlsanity.com/paper/2104.14764/) (2021)
- [Contrastive Spatio-Temporal Pretext Learning for Self-supervised Video Representation](https://mlsanity.com/paper/2112.08913/) (2021)
- [Controllable Augmentations for Video Representation Learning](https://mlsanity.com/paper/2203.16632/) (2022)
- [TransRank: Self-supervised Video Representation Learning via Ranking-based Transformation Recognition](https://mlsanity.com/paper/2205.02028/) (2022)
- [Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding](https://mlsanity.com/paper/2405.13206/) (2024)
- [Video-based Human Action Recognition using Deep Learning: A Review](https://mlsanity.com/paper/2208.03775/) (2022)
- [Modeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity](https://mlsanity.com/paper/2209.07811/) (2022)
- [Bi-Calibration Networks for Weakly-Supervised Video Representation Learning](https://mlsanity.com/paper/2206.10491/) (2022)
- [Unsupervised View-Invariant Human Posture Representation](https://mlsanity.com/paper/2109.08730/) (2021)
- [How Far Can I Go ? : A Self-Supervised Approach for Deterministic Video Depth Forecasting](https://mlsanity.com/paper/2207.00506/) (2022)
- [TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction](https://mlsanity.com/paper/2110.14392/) (2021)
- [Self-Supervised Video Representation Learning by Video Incoherence Detection](https://mlsanity.com/paper/2109.12493/) (2021)

## Abstract (arXiv)

We present an unsupervised representation learning approach that compactly encodes the motion dependencies in videos. Given a pair of images from a video clip, our framework learns to predict the long-term 3D motions. To reduce the complexity of the learning framework, we propose to describe the motion as a sequence of atomic 3D flows computed with RGB-D modality. We use a Recurrent Neural Network based Encoder-Decoder framework to predict these sequences of flows. We argue that in order for the decoder to reconstruct these sequences, the encoder must learn a robust video representation that captures long-term motion dependencies and spatial-temporal relations. We demonstrate the effectiveness of our learned temporal representations on activity classification across multiple modalities and datasets such as NTU RGB+D and MSR Daily Activity 3D. Our framework is generic to any input modality, i.e., RGB, Depth, and RGB-D videos.