mlsanityData to 7 Oct 2026

Unsupervised Learning of Video Representations using LSTMs

arXiv 1502.04681 ↗First posted 16 Feb 2015cs.LG cs.CV cs.NEcited by 434

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We use multilayer Long Short Term Memory (LSTM) networks to learn representations of video sequences. Our model uses an encoder LSTM to map an input sequence into a fixed length representation. This representation is decoded using single or multiple decoder LSTMs to perform different tasks, such as reconstructing the input sequence, or predicting the future sequence. We experiment with two kinds of input sequences - patches of image pixels and high-level representations ("percepts") of video frames extracted using a pretrained convolutional net. We explore different design choices such as whether the decoder LSTMs should condition on the generated output. We analyze the outputs of the model qualitatively to see how well the model can extrapolate the learned video representation into the future and into the past. We try to visualize and interpret the learned features. We stress test the model by running it on longer time scales and on out-of-domain data. We further evaluate the representations by finetuning them for a supervised learning problem - human action recognition on the UCF-101 and HMDB-51 datasets. We show that the representations help improve classification accuracy, especially when there are only a few training examples. Even models pretrained on unrelated datasets (300 hours of YouTube videos) can help action recognition performance.

Key facts. Cited by 434 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 124; 2022: 100; 2023: 89; 2024: 57; 2025: 41; 2026: 22.

434 papers.

Sort:

By area:

  1. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  2. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  5. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  6. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  7. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  8. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  9. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  10. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  11. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  12. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
38 more
  1. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  2. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  3. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  4. 2021A Comprehensive Survey on Graph Anomaly Detection with Deep Learningcited by 230
  5. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  6. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  7. 2023A Survey on Video Diffusion Modelscited by 203
  8. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 193
  9. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  10. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  11. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  12. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  13. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  14. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  15. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 146
  16. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  17. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  18. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  19. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  20. 20213D Human Action Representation Learning via Cross-View Consistency Pursuitcited by 105
  21. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  22. 2023PreDiff: Precipitation Nowcasting with Latent Diffusion Modelscited by 96
  23. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  24. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  25. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  26. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  27. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  28. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  29. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  30. 2022VIDM: Video Implicit Diffusion Modelscited by 78
  31. 2023Siamese Masked Autoencoderscited by 75
  32. 2022A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directionscited by 73
  33. 2021Self-supervised Video Transformercited by 64
  34. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  35. 2021iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysiscited by 63
  36. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  37. 2021Predicting Video with VQVAEcited by 61
  38. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61

50 most cited of 434 All 434 (JSON)