mlsanityData to 13 Oct 2026

Self-Supervised Visual Planning with Temporal Skip Connections

arXiv 1710.05268 ↗First posted 15 Oct 2017cs.RO cs.AI cs.CVcited by 139

Not on the map. Why?

Abstract (arXiv)

In order to autonomously learn wide repertoires of complex skills, robots must be able to learn from their own autonomously collected data, without human supervision. One learning signal that is always available for autonomously collected data is prediction: if a robot can learn to predict the future, it can use this predictive model to take actions to produce desired outcomes, such as moving an object to a particular location. However, in complex open-world scenarios, designing a representation for prediction is difficult. In this work, we instead aim to enable self-supervised robotic learning through direct video prediction: instead of attempting to design a good representation, we directly predict what the robot will see next, and then use this model to achieve desired goals. A key challenge in video prediction for robotic manipulation is handling complex spatial arrangements such as occlusions. To that end, we introduce a video prediction model that can keep track of objects through occlusion by incorporating temporal skip-connections. Together with a novel planning criterion and action space formulation, we demonstrate that this model substantially outperforms prior work on video prediction-based control. Our results show manipulation of objects not seen during training, handling multiple objects, and pushing objects around obstructions. These results represent a significant advance in the range and complexity of skills that can be performed entirely with self-supervised robotic learning.

Key facts. Cited by 139 papers in the map (data through 13 Oct 2026). First posted on arXiv 15 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 35; 2022: 27; 2023: 19; 2024: 22; 2025: 19; 2026: 15.

139 papers.

Sort:

By area:

  1. 2022Video Diffusion Modelscited by 2,293
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 769
  3. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 653
  4. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 477
  5. 2022MAGVIT: Masked Generative Video Transformercited by 343
  6. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 308
  7. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  8. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  9. 2023A Survey on Video Diffusion Modelscited by 206
  10. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  11. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 168
  12. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
38 more
  1. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  2. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 140
  3. 2022Diffusion Models for Video Prediction and Infillingcited by 135
  4. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  5. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  6. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 113
  7. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  8. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 110
  9. 2024Rolling Diffusion Modelscited by 104
  10. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  11. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  12. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  13. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  14. 2021Playable Video Generationcited by 64
  15. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  16. 2021Predicting Video with VQVAEcited by 61
  17. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 55
  18. 2024Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videoscited by 55
  19. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  20. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  21. 2024Restructuring Vector Quantization with the Rotation Trickcited by 50
  22. 2021Stochastic Image-to-Video Synthesis using cINNscited by 49
  23. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  24. 2024A Survey on Long Video Generation: Challenges, Methods, and Prospectscited by 40
  25. 2026Hierarchical Planning with Latent World Modelscited by 40
  26. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  27. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  28. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  29. 2021EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction taskcited by 34
  30. 2021SLAMP: Stochastic Latent Appearance and Motion Predictioncited by 32
  31. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  32. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  33. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  34. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  35. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  36. 2024Beyond FVD: Enhanced Evaluation Metrics for Video Generation Qualitycited by 22
  37. 2023LaMD: Latent Motion Diffusion for Image-Conditional Video Generationcited by 21
  38. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21

50 most cited of 139 All 139 (JSON)