mlsanityData to 7 Oct 2026

Object-centric Video Prediction without Annotation

arXiv 2105.02799 ↗First posted 6 May 2021cs.CV cs.ROcited by 5

Not on the map. Why?

Abstract (arXiv)

In order to interact with the world, agents must be able to predict the results of the world's dynamics. A natural approach to learn about these dynamics is through video prediction, as cameras are ubiquitous and powerful sensors. Direct pixel-to-pixel video prediction is difficult, does not take advantage of known priors, and does not provide an easy interface to utilize the learned dynamics. Object-centric video prediction offers a solution to these problems by taking advantage of the simple prior that the world is made of objects and by providing a more natural interface for control. However, existing object-centric video prediction pipelines require dense object annotations in training video sequences. In this work, we present Object-centric Prediction without Annotation (OPA), an object-centric video prediction method that takes advantage of priors from powerful computer vision models. We validate our method on a dataset comprised of video sequences of stacked objects falling, and demonstrate how to adapt a perception model in an environment through end-to-end video prediction training.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 41 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

37 of 41 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017Mask R-CNNcited by 7,259
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  8. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  9. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  10. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  11. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  12. 2018Stochastic Adversarial Video Predictioncited by 173
25 more
  1. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  2. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  3. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  4. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  5. 2017Visual Interaction Networkscited by 94
  6. 2018Visual Representations for Semantic Target Driven Navigationcited by 89
  7. 2017Recurrent Environment Simulatorscited by 68
  8. 2016Learning Physical Intuition of Block Towers by Examplecited by 66
  9. 2019Does computer vision matter for action?cited by 64
  10. 2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
  11. 2018ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stackingcited by 55
  12. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  13. 2019Towards Segmenting Anything That Movescited by 51
  14. 2017Deep Video Generation, Prediction and Completion of Human Action Sequencescited by 50
  15. 2016SE3-Nets: Learning Rigid Body Motion using Deep Neural Networkscited by 48
  16. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  17. 2020Learning Long-term Visual Dynamics with Region Proposal Interaction Networkscited by 44
  18. 2019Compositional Video Predictioncited by 43
  19. 2019Learning Predictive Models From Observation and Interactioncited by 41
  20. 2020RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spacescited by 32
  21. 2018Learning what you can do before doing anythingcited by 26
  22. 2019Learning to Navigate Using Mid-Level Visual Priorscited by 25
  23. 2019Object-centric Forward Modeling for Model Predictive Controlcited by 19
  24. 2019Simultaneous Mapping and Target Driven Navigationcited by 11
  25. 2018Pixel Objectness: Learning to Segment Generic Objects Automatically in Images and Videoscited by 9

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 2.

5 papers.

  1. 2023Benchmarks for Physical Reasoning AIcited by 8
  2. 2023Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generationcited by 7
  3. 2022WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Predictioncited by 5
  4. 2024Learning Physical Dynamics for Object-centric Visual Predictioncited by 1
  5. 2024Forecasting Future Videos from Novel Views via Disentangled 3D Scene Representationcited by 1