mlsanityData to 14 Oct 2026

GATSBI: Generative Agent-centric Spatio-temporal Object Interaction

arXiv 2104.04275 ↗First posted 9 Apr 2021cs.CV cs.LG cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

We present GATSBI, a generative model that can transform a sequence of raw observations into a structured latent representation that fully captures the spatio-temporal context of the agent's actions. In vision-based decision-making scenarios, an agent faces complex high-dimensional observations where multiple entities interact with each other. The agent requires a good scene representation of the visual observation that discerns essential components and consistently propagates along the time horizon. Our method, GATSBI, utilizes unsupervised object-centric scene representation learning to separate an active agent, static background, and passive objects. GATSBI then models the interactions reflecting the causal relationships among decomposed entities and predicts physically plausible future states. Our model generalizes to a variety of environments where different types of robots and objects dynamically interact with each other. We show GATSBI achieves superior performance on scene decomposition and video prediction compared to its state-of-the-art counterparts.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 9 Apr 2021.

Builds on

39 of 47 references.

References · 39

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2017Mask R-CNNcited by 7,277
  3. 2018YOLOv3: An Incremental Improvementcited by 2,659
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2018World Modelscited by 1,513
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  10. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  11. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  12. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
27 more
  1. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  2. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  3. 2017Stochastic Variational Video Predictioncited by 234
  4. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  5. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  6. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  7. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  8. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 156
  9. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 153
  10. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  11. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  12. 2017Neural Expectation Maximizationcited by 123
  13. 2016Sequential Neural Models with Stochastic Layerscited by 115
  14. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  15. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  16. 2019SCALOR: Generative World Models with Scalable Object Representationscited by 108
  17. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  18. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  19. 2020Improving Generative Imagination in Object-Centric World Modelscited by 71
  20. 2018Probabilistic Recurrent State-Space Modelscited by 64
  21. 2019COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Explorationcited by 57
  22. 2020Learning Physical Graph Representations from Visual Scenescited by 50
  23. 2019Structured Object-Aware Physics Prediction for Video Modeling and Planningcited by 43
  24. 2019Exploiting Spatial Invariance for Scalable Unsupervised Object Trackingcited by 40
  25. 2019R-SQAIR: Relational Sequential Attend, Infer, Repeatcited by 21
  26. 2018Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learningcited by 19
  27. 2020Unsupervised Video Decomposition using Spatio-temporal Iterative Inferencecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 0; 2024: 1; 2025: 2; 2026: 2.

8 papers.

  1. 2022Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motioncited by 42
  2. 2022GATSBI: Generative Adversarial Training for Simulation-Based Inferencecited by 36
  3. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
  4. 2022Motion-inductive Self-supervised Object Discovery in Videoscited by 14
  5. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  6. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 6
  7. 2024Compete and Compose: Learning Independent Mechanisms for Modular World Modelscited by 4
  8. 2025Autoregressive Meta-Actions for Unified Controllable Trajectory Generationcited by 2