mlsanityData to 7 Oct 2026

SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models

arXiv 2210.05861 ↗First posted 12 Oct 2022cs.CV cs.AI cs.LGcited by 146

Not on the map. Why?

Cited by 40 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Understanding dynamics from visual observations is a challenging problem that requires disentangling individual objects from the scene and learning their interactions. While recent object-centric models can successfully decompose a scene into objects, modeling their dynamics effectively still remains a challenge. We address this problem by introducing SlotFormer -- a Transformer-based autoregressive model operating on learned object-centric representations. Given a video clip, our approach reasons over object features to model spatio-temporal relationships and predicts accurate future object states. In this paper, we successfully apply SlotFormer to perform video prediction on datasets with complex object interactions. Moreover, the unsupervised SlotFormer's dynamics model can be used to improve the performance on supervised downstream tasks, such as Visual Question Answering (VQA), and goal-conditioned planning. Compared to past works on dynamics modeling, our method achieves significantly better long-term synthesis of object dynamics, while retaining high quality visual generation. Besides, SlotFormer enables VQA models to reason about the future without object-level labels, even outperforming counterparts that use ground-truth annotations. Finally, we show its ability to serve as a world model for model-based planning, which is competitive with methods designed specifically for such tasks.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Builds on 69 of its 88 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

69 of 88 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2016Layer Normalizationcited by 5,579
  11. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2015Fast R-CNNcited by 2,930
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  6. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  7. 2020On Layer Normalization in the Transformer Architecturecited by 906
  8. 2020Object-Centric Learning with Slot Attentioncited by 865
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  10. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  12. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  13. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  14. 2017A simple neural network module for relational reasoningcited by 345
  15. 2022Transformers are Sample-Efficient World Modelscited by 317
  16. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  17. 2021Conditional Object-Centric Learning from Videocited by 247
  18. 2018Stochastic Video Generation with a Learned Priorcited by 244
  19. 2018Graph networks as learnable physics engines for inference and controlcited by 242
  20. 2019Recurrent Independent Mechanismscited by 219
  21. 2019Contrastive Learning of Structured World Modelscited by 206
  22. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  23. 2018Stochastic Adversarial Video Predictioncited by 173
  24. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  25. 2021Illiterate DALL-E Learns to Composecited by 147
  26. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  27. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  28. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  29. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  30. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  31. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  32. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  33. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  34. 2019SCALOR: Generative World Models with Scalable Object Representationscited by 108
  35. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  36. 2017Visual Interaction Networkscited by 94
  37. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  38. 2016An Uncertain Future: Forecasting from Static Images using Variational Autoencoderscited by 86

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 35; 2024: 34; 2025: 30; 2026: 46.

146 papers.

Sort:

By area:

  1. 2023Learning Interactive Real-World Simulatorscited by 440
  2. 2022Disentangled Representation Learningcited by 111
  3. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  4. 2023Object-Centric Slot Diffusioncited by 90
  5. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  6. 2023Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similaritiescited by 78
  7. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 53
  8. 2023Invariant Slot Attention: Object Discovery with Slot-Centric Reference Framescited by 51
  9. 2023SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformerscited by 49
  10. 2023EventCLIP: Adapting CLIP for Event-based Object Recognitioncited by 36
  11. 2024Temporally Consistent Object-Centric Learning by Contrasting Slotscited by 30
  12. 2023Rotating Features for Object Discoverycited by 29
38 more
  1. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  2. 2024Entity-Centric Reinforcement Learning for Object Manipulation from Pixelscited by 27
  3. 2023Object Discovery from Motion-Guided Tokenscited by 25
  4. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  5. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 25
  6. 2026Causal-JEPA: Learning World Models through Object-Level Latent Maskingcited by 25
  7. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 25
  8. 2023Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactionscited by 19
  9. 2023Learning to reason over visual objectscited by 19
  10. 2023Systematic Visual Reasoning through Object-Centric Relational Abstractioncited by 19
  11. 2024Zero-Shot Object-Centric Representation Learningcited by 19
  12. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 18
  13. 2023FOCUS: Object-Centric World Models for Robotics Manipulationcited by 17
  14. 2024Slot State Space Modelscited by 17
  15. 2023Unlocking Slot Attention by Changing Optimal Transport Costscited by 16
  16. 2023Contrastive Training of Complex-Valued Autoencoders for Object Discoverycited by 16
  17. 2025Object-Centric World Model for Language-Guided Manipulationcited by 16
  18. 2023Object-Centric Multiple Object Trackingcited by 15
  19. 2024Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Modelscited by 14
  20. 2025Slot Attention with Re-Initialization and Self-Distillationcited by 14
  21. 2025Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generationcited by 13
  22. 2025PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planningcited by 13
  23. 2025FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocitycited by 13
  24. 2025CTRL-O: Language-Controllable Object-Centric Visual Representation Learningcited by 12
  25. 2025Dyn-O: Building Structured World Models with Object-Centric Representationscited by 12
  26. 2025Predicting Video Slot Attention Queries from Random Slot-Feature Pairscited by 12
  27. 2023Understanding Physical Dynamics with Counterfactual World Modelingcited by 11
  28. 2025Vector-Quantized Vision Foundation Models for Object-Centric Learningcited by 11
  29. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  30. 2024Slot Abstractors: Toward Scalable Abstract Visual Reasoningcited by 10
  31. 2024Towards Scene Graph Anticipationcited by 10
  32. 2024On permutation-invariant neural networkscited by 10
  33. 2024SPARTAN: A Sparse Transformer World Model Attending to What Matterscited by 10
  34. 2024Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learningcited by 9
  35. 2023Intrinsic Physical Concepts Discovery with Object-Centric Predictive Modelscited by 8
  36. 2023Benchmarks for Physical Reasoning AIcited by 8
  37. 2024CarFormer: Self-Driving with Learned Object-Centric Representationscited by 8
  38. 2024EC-Diffuser: Multi-Object Manipulation via Entity-Centric Behavior Generationcited by 8

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SlotGNN: Unsupervised Discovery of Multi-Object Representations and Visual Dynamics
  2. 2025Learning Object-Centric Representations Based on Slots in Real World Scenarios
  3. 2025SlotPi: Physics-informed Object-centric Reasoning Models
  4. 2024Slot-VLM: SlowFast Slots for Video-Language Modeling
  5. 2025Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers
  6. 2026Better Slots, Better Worlds: Representation Quality & Robustness in Object-Centric World Models
  7. 2026Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations
  8. 2026StrucPhysVideo: Learning Physical Dynamics from Structured Captions and Robot Actions