mlsanityData to 14 Oct 2026

Object Scene Representation Transformer

arXiv 2206.06922 ↗First posted 14 Jun 2022cs.CV cs.AI cs.LGcited by 90

Not on the map. Why?

Abstract (arXiv)

A compositional understanding of the world in terms of objects and their geometry in 3D space is considered a cornerstone of human cognition. Facilitating the learning of such a representation in neural networks holds promise for substantially improving labeled data efficiency. As a key step in this direction, we make progress on the problem of learning 3D-consistent decompositions of complex scenes into individual objects in an unsupervised fashion. We introduce Object Scene Representation Transformer (OSRT), a 3D-centric model in which individual object representations naturally emerge through novel view synthesis. OSRT scales to significantly more complex scenes with larger diversity of objects and backgrounds than existing methods. At the same time, it is multiple orders of magnitude faster at compositional rendering thanks to its light field parametrization and the novel Slot Mixer decoder. We believe this work will not only accelerate future architecture exploration and scaling efforts, but it will also serve as a useful tool for both object-centric as well as neural scene representation learning communities.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

31 of 47 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Layer Normalizationcited by 5,669
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2015Spatial Transformer Networkscited by 1,429
  7. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  8. 2020On Layer Normalization in the Transformer Architecturecited by 922
  9. 2020Object-Centric Learning with Slot Attentioncited by 876
  10. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  11. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  12. 2018Deep Learning for Generic Object Detection: A Surveycited by 392
19 more
  1. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  2. 2022Kubric: A scalable dataset generatorcited by 381
  3. 2021Advances in Neural Renderingcited by 365
  4. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  5. 2021Conditional Object-Centric Learning from Videocited by 248
  6. 2020On the Binding Problem in Artificial Neural Networkscited by 233
  7. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 227
  8. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  9. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  10. 2021Illiterate DALL-E Learns to Composecited by 147
  11. 2019Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEscited by 117
  12. 2021NeRF-VAE: A Geometry Aware 3D Scene Generative Modelcited by 112
  13. 2021Unsupervised Discovery of Object Radiance Fieldscited by 101
  14. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 100
  15. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  16. 2021Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentationcited by 88
  17. 2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
  18. 2020ROOTS: Object-Centric Representation and Rendering of 3D Scenescited by 40
  19. 2020Object-Centric Image Generation with Factored Depths, Locations, and Appearancescited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 38; 2024: 27; 2025: 10; 2026: 11.

90 papers.

Sort:

By area:

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  3. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  4. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  5. 2023Large Language Models for Robotics: A Surveycited by 133
  6. 2022Is Attention All That NeRF Needs?cited by 113
  7. 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 104
  8. 2023Object-Centric Slot Diffusioncited by 90
  9. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  10. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 80
  11. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Modelscited by 79
  12. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 79
38 more
  1. 2023Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similaritiescited by 78
  2. 2024Large Spatial Model: End-to-end Unposed Images to Semantic 3Dcited by 73
  3. 2023Free3D: Consistent Novel View Synthesis without 3D Representationcited by 72
  4. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  5. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  6. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 56
  7. 2023Invariant Slot Attention: Object Discovery with Slot-Centric Reference Framescited by 51
  8. 2023Provably Learning Object-Centric Representationscited by 45
  9. 2022RUST: Latent Neural Scene Representations from Unposed Imagerycited by 43
  10. 2022Neural Systematic Bindercited by 37
  11. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  12. 2024Temporally Consistent Object-Centric Learning by Contrasting Slotscited by 31
  13. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  14. 2023Rotating Features for Object Discoverycited by 29
  15. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  16. 2023Pseudo-Generalized Dynamic View Synthesis from a Videocited by 27
  17. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 27
  18. 2023Object Discovery from Motion-Guided Tokenscited by 25
  19. 2023GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigationcited by 24
  20. 2024From an Image to a Scene: Learning to Imagine the World from a Million 360 Videoscited by 23
  21. 2024Interaction Asymmetry: A General Principle for Learning Composable Abstractionscited by 20
  22. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 19
  23. 2023NViST: In the Wild New View Synthesis from a Single Image with Transformerscited by 19
  24. 2024Zero-Shot Object-Centric Representation Learningcited by 19
  25. 2023Inferring Hybrid Neural Fluid Fields from Videoscited by 18
  26. 2023Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learningcited by 17
  27. 2023Unlocking Slot Attention by Changing Optimal Transport Costscited by 16
  28. 2023DORSal: Diffusion for Object-centric Representations of Scenes et alcited by 16
  29. 2023Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesiscited by 15
  30. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
  31. 2024FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Featurescited by 14
  32. 2024Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Modelscited by 14
  33. 2023AutoRecon: Automated 3D Object Discovery and Reconstructioncited by 13
  34. 2024Learning to Compose: Improving Object Centric Learning by Injecting Compositionalitycited by 13
  35. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  36. 2023Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenescited by 12
  37. 2024Unsupervised Discovery of Object-Centric Neural Fieldscited by 12
  38. 2024SlotLifter: Slot-guided Feature Lifting for Learning Object-centric Radiance Fieldscited by 12

50 most cited of 90 All 90 (JSON)