mlsanityData to 7 Oct 2026

Learning 3D Dynamic Scene Representations for Robot Manipulation

arXiv 2011.01968 ↗First posted 3 Nov 2020cs.RO cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

3D scene representation for robot manipulation should capture three key object properties: permanency -- objects that become occluded over time continue to exist; amodal completeness -- objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity -- the movement of each object is continuous over space and time. In this paper, we introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene representation that simultaneously discovers, tracks, reconstructs objects, and predicts their dynamics while capturing all three properties. We further propose DSR-Net, which learns to aggregate visual observations over multiple interactions to gradually build and refine DSR. Our model achieves state-of-the-art performance in modeling 3D scene dynamics with DSR on both simulated and real data. Combined with model predictive control, DSR-Net enables accurate planning in downstream robotic manipulation tasks such as planar pushing. Video is available at https://youtu.be/GQjYG3nQJ80.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 10; 2023: 8; 2024: 8; 2025: 7; 2026: 2.

39 papers.

Sort:

By area:

  1. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  2. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  3. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  4. 2022CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphismcited by 56
  5. 2023Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmarkcited by 51
  6. 2021Act the Part: Learning Interaction Strategies for Articulated Object Part Discoverycited by 45
  7. 2023Pyramid Diffusion for Fine 3D Large Scene Generationcited by 44
  8. 2024PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulationcited by 36
  9. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
  10. 2021Learning Models as Functionals of Signed-Distance Fields for Manipulation Planningcited by 29
  11. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  12. 2022ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulationcited by 24
27 more
  1. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  2. 2022A Survey on Machine Learning Approaches for Modelling Intuitive Physicscited by 20
  3. 2022DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulationcited by 18
  4. 2022Revealing Occlusions with 4D Neural Fieldscited by 17
  5. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  6. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  7. 2024Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Modelscited by 12
  8. 2021Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assemblycited by 6
  9. 2023Neural Field Dynamics Model for Granular Object Piles Manipulationcited by 6
  10. 2023Teaching Robots to Build Simulations of Themselvescited by 6
  11. 2023Perceiving Unseen 3D Objects by Poking the Objectscited by 5
  12. 2023Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Modelscited by 5
  13. 2024Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Priorcited by 5
  14. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  15. 2026APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policiescited by 5
  16. 2022LOPR: Latent Occupancy PRediction using Generative Modelscited by 4
  17. 2024DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Renderingcited by 4
  18. 2025Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generationcited by 4
  19. 2025FUNCanon: Learning Pose-Aware Action Primitives via Functional Object Canonicalization for Generalizable Robotic Manipulationcited by 3
  20. 2023Object-Centric Scene Representations using Active Inferencecited by 2
  21. 2025CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3Dcited by 2
  22. 2022T3VIP: Transformation-based 3D Video Predictioncited by 1
  23. 2025Estimating Scene Flow in Robot Surroundings with Distributed Miniaturized Time-of-Flight Sensorscited by 1
  24. 2025UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learningcited by 1
  25. 2022A Review of Scene Representations for Robot Manipulatorscited by 0
  26. 2024Leveraging Foundation Models To learn the shape of semi-fluid deformable objectscited by 0
  27. 2026Category-Level 3D Correspondence in Camera Space via Morphable Object Priorscited by 0