mlsanityData to 14 Oct 2026

Multi-Object Manipulation via Object-Centric Neural Scattering Functions

arXiv 2306.08748 ↗First posted 14 Jun 2023cs.RO cs.AI cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Learned visual dynamics models have proven effective for robotic manipulation tasks. Yet, it remains unclear how best to represent scenes involving multi-object interactions. Current methods decompose a scene into discrete objects, but they struggle with precise modeling and manipulation amid challenging lighting conditions as they only encode appearance tied with specific illuminations. In this work, we propose using object-centric neural scattering functions (OSFs) as object representations in a model-predictive control framework. OSFs model per-object light transport, enabling compositional scene re-rendering under object rearrangement and varying lighting conditions. By combining this approach with inverse parameter estimation and graph-based neural dynamics models, we demonstrate improved model-predictive control performance and generalization in compositional multi-object environments, even in previously unseen scenarios and harsh lighting conditions.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 52 of its 64 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

52 of 64 references.

References · 52

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2018World Modelscited by 1,513
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  6. 2019Objects as Pointscited by 1,175
  7. 2020Mastering Atari with Discrete World Modelscited by 1,007
  8. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  9. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 673
  10. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  11. 2018PVNet: Pixel-wise Voting Network for 6DoF Pose Estimationcited by 385
  12. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 382
38 more
  1. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  2. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  3. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  4. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  5. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 220
  6. 2019Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimationcited by 218
  7. 2017Real-Time Seamless Single Shot 6D Object Pose Predictioncited by 210
  8. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  9. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  10. 2019DPOD: 6D Pose Object Detector and Refinercited by 189
  11. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  12. 2021Vision-Only Robot Navigation in a Neural Radiance Worldcited by 155
  13. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 153
  14. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  15. 2021Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objectscited by 119
  16. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
  17. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  18. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 108
  19. 2021Unsupervised Discovery of Object Radiance Fieldscited by 101
  20. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 99
  21. 2019PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Trackingcited by 91
  22. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  23. 2020Causal Discovery in Physical Systems from Videoscited by 77
  24. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  25. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 70
  26. 2018Propagation Networks for Model-Based Control Under Partial Observationcited by 64
  27. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  28. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 62
  29. 2020Visual Grounding of Learned Physical Modelscited by 55
  30. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  31. 2022Differentiable Physics Simulation of Dynamics-Augmented Neural Objectscited by 45
  32. 2020Object Rearrangement Using Learned Implicit Collision Functionscited by 43
  33. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  34. 2020VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulationcited by 40
  35. 2021NeRP: Neural Rearrangement Planning for Unknown Objectscited by 34
  36. 2022Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulationcited by 34
  37. 2020From Points to Multi-Object 3D Reconstructioncited by 29
  38. 2022Unsupervised Discovery and Composition of Object Light Fieldscited by 28

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 3; 2026: 1.

9 papers.

  1. 2023Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmarkcited by 51
  2. 2025Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Datasetcited by 29
  3. 2023Inferring Hybrid Neural Fluid Fields from Videoscited by 18
  4. 2025Do computer vision foundation models learn the low-level characteristics of the human visual system?cited by 12
  5. 2024DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Renderingcited by 4
  6. 2024SUP-NeRF: A Streamlined Unification of Pose Estimation and NeRF for Monocular 3D Object Reconstructioncited by 3
  7. 2026AdaReP:Adaptive Re-Planning under Model Mismatch for Neural World-Model Predictive Controlcited by 3
  8. 2023Multi-Object Graph Affordance Network: Goal-Oriented Planning through Learned Compound Object Affordancescited by 2
  9. 2025Break Stylistic Sophon: Are We Really Meant to Confine the Imagination in Style Transfer?cited by 0