mlsanityData to 7 Oct 2026

Learning rigid-body simulators over implicit shapes for large-scale scenes and vision

arXiv 2405.14045 ↗First posted 22 May 2024cs.LG cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Simulating large scenes with many rigid objects is crucial for a variety of applications, such as robotics, engineering, film and video games. Rigid interactions are notoriously hard to model: small changes to the initial state or the simulation parameters can lead to large changes in the final state. Recently, learned simulators based on graph networks (GNNs) were developed as an alternative to hand-designed simulators like MuJoCo and PyBullet. They are able to accurately capture dynamics of real objects directly from real-world observations. However, current state-of-the-art learned simulators operate on meshes and scale poorly to scenes with many objects or detailed shapes. Here we present SDF-Sim, the first learned rigid-body simulator designed for scale. We use learned signed-distance functions (SDFs) to represent the object shapes and to speed up distance computation. We design the simulator to leverage SDFs and avoid the fundamental bottleneck of the previous simulators associated with collision detection. For the first time in literature, we demonstrate that we can scale the GNN-based simulators to scenes with hundreds of objects and up to 1.1 million nodes, where mesh-based approaches run out of memory. Finally, we show that SDF-Sim can be applied to real world scenes by extracting SDFs from multi-view images.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

35 of 47 references.

References · 35

  1. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  2. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  3. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  4. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  5. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  6. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  7. 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
  8. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  9. 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
  10. 2021Neural RGB-D Surface Reconstructioncited by 312
  11. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  12. 2020Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfacescited by 146
23 more
  1. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  2. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functionscited by 133
  3. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 113
  4. 2022iSDF: Real-Time Neural Signed Distance Fields for Robot Perceptioncited by 107
  5. 2018Propagation Networks for Model-Based Control Under Partial Observationcited by 64
  6. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  7. 2022Differentiable Physics Simulation of Dynamics-Augmented Neural Objectscited by 45
  8. 2022Learning rigid dynamics with face interaction graph networkscited by 43
  9. 2023Fast-Grasp'D: Dexterous Multi-finger Grasp Generation Through Differentiable Simulationcited by 35
  10. 2021Learning Models as Functionals of Signed-Distance Fields for Manipulation Planningcited by 29
  11. 2021Dynamic Neural Garmentscited by 27
  12. 2023Learning Signed Distance Functions from Noisy 3D Point Clouds via Noise to Noise Mappingcited by 27
  13. 2022VIRDO: Visio-tactile Implicit Representations of Deformable Objectscited by 25
  14. 2023What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discoverycited by 22
  15. 2023Dynamic Mesh-Aware Radiance Fieldscited by 21
  16. 2023CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generationcited by 17
  17. 2023Learning 3D Particle-based Simulators from RGB-D Videoscited by 17
  18. 2021DiffSDFSim: Differentiable Rigid-Body Dynamics With Implicit Shapescited by 14
  19. 2021Validating Robotics Simulators on Real-World Impactscited by 11
  20. 2024Scaling Face Interaction Graph Networks to Real World Scenescited by 5
  21. 20223D Neural Sculpting (3DNS): Editing Neural Signed Distance Functionscited by 4
  22. 2023SURFSUP: Learning Fluid Simulation for Novel Surfacescited by 3
  23. 2021Neural Implicit Surfaces for Efficient and Accurate Collisions in Physically Based Simulationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 4.

12 papers.

  1. 2024Does equivariance matter at scale?cited by 49
  2. 2024Modeling the Real World with High-Density Visual Particle Dynamicscited by 12
  3. 2025Learning 3D-Gaussian Simulators from RGB Videoscited by 8
  4. 2026Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fieldscited by 5
  5. 2025Neural Force Field: Few-shot Learning of Generalized Physical Reasoningcited by 1
  6. 2025EqCollide: Equivariant and Collision-Aware Deformable Objects Neural Simulatorcited by 1
  7. 2025Recurrent U-Net-Based Graph Neural Network (RUGNN) for Accurate Deformation Predictions in Sheet Material Formingcited by 1
  8. 2026RigidFormer: Learning Rigid Dynamics using Transformerscited by 1
  9. 2026PhysiFormer: Learning to Simulate Mechanics in World Spacecited by 1
  10. 2025Learning Deformable Body Interactions With Adaptive Spatial Tokenizationcited by 0
  11. 2025Learning Particle Dynamics Subject to Rigid Body Manipulations Using Graph Neural Networkscited by 0
  12. 2026PersistGS: Differentiable Physics for Object Permanence in 4D Gaussian Splattingcited by 0