mlsanityData to 7 Oct 2026

Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning

arXiv 2110.00792 ↗First posted 2 Oct 2021cs.RO cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance fields that represent objects in the scene. Most manipulation planning approaches rely on analytical models and carefully chosen abstractions/state-spaces to be effective. A central question is how models can be obtained from data that are not primarily accurate in their predictions, but, more importantly, enable efficient reasoning within a planning framework, while at the same time being closely coupled to perception spaces. We show that representing objects as signed-distance fields not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations, but also that SDF-based models are suitable for optimization-based planning. To demonstrate the versatility of our approach, we learn both kinematic and dynamic models to solve tasks that involve hanging mugs on hooks and pushing objects on a table. We can unify these quite different tasks within one framework, since SDFs are the common object representation. Video: https://youtu.be/ga8Wlkss7co

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 2 Oct 2021.

Builds on

28 of 47 references.

References · 28

  1. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  2. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  3. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  4. 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
  5. 2019SAL: Sign Agnostic Learning of Shapes from Raw Datacited by 389
  6. 2020Local Implicit Grid Representations for 3D Scenescited by 372
  7. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  8. 2020Integrated Task and Motion Planningcited by 281
  9. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  10. 2020MetaSDF: Meta-learning Signed Distance Functionscited by 214
  11. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 107
  12. 2019Learning to Infer Implicit Surfaces without 3D Supervisioncited by 103
16 more
  1. 2021Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Cluttercited by 94
  2. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 69
  3. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
  4. 2016SE3-Nets: Learning Rigid Body Motion using Deep Neural Networkscited by 48
  5. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 45
  6. 2020Object Rearrangement Using Learned Implicit Collision Functionscited by 43
  7. 2020Learning 3D Dynamic Scene Representations for Robot Manipulationcited by 39
  8. 2020ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representationscited by 38
  9. 2019Learning Continuous 3D Reconstructions for Geometrically Aware Graspingcited by 28
  10. 2020Learning to Identify Physical Parameters from Video Using Differentiable Physicscited by 11
  11. 2020Learning Equality Constraints for Motion Planning on Manifoldscited by 9
  12. 2021OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimationcited by 8
  13. 2020Describing Physics For Physical Reasoning: Force-based Sequential Manipulation Planningcited by 6
  14. 2020Robust Task and Motion Planning for Long-Horizon Architectural Construction Planningcited by 6
  15. 2017Reactive Planar Manipulation with Convex Hybrid MPCcited by 4
  16. 2019Hybrid Differential Dynamic Programming for Planar Manipulation Primitivescited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 9; 2023: 12; 2024: 5; 2025: 2.

29 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 125
  3. 2022iSDF: Real-Time Neural Signed Distance Fields for Robot Perceptioncited by 107
  4. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  5. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  6. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  7. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  8. 2023LODE: Locally Conditioned Eikonal Implicit Scene Completion from Sparse LiDARcited by 31
  9. 2022GenSDF: Two-Stage Learning of Generalizable Signed Distance Functionscited by 30
  10. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  11. 2023DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observabilitycited by 21
  12. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
17 more
  1. 2022PRIF: Primary Ray-based Implicit Functioncited by 17
  2. 2024Composable Part-Based Manipulationcited by 17
  3. 2023TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensingcited by 15
  4. 2023Few-Shot In-Context Imitation Learning via Implicit Graph Alignmentcited by 14
  5. 2022NeuralGrasps: Learning Implicit Representations for Grasps of Multiple Robotic Handscited by 12
  6. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  7. 2024Learning rigid-body simulators over implicit shapes for large-scale scenes and visioncited by 12
  8. 2022RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Controlcited by 10
  9. 2022Where To Start? Transferring Simple Skills to Complex Environmentscited by 9
  10. 2023Local object crop collision network for efficient simulation of non-convex objects in GPU-based simulatorscited by 5
  11. 2023STAMP: Differentiable Task and Motion Planning via Stein Variational Gradient Descentcited by 4
  12. 2024DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Renderingcited by 4
  13. 2021Learning Linear Complementarity Systemscited by 3
  14. 2023HIO-SDF: Hierarchical Incremental Online Signed Distance Fieldscited by 3
  15. 2025Neural Configuration-Space Barriers for Manipulation Planning and Controlcited by 3
  16. 2023Rejecting Cognitivism: Computational Phenomenology for Deep Learningcited by 2
  17. 2025Efficient Construction of Implicit Surface Models From a Single Image for Motion Generationcited by 0