# SE(3) Neural Potential Fields for 6-DoF Trajectory Planning Directly from Images Without Explicit 3D Reconstruction

arXiv: https://arxiv.org/abs/2609.24864 · HTML: https://mlsanity.com/paper/2609.24864/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)
- [Vision-Only Robot Navigation in a Neural Radiance World](https://mlsanity.com/paper/2110.00168/) (2021)
- [GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping](https://mlsanity.com/paper/2403.09637/) (2024)
- [CATNIPS: Collision Avoidance Through Neural Implicit Probabilistic Scenes](https://mlsanity.com/paper/2302.12931/) (2023)
- [Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans](https://mlsanity.com/paper/2107.01518/) (2021)
- [FastGeodis: Fast Generalised Geodesic Distance Transform](https://mlsanity.com/paper/2208.00001/) (2022)
- [Spatial RoboGrasp: Generalized Robotic Grasping Control Policy](https://mlsanity.com/paper/2505.20814/) (2025)
- [DiffusionRL: Efficient Training of Diffusion Policies for Robotic Grasping Using RL-Adapted Large-Scale Datasets](https://mlsanity.com/paper/2505.18876/) (2025)
- [Whole-Body Coordination for Dynamic Object Grasping with Legged Manipulators](https://mlsanity.com/paper/2508.08328/) (2025)
- [Grasping Trajectory Optimization with Point Clouds](https://mlsanity.com/paper/2403.05466/) (2024)
- [Online Trajectory Replanner for Dynamically Grasping Irregular Objects](https://mlsanity.com/paper/2501.17968/) (2025)
- [B4P: Simultaneous Grasp and Motion Planning for Object Placement via Parallelized Bidirectional Forests and Path Repair](https://mlsanity.com/paper/2504.04598/) (2025)
- [Benchmarking Shortcutting Techniques for Multi-Robot-Arm Motion Planning](https://mlsanity.com/paper/2508.05027/) (2025)
- [Safe Navigation using Neural Radiance Fields via Reachable Sets](https://mlsanity.com/paper/2604.26899/) (2026)

## Abstract (arXiv)

Reaching a 6-DoF grasp pose in clutter requires a collision-free trajectory, conventionally obtained by reconstructing the scene in 3D and planning inside that reconstruction, at the cost of its accuracy and compute. Potential fields learned directly from images remove that dependency but inherit the classical weakness of artificial potential fields: where attractive and repulsive gradients cancel, the descent grazes the obstacle instead of going around it, and can stall short of the goal. We present an SE(3) neural potential field learned from posed RGB images and supervised with a navigation function, the geodesic distance to the grasp through free space recovered from those same images during training, which removes both failures. On two tabletop scenes, from obstacle-blocked starts executed on a UR10, the field converges within 3 cm of the grasp from every start and every path it executes is collision-free against the ground-truth geometry, against 25% and 0% under image supervision alone; mean clearance rises from under a centimeter to 8.6-8.8 cm and arm-link contacts fall from 20.6-50.4% to 2.7-5.5% of executed configurations. Executed grasp success is 90.0% and 40.0% on the two scenes, the residual failures being refusals of the Cartesian executor rather than of the field. Planning takes about 2 s against 67-133 s for RRT* on a reconstruction of the same images, though under a common offline harness the two are comparable: the deployed margin is the cost of collision-checking a dense reconstruction, not planner complexity.