# Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering

arXiv: https://arxiv.org/abs/2604.16388 · HTML: https://mlsanity.com/paper/2604.16388/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 27 Mar 2026.

## Builds on (most cited first, 27 of 27)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Vision-Only Robot Navigation in a Neural Radiance World](https://mlsanity.com/paper/2110.00168/) (2021)
- [ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation](https://mlsanity.com/paper/2403.08321/) (2024)
- [Motion Planning Networks](https://mlsanity.com/paper/1806.05767/) (2018)
- [Motion Planning Networks: Bridging the Gap Between Learning-based and Classical Motion Planners](https://mlsanity.com/paper/1907.06013/) (2019)
- [Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Maps](https://mlsanity.com/paper/2403.02751/) (2024)
- [Informed RRT*: Optimal Sampling-based Path Planning Focused via Direct Sampling of an Admissible Ellipsoidal Heuristic](https://mlsanity.com/paper/1404.2334/) (2014)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [3D-HGS: 3D Half-Gaussian Splatting](https://mlsanity.com/paper/2406.02720/) (2024)
- [Robot Motion Planning in Learned Latent Spaces](https://mlsanity.com/paper/1807.10366/) (2018)
- [GWM: Towards Scalable Gaussian World Models for Robotic Manipulation](https://mlsanity.com/paper/2508.17600/) (2025)
- [Differentiable Robot Rendering](https://mlsanity.com/paper/2410.13851/) (2024)
- [Deeply Informed Neural Sampling for Robot Motion Planning](https://mlsanity.com/paper/1809.10252/) (2018)
- [Controlling diverse robots by inferring Jacobian fields with deep networks](https://mlsanity.com/paper/2407.08722/) (2024)
- [RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training](https://mlsanity.com/paper/2411.17662/) (2024)
- [Potential Functions based Sampling Heuristic For Optimal Path Planning](https://mlsanity.com/paper/1704.00264/) (2017)
- [Real-time Holistic Robot Pose Estimation with Unknown States](https://mlsanity.com/paper/2402.05655/) (2024)
- [SAFER-Splat: A Control Barrier Function for Safe Navigation with Online Gaussian Splatting Maps](https://mlsanity.com/paper/2409.09868/) (2024)
- [ST-RRT*: Asymptotically-Optimal Bidirectional Motion Planning through Space-Time](https://mlsanity.com/paper/2203.02176/) (2022)
- [Neural Informed RRT*: Learning-based Path Planning with Point Cloud State Representations under Admissible Ellipsoidal Constraints](https://mlsanity.com/paper/2309.14595/) (2023)
- [Revisiting the Asymptotic Optimality of RRT^*](https://mlsanity.com/paper/1909.09688/) (2019)
- [iDb-RRT: Sampling-based Kinodynamic Motion Planning with Motion Primitives and Trajectory Optimization](https://mlsanity.com/paper/2403.10745/) (2024)
- [Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisions](https://mlsanity.com/paper/2503.11269/) (2025)
- [Bi-AM-RRT*: A Fast and Efficient Sampling-Based Motion Planning Algorithm in Dynamic Environments](https://mlsanity.com/paper/2301.11816/) (2023)

## Abstract (arXiv)

Rapidly-exploring random trees (RRTs) have been widely adopted for robot motion planning due to their robustness and theoretical guarantees. However, existing RRT-based planners require explicit goal configurations specified as numerical joint angles, while many practical applications provide goal specifications through visual observations such as images or demonstration videos where precise goal configurations are unavailable. In this paper, we propose visual-RRT (vRRT), a motion planner that enables visual-goal planning by unifying gradient-based exploitation from differentiable robot rendering with sampling-based exploration from RRTs. We further introduce (i) a frontier-based exploration-exploitation strategy that adaptively prioritizes visually promising search regions, and (ii) inertial gradient tree expansion that inherits optimization states across tree branches for momentum-consistent gradient exploitation. Extensive experiments across various robot manipulators including Franka, UR5e, and Fetch demonstrate that vRRT achieves effective visual-goal planning in both simulated and real-world settings, bridging the gap between sampling-based planning and vision-centric robot applications. Our code is available at https://sgvr.kaist.ac.kr/Visual-RRT.