# Invisible Servoing: a Visual Servoing Approach with Return-Conditioned Latent Diffusion

arXiv: https://arxiv.org/abs/2409.13337 · HTML: https://mlsanity.com/paper/2409.13337/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 18 of 18)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Cross-modal Deep Variational Hand Pose Estimation](https://mlsanity.com/paper/1803.11404/) (2018)
- [Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations](https://mlsanity.com/paper/1909.06993/) (2019)
- [Multi-UAV trajectory planning for 3D visual inspection of complex structures](https://mlsanity.com/paper/2204.10070/) (2022)
- [Model Predictive Spherical Image-Based Visual Servoing On SO(3) for Aggressive Aerial Tracking](https://mlsanity.com/paper/2212.09613/) (2022)
- [Direct Visual Servoing Based on Discrete Orthogonal Moments](https://mlsanity.com/paper/2304.14012/) (2023)

## Built on by (most cited first, 1 of 1)

- [Denoising Diffusion Planner: Learning Complex Paths from Low-Quality Demonstrations](https://mlsanity.com/paper/2410.21497/) (2024)

## Abstract (arXiv)

In this paper, we present a novel visual servoing (VS) approach based on latent Denoising Diffusion Probabilistic Models (DDPMs), that explores the application of generative models for vision-based navigation of UAVs (Uncrewed Aerial Vehicles). Opposite to classical VS methods, the proposed approach allows reaching the desired target view, even when the target is initially not visible. This is possible thanks to the learning of a latent representation that the DDPM uses for planning and a dataset of trajectories encompassing target-invisible initial views. A compact representation is learned from raw images using a Cross-Modal Variational Autoencoder. Given the current image, the DDPM generates trajectories in the latent space driving the robotic platform to the desired visual target. The approach has been validated in simulation using two generic multi-rotor UAVs (a quadrotor and a hexarotor). The results show that we can successfully reach the visual target, even if not visible in the initial view.