# HITL-D: Human In The Loop Diffusion Assisted Shared Control

arXiv: https://arxiv.org/abs/2605.21460 · HTML: https://mlsanity.com/paper/2605.21460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 20 May 2026.

## Builds on (most cited first, 20 of 20)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [Shared Autonomy via Hindsight Optimization](https://mlsanity.com/paper/1503.07619/) (2015)
- [Conditional Image Synthesis with Diffusion Models: A Survey](https://mlsanity.com/paper/2409.19365/) (2024)
- [Conditional Denoising Diffusion for Sequential Recommendation](https://mlsanity.com/paper/2304.11433/) (2023)
- [Learning Models for Shared Control of Human-Machine Systems with Unknown Dynamics](https://mlsanity.com/paper/1808.08268/) (2018)
- [The Sense of Agency in Assistive Robotics Using Shared Autonomy](https://mlsanity.com/paper/2501.07462/) (2025)
- [A Quantitative Analysis of Activities of Daily Living: Insights into Improving Functional Independence with Assistive Robotics](https://mlsanity.com/paper/2104.03892/) (2021)
- [Learning State Conditioned Linear Mappings for Low-Dimensional Control of Robotic Manipulators](https://mlsanity.com/paper/2410.21441/) (2024)
- [Point and Go: Intuitive Reference Frame Reallocation in Mode Switching for Assistive Robotics](https://mlsanity.com/paper/2510.08753/) (2025)

## Abstract (arXiv)

Autonomous manipulation systems have achieved remarkable capabilities, yet the integration of human expertise with diffusion-based policies in shared control remains relatively unexplored. In this paper, we propose Human-In-The-Loop Diffusion (HITL-D), a shared control framework that enhances user performance in multi-step, insertion, and fine manipulation tasks. HITL-D leverages a novel combination of diffusion-based policies and human control to provide autonomous end effector orientation updates conditioned on a scene point cloud and the Cartesian position of the end effector. This approach reduces the number of joystick control axes required, thereby lowering mental workload. In a multi-task user study with 12 participants, HITL-D reduced average task completion times by 40%, decreased perceived workload by 37%, and improved Likert-scale ratings for independence, intuitiveness, and confidence compared to traditional teleoperation methods. These results demonstrate that HITL-D effectively integrates human expertise with autonomous assistance, improving both objective and subjective aspects of teleoperation.