# Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition

arXiv: https://arxiv.org/abs/2407.00299 · HTML: https://mlsanity.com/paper/2407.00299/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 29 Jun 2024.

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Understanding Diffusion Models: A Unified Perspective](https://mlsanity.com/paper/2208.11970/) (2022)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System](https://mlsanity.com/paper/1910.03135/) (2019)
- [Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment](https://mlsanity.com/paper/2211.08416/) (2022)
- [XSkill: Cross Embodiment Skill Discovery](https://mlsanity.com/paper/2307.09955/) (2023)
- [Shared Autonomy via Deep Reinforcement Learning](https://mlsanity.com/paper/1802.01744/) (2018)
- [Holo-Dex: Teaching Dexterity with Immersive Mixed Reality](https://mlsanity.com/paper/2210.06463/) (2022)
- [AirExo: Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild](https://mlsanity.com/paper/2309.14975/) (2023)
- [Shared Autonomy via Hindsight Optimization](https://mlsanity.com/paper/1503.07619/) (2015)
- [Shared Autonomy via Hindsight Optimization for Teleoperation and Teaming](https://mlsanity.com/paper/1706.00155/) (2017)
- [Shared Autonomy with Learned Latent Actions](https://mlsanity.com/paper/2005.03210/) (2020)
- [Residual Policy Learning for Shared Autonomy](https://mlsanity.com/paper/2004.05097/) (2020)
- [Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation](https://mlsanity.com/paper/1810.01845/) (2018)
- [To the Noise and Back: Diffusion for Shared Autonomy](https://mlsanity.com/paper/2302.12244/) (2023)
- [Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing](https://mlsanity.com/paper/1703.01270/) (2017)
- [Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network](https://mlsanity.com/paper/1809.06268/) (2018)
- [Learning Latent Actions to Control Assistive Robots](https://mlsanity.com/paper/2107.02907/) (2021)
- [HandTailor: Towards High-Precision Monocular 3D Hand Recovery](https://mlsanity.com/paper/2102.09244/) (2021)

## Built on by (most cited first, 10 of 10)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletons](https://mlsanity.com/paper/2503.03081/) (2025)
- [Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols](https://mlsanity.com/paper/2512.02787/) (2025)
- [SIME: Enhancing Policy Self-Improvement with Modal-level Exploration](https://mlsanity.com/paper/2505.01396/) (2025)
- [AoE: Always-on Egocentric Human Video Collection for Embodied AI](https://mlsanity.com/paper/2602.23893/) (2026)
- [Embodiment-Aware Generalist Specialist Distillation for Unified Humanoid Whole-Body Control](https://mlsanity.com/paper/2602.02960/) (2026)
- [InterPReT: Interactive Policy Restructuring and Training Enable Effective Imitation Learning from Laypersons](https://mlsanity.com/paper/2602.04213/) (2026)
- [DexMani: Human-Derived Manipulability Guidance for Dexterous Rotation](https://mlsanity.com/paper/2608.00554/) (2026)
- [Assistron: Bayesian Shared Autonomy with Off-the-shelf Vision-Language-Action Models](https://mlsanity.com/paper/2606.23147/) (2026)
- [Touch2Robot: Robot Touch in the Human Demonstration Loop](https://mlsanity.com/paper/2609.24660/) (2026)

## Abstract (arXiv)

Employing a teleoperation system for gathering demonstrations offers the potential for more efficient learning of robot manipulation. However, teleoperating a robot arm equipped with a dexterous hand or gripper, via a teleoperation system presents inherent challenges due to the task's high dimensionality, complexity of motion, and differences between physiological structures. In this study, we introduce a novel system for joint learning between human operators and robots, that enables human operators to share control of a robot end-effector with a learned assistive agent, simplifies the data collection process, and facilitates simultaneous human demonstration collection and robot manipulation training. As data accumulates, the assistive agent gradually learns. Consequently, less human effort and attention are required, enhancing the efficiency of the data collection process. It also allows the human operator to adjust the control ratio to achieve a trade-off between manual and automated control. We conducted experiments in both simulated environments and physical real-world settings. Through user studies and quantitative evaluations, it is evident that the proposed system could enhance data collection efficiency and reduce the need for human adaptation while ensuring the collected data is of sufficient quality for downstream tasks. textitFor more details, please refer to our webpage https://norweig1an.github.io/HAJL.github.io/.