# Autonomous Human-Robot Interaction via Operator Imitation

arXiv: https://arxiv.org/abs/2504.02724 · HTML: https://mlsanity.com/paper/2504.02724/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 50 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 30 of 35)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control](https://mlsanity.com/paper/2410.03441/) (2024)
- [Learning Latent Representations to Influence Multi-Agent Interaction](https://mlsanity.com/paper/2011.06619/) (2020)
- [HOVER: Versatile Neural Whole-Body Controller for Humanoid Robots](https://mlsanity.com/paper/2410.21229/) (2024)
- [Taming Diffusion Probabilistic Models for Character Control](https://mlsanity.com/paper/2404.15121/) (2024)
- [Physically Plausible Full-Body Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2309.07907/) (2023)
- [Efficient Model Learning for Human-Robot Collaborative Tasks](https://mlsanity.com/paper/1405.6341/) (2014)
- [Learning Human-to-Robot Handovers from Point Clouds](https://mlsanity.com/paper/2303.17592/) (2023)
- [To the Noise and Back: Diffusion for Shared Autonomy](https://mlsanity.com/paper/2302.12244/) (2023)
- [Design and Control of a Bipedal Robotic Character](https://mlsanity.com/paper/2501.05204/) (2025)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)
- [Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimicking](https://mlsanity.com/paper/2412.17730/) (2024)
- [Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks](https://mlsanity.com/paper/2305.12171/) (2023)
- [SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handovers](https://mlsanity.com/paper/2311.05599/) (2023)
- [Intrinsically motivated reinforcement learning for human-robot interaction in the real-world](https://mlsanity.com/paper/1804.05259/) (2018)
- [It Takes Two: Learning to Plan for Human-Robot Cooperative Carrying](https://mlsanity.com/paper/2209.12890/) (2022)
- [Imitating by generating: deep generative models for imitation of interactive tasks](https://mlsanity.com/paper/1910.06031/) (2019)
- [APRICOT: Active Preference Learning and Constraint-Aware Task Planning with LLMs](https://mlsanity.com/paper/2410.19656/) (2024)
- [Human-Robot Gym: Benchmarking Reinforcement Learning in Human-Robot Collaboration](https://mlsanity.com/paper/2310.06208/) (2023)
- [Robot gains Social Intelligence through Multimodal Deep Reinforcement Learning](https://mlsanity.com/paper/1702.07492/) (2017)
- [Robot Interaction Behavior Generation based on Social Motion Forecasting for Human-Robot Interaction](https://mlsanity.com/paper/2402.04768/) (2024)

## Built on by (most cited first, 3 of 3)

- [Robot Crash Course: Learning Soft and Stylized Falling](https://mlsanity.com/paper/2511.10635/) (2025)
- [Structured Imitation Learning of Interactive Policies through Inverse Games](https://mlsanity.com/paper/2511.12848/) (2025)
- [MulDP: Multimodal Diffusion Policy for Autonomous Quadruped Parkour Navigation across Complex Terrains](https://mlsanity.com/paper/2609.03984/) (2026)

## Abstract (arXiv)

Teleoperated robotic characters can perform expressive interactions with humans, relying on the operators' experience and social intuition. In this work, we propose to create autonomous interactive robots, by training a model to imitate operator data. Our model is trained on a dataset of human-robot interactions, where an expert operator is asked to vary the interactions and mood of the robot, while the operator commands as well as the pose of the human and robot are recorded. Our approach learns to predict continuous operator commands through a diffusion process and discrete commands through a classifier, all unified within a single transformer architecture. We evaluate the resulting model in simulation and with a user study on the real system. We show that our method enables simple autonomous human-robot interactions that are comparable to the expert-operator baseline, and that users can recognize the different robot moods as generated by our model. Finally, we demonstrate a zero-shot transfer of our model onto a different robotic platform with the same operator interface.