# Shared Autonomy via Hindsight Optimization for Teleoperation and Teaming

arXiv: https://arxiv.org/abs/1706.00155 · HTML: https://mlsanity.com/paper/1706.00155/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2017.

## Built on by (most cited first, 30 of 40)

- [Language-Driven Representation Learning for Robotics](https://mlsanity.com/paper/2302.12766/) (2023)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [LILA: Language-Informed Latent Actions](https://mlsanity.com/paper/2111.03205/) (2021)
- [NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants](https://mlsanity.com/paper/2301.05223/) (2023)
- [Physical Interaction as Communication: Learning Robot Objectives Online from Human Corrections](https://mlsanity.com/paper/2107.02349/) (2021)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots](https://mlsanity.com/paper/2404.04066/) (2024)
- [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023)
- [Gaze-based intention estimation: principles, methodologies, and applications in HRI](https://mlsanity.com/paper/2302.04530/) (2023)
- [Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition](https://mlsanity.com/paper/2407.00299/) (2024)
- [Assistive Teaching of Motor Control Tasks to Humans](https://mlsanity.com/paper/2211.14003/) (2022)
- [Surrogate Assisted Generation of Human-Robot Interaction Scenarios](https://mlsanity.com/paper/2304.13787/) (2023)
- [Robot Trajectron: Trajectory Prediction-based Shared Control for Robot Manipulation](https://mlsanity.com/paper/2402.02499/) (2024)
- [Genie Centurion: Accelerating Scalable Real-World Robot Training with Human Rewind-and-Refine Guidance](https://mlsanity.com/paper/2505.18793/) (2025)
- [Conformalized Teleoperation: Confidently Mapping Human Inputs to High-Dimensional Robot Actions](https://mlsanity.com/paper/2406.07767/) (2024)
- [Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration](https://mlsanity.com/paper/2411.02599/) (2024)
- [On Optimizing Interventions in Shared Autonomy](https://mlsanity.com/paper/2112.09169/) (2021)
- [Bootstrapping Linear Models for Fast Online Adaptation in Human-Agent Collaboration](https://mlsanity.com/paper/2404.10733/) (2024)
- [A Bayesian Approach to Identifying Representational Errors](https://mlsanity.com/paper/2103.15171/) (2021)
- [Collaborative Decision Making Using Action Suggestions](https://mlsanity.com/paper/2209.13160/) (2022)
- [Coaching a Robotic Sonographer: Learning Robotic Ultrasound with Sparse Expert's Feedback](https://mlsanity.com/paper/2409.02337/) (2024)
- [Towards an LLM-Based Speech Interface for Robot-Assisted Feeding](https://mlsanity.com/paper/2410.20624/) (2024)
- [ProVox: Personalization and Proactive Planning for Situated Human-Robot Collaboration](https://mlsanity.com/paper/2506.12248/) (2025)
- [Probabilistic Human Intent Prediction for Mobile Manipulation: An Evaluation with Human-Inspired Constraints](https://mlsanity.com/paper/2507.10131/) (2025)
- [Shared Control with Black Box Agents using Oracle Queries](https://mlsanity.com/paper/2410.19612/) (2024)
- [Learning Multimodal AI Algorithms for Amplifying Limited User Input into High-dimensional Control Space](https://mlsanity.com/paper/2505.11366/) (2025)
- [Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models](https://mlsanity.com/paper/2506.14727/) (2025)
- [Enhancing Goal Inference via Correction Timing](https://mlsanity.com/paper/2602.18603/) (2026)
- [Confidence-Gated Robot Autonomy: When Does Uncertainty Actually Help?](https://mlsanity.com/paper/2605.18045/) (2026)
- [Deep Reinforcement Learning for Haptic Shared Control in Unknown Tasks](https://mlsanity.com/paper/2101.06227/) (2021)

## Abstract (arXiv)

In shared autonomy, a user and autonomous system work together to achieve shared goals. To collaborate effectively, the autonomous system must know the user's goal. As such, most prior works follow a predict-then-act model, first predicting the user's goal with high confidence, then assisting given that goal. Unfortunately, confidently predicting the user's goal may not be possible until they have nearly achieved it, causing predict-then-act methods to provide little assistance. However, the system can often provide useful assistance even when confidence for any single goal is low (e.g. move towards multiple goals). In this work, we formalize this insight by modelling shared autonomy as a Partially Observable Markov Decision Process (POMDP), providing assistance that minimizes the expected cost-to-go with an unknown goal. As solving this POMDP optimally is intractable, we use hindsight optimization to approximate. We apply our framework to both shared-control teleoperation and human-robot teaming. Compared to predict-then-act methods, our method achieves goals faster, requires less user input, decreases user idling time, and results in fewer user-robot collisions.