# Semi-Autonomous Teleoperation via Learning Non-Prehensile Manipulation Skills

arXiv: https://arxiv.org/abs/2109.13081 · HTML: https://mlsanity.com/paper/2109.13081/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Shared Autonomy with Learned Latent Actions](https://mlsanity.com/paper/2005.03210/) (2020)
- [Controlling Assistive Robots with Learned Latent Actions](https://mlsanity.com/paper/1909.09674/) (2019)
- [Learning Visually Guided Latent Actions for Assistive Teleoperation](https://mlsanity.com/paper/2105.00580/) (2021)
- [Rearrangement with Nonprehensile Manipulation Using Deep Reinforcement Learning](https://mlsanity.com/paper/1803.05752/) (2018)
- [Efficient Obstacle Rearrangement for Object Manipulation Tasks in Cluttered Environments](https://mlsanity.com/paper/1902.06907/) (2019)
- [Fast and resilient manipulation planning for target retrieval in clutter](https://mlsanity.com/paper/2003.11420/) (2020)
- [Non-Prehensile Manipulation in Clutter with Human-In-The-Loop](https://mlsanity.com/paper/1904.03748/) (2019)

## Built on by (most cited first, 2 of 2)

- [Planning for Manipulation among Movable Objects: Deciding Which Objects Go Where, in What Order, and How](https://mlsanity.com/paper/2303.13385/) (2023)
- [SPOTS: Stable Placement of Objects with Reasoning in Semi-Autonomous Teleoperation Systems](https://mlsanity.com/paper/2309.13937/) (2023)

## Abstract (arXiv)

In this paper, we present a semi-autonomous teleoperation framework for a pick-and-place task using an RGB-D sensor. In particular, we assume that the target object is located in a cluttered environment where both prehensile grasping and non-prehensile manipulation are combined for efficient teleoperation. A trajectory-based reinforcement learning is utilized for learning the non-prehensile manipulation to rearrange the objects for enabling direct grasping. From the depth image of the cluttered environment and the location of the goal object, the learned policy can provide multiple options of non-prehensile manipulation to the human operator. We carefully design a reward function for the rearranging task where the policy is trained in a simulational environment. Then, the trained policy is transferred to a real-world and evaluated in a number of real-world experiments with the varying number of objects where we show that the proposed method outperforms manual keyboard control in terms of the time duration for the grasping.