# Imitation Learning for Nonprehensile Manipulation through Self-Supervised Learning Considering Motion Speed

arXiv: https://arxiv.org/abs/2206.10289 · HTML: https://mlsanity.com/paper/2206.10289/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 30 of its 61 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Deep Spatial Autoencoders for Visuomotor Learning](https://mlsanity.com/paper/1509.06113/) (2015)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration](https://mlsanity.com/paper/1707.02920/) (2017)
- [Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor](https://mlsanity.com/paper/1910.04854/) (2019)
- [Imitation Learning for Object Manipulation Based on Position/Force Information Using Bilateral Control](https://mlsanity.com/paper/1811.03759/) (2018)
- [Self-supervised Learning for Precise Pick-and-place without Object Model](https://mlsanity.com/paper/2006.08373/) (2020)
- [Imitation Learning for Variable Speed Contact Motion for Operation up to Control Bandwidth](https://mlsanity.com/paper/2102.10283/) (2021)
- [How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning](https://mlsanity.com/paper/2106.02445/) (2021)
- [Motion Generation Using Bilateral Control-Based Imitation Learning with Autoregressive Learning](https://mlsanity.com/paper/2011.06192/) (2020)
- [Rearrangement with Nonprehensile Manipulation Using Deep Reinforcement Learning](https://mlsanity.com/paper/1803.05752/) (2018)
- [Robot Gaining Accurate Pouring Skills through Self-Supervised Learning and Generalization](https://mlsanity.com/paper/2011.10150/) (2020)
- [Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods](https://mlsanity.com/paper/2105.12946/) (2021)
- [Probabilistic Learning of Torque Controllers from Kinematic and Force Constraints](https://mlsanity.com/paper/1712.07249/) (2017)
- [In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning](https://mlsanity.com/paper/2103.09402/) (2021)
- [Pixels to Plans: Learning Non-Prehensile Manipulation by Imitating a Planner](https://mlsanity.com/paper/1904.03260/) (2019)
- [Pushing Fast and Slow: Task-Adaptive Planning for Non-prehensile Manipulation Under Uncertainty](https://mlsanity.com/paper/1805.03005/) (2018)
- [Imitation learning for variable speed motion generation over multiple actions](https://mlsanity.com/paper/2103.06466/) (2021)
- [A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation](https://mlsanity.com/paper/2111.05318/) (2021)

## Built on by (most cited first, 14 of 14)

- [Bi-ACT: Bilateral Control-Based Imitation Learning via Action Chunking with Transformer](https://mlsanity.com/paper/2401.17698/) (2024)
- [SAIL: Faster-than-Demonstration Execution of Imitation Learning Policies](https://mlsanity.com/paper/2506.11948/) (2025)
- [Soft and Rigid Object Grasping With Cross-Structure Hand Using Bilateral Control-Based Imitation Learning](https://mlsanity.com/paper/2311.09555/) (2023)
- [Learning Variable Compliance Control From a Few Demonstrations for Bimanual Robot with Haptic Feedback Teleoperation System](https://mlsanity.com/paper/2406.14990/) (2024)
- [ILBiT: Imitation Learning for Robot Using Position and Torque Information based on Bilateral Control with Transformer](https://mlsanity.com/paper/2401.16653/) (2024)
- [Loss Function Considering Dead Zone for Neural Networks](https://mlsanity.com/paper/2402.00393/) (2024)
- [Variable-Frequency Imitation Learning for Variable-Speed Motion](https://mlsanity.com/paper/2411.12310/) (2024)
- [Variable-Speed Teaching-Playback as Real-World Data Augmentation for Imitation Learning](https://mlsanity.com/paper/2412.03252/) (2024)
- [Learning Multimodal Attention for Manipulating Deformable Objects with Changing States](https://mlsanity.com/paper/2309.14837/) (2023)
- [Imitation Learning Inputting Image Feature to Each Layer of Neural Network](https://mlsanity.com/paper/2401.09691/) (2024)
- [Error-Feedback Model for Output Correction in Bilateral Control-Based Imitation Learning](https://mlsanity.com/paper/2411.12255/) (2024)
- [Imitation Learning Based on Disentangled Representation Learning of Behavioral Characteristics](https://mlsanity.com/paper/2509.04737/) (2025)
- [Force Generative Imitation Learning: Bridging Position Trajectory and Force Commands through Control Technique](https://mlsanity.com/paper/2602.06620/) (2026)
- [VFILC: Accurate Frequency Extrapolations in Imitation Learning via Sampling Frequency ILC](https://mlsanity.com/paper/2606.20056/) (2026)

## Abstract (arXiv)

Robots are expected to replace menial tasks such as housework. Some of these tasks include nonprehensile manipulation performed without grasping objects. Nonprehensile manipulation is very difficult because it requires considering the dynamics of environments and objects. Therefore imitating complex behaviors requires a large number of human demonstrations. In this study, a self-supervised learning that considers dynamics to achieve variable speed for nonprehensile manipulation is proposed. The proposed method collects and fine-tunes only successful action data obtained during autonomous operations. By fine-tuning the successful data, the robot learns the dynamics among itself, its environment, and objects. We experimented with the task of scooping and transporting pancakes using the neural network model trained on 24 human-collected training data. The proposed method significantly improved the success rate from 40.2% to 85.7%, and succeeded the task more than 75% for other objects.