# ILBiT: Imitation Learning for Robot Using Position and Torque Information based on Bilateral Control with Transformer arXiv: https://arxiv.org/abs/2401.16653 ยท HTML: https://mlsanity.com/paper/2401.16653/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. - Builds on 24 of its 51 references that are in the map. - First posted on arXiv 30 Jan 2024. ## Placed in - Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/) ## Builds on (most cited first, 24 of 24) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022) - [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019) - [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022) - [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018) - [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017) - [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018) - [Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation](https://mlsanity.com/paper/1707.03374/) (2017) - [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021) - [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021) - [Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency](https://mlsanity.com/paper/2012.09811/) (2020) - [Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning](https://mlsanity.com/paper/2204.02863/) (2022) - [Learning One-Shot Imitation from Humans without Humans](https://mlsanity.com/paper/1911.01103/) (2019) - [Transformer-based deep imitation learning for dual-arm robot manipulation](https://mlsanity.com/paper/2108.00385/) (2021) - [Imitation Learning for Object Manipulation Based on Position/Force Information Using Bilateral Control](https://mlsanity.com/paper/1811.03759/) (2018) - [Imitation Learning for Variable Speed Contact Motion for Operation up to Control Bandwidth](https://mlsanity.com/paper/2102.10283/) (2021) - [Soft and Rigid Object Grasping With Cross-Structure Hand Using Bilateral Control-Based Imitation Learning](https://mlsanity.com/paper/2311.09555/) (2023) - [An Independently Learnable Hierarchical Model for Bilateral Control-Based Imitation Learning Applications](https://mlsanity.com/paper/2203.08619/) (2022) - [Imitation Learning for Nonprehensile Manipulation through Self-Supervised Learning Considering Motion Speed](https://mlsanity.com/paper/2206.10289/) (2022) - [Motion Generation Using Bilateral Control-Based Imitation Learning with Autoregressive Learning](https://mlsanity.com/paper/2011.06192/) (2020) - [Time Series Motion Generation Considering Long Short-Term Motion](https://mlsanity.com/paper/1909.03755/) (2019) - [Geometric Reinforcement Learning For Robotic Manipulation](https://mlsanity.com/paper/2210.08126/) (2022) - [A Framework for Few-Shot Policy Transfer through Observation Mapping and Behavior Cloning](https://mlsanity.com/paper/2310.08836/) (2023) ## Built on by (most cited first, 10 of 10) - [Bi-ACT: Bilateral Control-Based Imitation Learning via Action Chunking with Transformer](https://mlsanity.com/paper/2401.17698/) (2024) - [Mamba as a motion encoder for robotic imitation learning](https://mlsanity.com/paper/2409.02636/) (2024) - [DABI: Evaluation of Data Augmentation Methods Using Downsampling in Bilateral Control-Based Imitation Learning with Images](https://mlsanity.com/paper/2410.04370/) (2024) - [ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems](https://mlsanity.com/paper/2510.07151/) (2025) - [Society of Mind Meets Real-Time Strategy: A Hierarchical Multi-Agent Framework for Strategic Reasoning](https://mlsanity.com/paper/2508.06042/) (2025) - [Variable-Speed Teaching-Playback as Real-World Data Augmentation for Imitation Learning](https://mlsanity.com/paper/2412.03252/) (2024) - [Bi-HIL: Bilateral Control-Based Multimodal Hierarchical Imitation Learning via Subtask-Level Progress Rate and Keyframe Memory for Long-Horizon Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2603.13315/) (2026) - [NeuralActuator: Neural Actuation Modeling for Robot Dynamics and External Force Perception](https://mlsanity.com/paper/2607.11734/) (2026) - [Force Generative Imitation Learning: Bridging Position Trajectory and Force Commands through Control Technique](https://mlsanity.com/paper/2602.06620/) (2026) - [Learning human joint torques from pixels](https://mlsanity.com/paper/2608.09083/) (2026) ## Abstract (arXiv) Autonomous manipulation in robot arms is a complex and evolving field of study in robotics. This paper introduces an innovative approach to this challenge by focusing on imitation learning (IL). Unlike traditional imitation methods, our approach uses IL based on bilateral control, allowing for more precise and adaptable robot movements. The conventional IL based on bilateral control method have relied on Long Short-Term Memory (LSTM) networks. In this paper, we present the IL for robot using position and torque information based on Bilateral control with Transformer (ILBiT). This proposed method employs the Transformer model, known for its robust performance in handling diverse datasets and its capability to surpass LSTM's limitations, especially in tasks requiring detailed force adjustments. A standout feature of ILBiT is its high-frequency operation at 100 Hz, which significantly improves the system's adaptability and response to varying environments and objects of different hardness levels. The effectiveness of the Transformer-based ILBiT method can be seen through comprehensive real-world experiments.