# Transformer-based deep imitation learning for dual-arm robot manipulation arXiv: https://arxiv.org/abs/2108.00385 ยท HTML: https://mlsanity.com/paper/2108.00385/ ## Key facts - Cited by 32 papers in the map (data through 14 Oct 2026). - Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. - Builds on 14 of its 25 references that are in the map. - First posted on arXiv 1 Aug 2021. ## Placed in - Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/) ## Builds on (most cited first, 14 of 14) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020) - [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020) - [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020) - [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019) - [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013) - [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017) - [One Model To Learn Them All](https://mlsanity.com/paper/1706.05137/) (2017) - [Transformers for One-Shot Visual Imitation](https://mlsanity.com/paper/2011.05970/) (2020) - [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020) - [Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation](https://mlsanity.com/paper/2102.01295/) (2021) ## Built on by (most cited first, 30 of 32) - [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022) - [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023) - [RVT-2: Learning Precise Manipulation from Few Demonstrations](https://mlsanity.com/paper/2406.08545/) (2024) - [Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation](https://mlsanity.com/paper/2410.08001/) (2024) - [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022) - [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024) - [Goal-conditioned dual-action imitation learning for dexterous dual-arm robot manipulation](https://mlsanity.com/paper/2203.09749/) (2022) - [ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection](https://mlsanity.com/paper/2405.03666/) (2024) - [From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2506.12779/) (2025) - [ILBiT: Imitation Learning for Robot Using Position and Torque Information based on Bilateral Control with Transformer](https://mlsanity.com/paper/2401.16653/) (2024) - [Memory-based gaze prediction in deep imitation learning for robot manipulation](https://mlsanity.com/paper/2202.04877/) (2022) - [Training Robots without Robots: Deep Imitation Learning for Master-to-Robot Policy Transfer](https://mlsanity.com/paper/2202.09574/) (2022) - [PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining](https://mlsanity.com/paper/2303.08789/) (2023) - [One-Shot Dual-Arm Imitation Learning](https://mlsanity.com/paper/2503.06831/) (2025) - [APEX: Ambidextrous Dual-Arm Robotic Manipulation Using Collision-Free Generative Diffusion Models](https://mlsanity.com/paper/2404.02284/) (2024) - [Multi-task real-robot data with gaze attention for dual-arm fine manipulation](https://mlsanity.com/paper/2401.07603/) (2024) - [Distracted Robot: How Visual Clutter Undermine Robotic Manipulation](https://mlsanity.com/paper/2511.22780/) (2025) - [Data-Driven Stochastic Motion Evaluation and Optimization with Image by Spatially-Aligned Temporal Encoding](https://mlsanity.com/paper/2302.05041/) (2023) - [R2-Diff: Denoising by diffusion as a refinement of retrieved motion for image-based motion prediction](https://mlsanity.com/paper/2306.09483/) (2023) - [Optimizing Active Perception for Learning Simultaneous Viewpoint Selection and Manipulation with Diffusion Policy](https://mlsanity.com/paper/2409.14615/) (2024) - [DA-VIL: Adaptive Dual-Arm Manipulation with Reinforcement Learning and Variable Impedance Control](https://mlsanity.com/paper/2410.19712/) (2024) - [VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation](https://mlsanity.com/paper/2501.03606/) (2025) - [Improving Robotic Manipulation Robustness via NICE Scene Surgery](https://mlsanity.com/paper/2511.22777/) (2025) - [M4Diffuser: Multi-View Diffusion Policy with Manipulability-Aware Control for Robust Mobile Manipulation](https://mlsanity.com/paper/2509.14980/) (2025) - [MILE: A Mechanically Isomorphic Hand Exoskeleton and Visuotactile Robotic Hand for Data Collection in Dexterous Manipulation](https://mlsanity.com/paper/2512.00324/) (2025) - [TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference](https://mlsanity.com/paper/2303.14882/) (2023) - [On Combining Expert Demonstrations in Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2307.10810/) (2023) - [Transformer-XL for Long Sequence Tasks in Robotic Learning from Demonstration](https://mlsanity.com/paper/2405.15562/) (2024) - [Bridging the Resource Gap: Deploying Advanced Imitation Learning Models onto Affordable Embedded Platforms](https://mlsanity.com/paper/2411.11406/) (2024) - [FABG : End-to-end Imitation Learning for Embodied Affective Human-Robot Interaction](https://mlsanity.com/paper/2503.01363/) (2025) ## Abstract (arXiv) Deep imitation learning is promising for solving dexterous manipulation tasks because it does not require an environment model and pre-programmed robot behavior. However, its application to dual-arm manipulation tasks remains challenging. In a dual-arm manipulation setup, the increased number of state dimensions caused by the additional robot manipulators causes distractions and results in poor performance of the neural networks. We address this issue using a self-attention mechanism that computes dependencies between elements in a sequential input and focuses on important elements. A Transformer, a variant of self-attention architecture, is applied to deep imitation learning to solve dual-arm manipulation tasks in the real world. The proposed method has been tested on dual-arm manipulation tasks using a real robot. The experimental results demonstrated that the Transformer-based deep imitation learning architecture can attend to the important features among the sensory inputs, therefore reducing distractions and improving manipulation performance when compared with the baseline architecture without the self-attention mechanisms. Data from this and related works are available at: https://sites.google.com/view/multi-task-fine.