mlsanityData to 7 Oct 2026

Transformer-based deep imitation learning for dual-arm robot manipulation

arXiv 2108.00385 ↗First posted 1 Aug 2021cs.RO cs.AIcited by 31

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Deep imitation learning is promising for solving dexterous manipulation tasks because it does not require an environment model and pre-programmed robot behavior. However, its application to dual-arm manipulation tasks remains challenging. In a dual-arm manipulation setup, the increased number of state dimensions caused by the additional robot manipulators causes distractions and results in poor performance of the neural networks. We address this issue using a self-attention mechanism that computes dependencies between elements in a sequential input and focuses on important elements. A Transformer, a variant of self-attention architecture, is applied to deep imitation learning to solve dual-arm manipulation tasks in the real world. The proposed method has been tested on dual-arm manipulation tasks using a real robot. The experimental results demonstrated that the Transformer-based deep imitation learning architecture can attend to the important features among the sensory inputs, therefore reducing distractions and improving manipulation performance when compared with the baseline architecture without the self-attention mechanisms. Data from this and related works are available at: https://sites.google.com/view/multi-task-fine.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 25 references that are in the map. First posted on arXiv 1 Aug 2021.

Builds on

14 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Deep Imitation Learning for Bimanual Robotic Manipulation11 Oct

Other references · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  8. 2020Quantifying Attention Flow in Transformerscited by 759
  9. 2013Network In Networkcited by 600
  10. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  11. 2017One Model To Learn Them Allcited by 86
  12. 2020Transformers for One-Shot Visual Imitationcited by 77
1 more
  1. 2021Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulationcited by 16

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 7; 2024: 10; 2025: 9.

31 papers.

Sort:

By area:

  1. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  2. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  3. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  4. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  5. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  6. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  7. 2022Goal-conditioned dual-action imitation learning for dexterous dual-arm robot manipulationcited by 21
  8. 2024ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projectioncited by 16
  9. 2025From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robotscited by 16
  10. 2024ILBiT: Imitation Learning for Robot Using Position and Torque Information based on Bilateral Control with Transformercited by 10
  11. 2022Memory-based gaze prediction in deep imitation learning for robot manipulationcited by 9
  12. 2022Training Robots without Robots: Deep Imitation Learning for Master-to-Robot Policy Transfercited by 9
19 more
  1. 2023PLEX: Making the Most of the Available Data for Robotic Manipulation Pretrainingcited by 9
  2. 2024APEX: Ambidextrous Dual-Arm Robotic Manipulation Using Collision-Free Generative Diffusion Modelscited by 7
  3. 2025One-Shot Dual-Arm Imitation Learningcited by 7
  4. 2024Multi-task real-robot data with gaze attention for dual-arm fine manipulationcited by 5
  5. 2025Distracted Robot: How Visual Clutter Undermine Robotic Manipulationcited by 5
  6. 2023Data-Driven Stochastic Motion Evaluation and Optimization with Image by Spatially-Aligned Temporal Encodingcited by 3
  7. 2023R2-Diff: Denoising by diffusion as a refinement of retrieved motion for image-based motion predictioncited by 3
  8. 2024Optimizing Active Perception for Learning Simultaneous Viewpoint Selection and Manipulation with Diffusion Policycited by 3
  9. 2025Improving Robotic Manipulation Robustness via NICE Scene Surgerycited by 3
  10. 2025VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulationcited by 2
  11. 2025M4Diffuser: Multi-View Diffusion Policy with Manipulability-Aware Control for Robust Mobile Manipulationcited by 2
  12. 2025MILE: A Mechanically Isomorphic Hand Exoskeleton and Visuotactile Robotic Hand for Data Collection in Dexterous Manipulationcited by 2
  13. 2023TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inferencecited by 1
  14. 2024Transformer-XL for Long Sequence Tasks in Robotic Learning from Demonstrationcited by 1
  15. 2024Bridging the Resource Gap: Deploying Advanced Imitation Learning Models onto Affordable Embedded Platformscited by 1
  16. 2025FABG : End-to-end Imitation Learning for Embodied Affective Human-Robot Interactioncited by 1
  17. 2023On Combining Expert Demonstrations in Imitation Learning via Optimal Transportcited by 0
  18. 2023Knolling Bot: Teaching Robots the Human Notion of Tidinesscited by 0
  19. 2025Variational Adaptive Noise and Dropout towards Stable Recurrent Neural Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Imitating Task and Motion Planning with Visuomotor Transformers
  2. 2022Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations
  3. 2025Dexterous Manipulation through Imitation Learning: A Survey
  4. 2025MTDP: A Modulated Transformer based Diffusion Policy Model
  5. 2025ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation
  6. 2025D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentation
  7. 2024Bi-ACT: Bilateral Control-Based Imitation Learning via Action Chunking with Transformer
  8. 2024Leveraging Pretrained Latent Representations for Few-Shot Imitation Learning on a Dexterous Robotic Hand