# Goal-conditioned dual-action imitation learning for dexterous dual-arm robot manipulation

arXiv: https://arxiv.org/abs/2203.09749 · HTML: https://mlsanity.com/paper/2203.09749/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 26 of its 74 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation](https://mlsanity.com/paper/1707.03374/) (2017)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Deep Spatial Autoencoders for Visuomotor Learning](https://mlsanity.com/paper/1509.06113/) (2015)
- [Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations](https://mlsanity.com/paper/2003.06085/) (2020)
- [AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos](https://mlsanity.com/paper/1912.04443/) (2019)
- [IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data](https://mlsanity.com/paper/1911.05321/) (2019)
- [Batch Informed Trees (BIT*): Sampling-based Optimal Planning via the Heuristically Guided Search of Implicit Random Geometric Graphs](https://mlsanity.com/paper/1405.5848/) (2014)
- [Transformer-based deep imitation learning for dual-arm robot manipulation](https://mlsanity.com/paper/2108.00385/) (2021)
- [Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning](https://mlsanity.com/paper/2005.10872/) (2020)
- [Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments](https://mlsanity.com/paper/2010.11940/) (2020)
- [Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation](https://mlsanity.com/paper/2102.01295/) (2021)
- [Recurrent Mixture Density Network for Spatiotemporal Visual Attention](https://mlsanity.com/paper/1603.08199/) (2016)
- [Learning Periodic Tasks from Human Demonstrations](https://mlsanity.com/paper/2109.14078/) (2021)
- [Memory-based gaze prediction in deep imitation learning for robot manipulation](https://mlsanity.com/paper/2202.04877/) (2022)
- [Training Robots without Robots: Deep Imitation Learning for Master-to-Robot Policy Transfer](https://mlsanity.com/paper/2202.09574/) (2022)
- [Reactive Trajectory Generation in an Unknown Environment](https://mlsanity.com/paper/1703.00461/) (2017)

## Built on by (most cited first, 22 of 22)

- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [FlowVLA: Visual Chain of Thought-based Motion Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2508.18269/) (2025)
- [ForceMimic: Force-Centric Imitation Learning with Force-Motion Capture System for Contact-Rich Manipulation](https://mlsanity.com/paper/2410.07554/) (2024)
- [Dexterous Manipulation through Imitation Learning: A Survey](https://mlsanity.com/paper/2504.03515/) (2025)
- [Act2Goal: From World Model To General Goal-conditioned Policy](https://mlsanity.com/paper/2512.23541/) (2025)
- [f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergences](https://mlsanity.com/paper/2310.06794/) (2023)
- [DAG-Plan: Generating Directed Acyclic Dependency Graphs for Dual-Arm Cooperative Planning](https://mlsanity.com/paper/2406.09953/) (2024)
- [METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model](https://mlsanity.com/paper/2511.17366/) (2025)
- [Vegetable Peeling: A Case Study in Constrained Dexterous Manipulation](https://mlsanity.com/paper/2407.07884/) (2024)
- [Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers](https://mlsanity.com/paper/2503.13916/) (2025)
- [APEX: Ambidextrous Dual-Arm Robotic Manipulation Using Collision-Free Generative Diffusion Models](https://mlsanity.com/paper/2404.02284/) (2024)
- [Lemmanaid: Neuro-Symbolic Lemma Conjecturing](https://mlsanity.com/paper/2504.04942/) (2025)
- [UniConFlow: A Unified Constrained Flow-Matching Framework for Certified Motion Planning](https://mlsanity.com/paper/2506.02955/) (2025)
- [Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulation](https://mlsanity.com/paper/2501.15071/) (2025)
- [Enhancing Reusability of Learned Skills for Robot Manipulation via Gaze Information and Motion Bottlenecks](https://mlsanity.com/paper/2502.18121/) (2025)
- [Learning Dual-Arm Push and Grasp Synergy in Dense Clutter](https://mlsanity.com/paper/2412.04052/) (2024)
- [ChicGrasp: Imitation-Learning based Customized Dual-Jaw Gripper Control for Delicate, Irregular Bio-products Manipulation](https://mlsanity.com/paper/2505.08986/) (2025)
- [Exploration-assisted Bottleneck Transition Toward Robust and Data-efficient Deformable Object Manipulation](https://mlsanity.com/paper/2603.13756/) (2026)
- [Learning Dual-Arm Coordination for Grasping Large Flat Objects](https://mlsanity.com/paper/2504.03500/) (2025)
- [Imitation Learning for Active Neck Motion Enabling Robot Manipulation beyond the Field of View](https://mlsanity.com/paper/2506.17624/) (2025)
- [Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactions](https://mlsanity.com/paper/2602.07341/) (2026)

## Abstract (arXiv)

Long-horizon dexterous robot manipulation of deformable objects, such as banana peeling, is a problematic task because of the difficulties in object modeling and a lack of knowledge about stable and dexterous manipulation skills. This paper presents a goal-conditioned dual-action (GC-DA) deep imitation learning (DIL) approach that can learn dexterous manipulation skills using human demonstration data. Previous DIL methods map the current sensory input and reactive action, which often fails because of compounding errors in imitation learning caused by the recurrent computation of actions. The method predicts reactive action only when the precise manipulation of the target object is required (local action) and generates the entire trajectory when precise manipulation is not required (global action). This dual-action formulation effectively prevents compounding error in the imitation learning using the trajectory-based global action while responding to unexpected changes in the target object during the reactive local action. The proposed method was tested in a real dual-arm robot and successfully accomplished the banana-peeling task. Data from this and related works are available at: https://sites.google.com/view/multi-task-fine.