# Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization

arXiv: https://arxiv.org/abs/2111.07999 · HTML: https://mlsanity.com/paper/2111.07999/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Deep Spatial Autoencoders for Visuomotor Learning](https://mlsanity.com/paper/1509.06113/) (2015)
- [Reinforcement and Imitation Learning for Diverse Visuomotor Skills](https://mlsanity.com/paper/1802.09564/) (2018)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies](https://mlsanity.com/paper/1905.09808/) (2019)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks](https://mlsanity.com/paper/1911.07246/) (2019)
- [Divide-and-Conquer Reinforcement Learning](https://mlsanity.com/paper/1711.09874/) (2017)
- [Task-Relevant Adversarial Imitation Learning](https://mlsanity.com/paper/1910.01077/) (2019)
- [Hierarchical visuomotor control of humanoids](https://mlsanity.com/paper/1811.09656/) (2018)
- [Transferable Task Execution from Pixels through Deep Planning Domain Learning](https://mlsanity.com/paper/2003.03726/) (2020)
- [Learning Task Decomposition with Ordered Memory Policy Network](https://mlsanity.com/paper/2103.10972/) (2021)
- [A Framework for Fine Robotic Assembly](https://mlsanity.com/paper/1509.04806/) (2015)

## Built on by (most cited first, 30 of 46)

- [Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models](https://mlsanity.com/paper/2401.03360/) (2023)
- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)
- [TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization](https://mlsanity.com/paper/2503.19901/) (2025)
- [Skill-based Model-based Reinforcement Learning](https://mlsanity.com/paper/2207.07560/) (2022)
- [Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation](https://mlsanity.com/paper/2309.00987/) (2023)
- [Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulation](https://mlsanity.com/paper/2508.19958/) (2025)
- [Multi-skill Mobile Manipulation for Object Rearrangement](https://mlsanity.com/paper/2209.02778/) (2022)
- [Towards Generalist Robot Learning from Internet Video: A Survey](https://mlsanity.com/paper/2404.19664/) (2024)
- [Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotion](https://mlsanity.com/paper/2303.11330/) (2023)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks](https://mlsanity.com/paper/2412.13211/) (2024)
- [Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning](https://mlsanity.com/paper/2203.13733/) (2022)
- [Strategy and Skill Learning for Physics-based Table Tennis Animation](https://mlsanity.com/paper/2407.16210/) (2024)
- [Skill Transformer: A Monolithic Policy for Mobile Manipulation](https://mlsanity.com/paper/2308.09873/) (2023)
- [Bi-Manual Manipulation and Attachment via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2203.08277/) (2022)
- [SEABO: A Simple Search-Based Method for Offline Imitation Learning](https://mlsanity.com/paper/2402.03807/) (2024)
- [A Survey on Imitation Learning for Contact-Rich Tasks in Robotics](https://mlsanity.com/paper/2506.13498/) (2025)
- [DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks](https://mlsanity.com/paper/2404.16779/) (2024)
- [MTIL: Encoding Full History with Mamba for Temporal Imitation Learning](https://mlsanity.com/paper/2505.12410/) (2025)
- [Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robot](https://mlsanity.com/paper/2307.16503/) (2023)
- [Generalize by Touching: Tactile Ensemble Skill Transfer for Robotic Furniture Assembly](https://mlsanity.com/paper/2404.17684/) (2024)
- [BOSS: Benchmark for Observation Space Shift in Long-Horizon Task](https://mlsanity.com/paper/2502.15679/) (2025)
- [MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policies](https://mlsanity.com/paper/2509.14159/) (2025)
- [SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration](https://mlsanity.com/paper/2211.13743/) (2022)
- [Learning to Ball: Composing Policies for Long-Horizon Basketball Moves](https://mlsanity.com/paper/2509.22442/) (2025)
- [Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memory](https://mlsanity.com/paper/2608.16889/) (2026)
- [Developing cooperative policies for multi-stage reinforcement learning tasks](https://mlsanity.com/paper/2205.05230/) (2022)
- [Robust Subtask Learning for Compositional Generalization](https://mlsanity.com/paper/2302.02984/) (2023)
- [Bi-Manual Block Assembly via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2303.14870/) (2023)
- [Multi-objective Cross-task Learning via Goal-conditioned GPT-based Decision Transformers for Surgical Robot Task Automation](https://mlsanity.com/paper/2405.18757/) (2024)

## Abstract (arXiv)

Skill chaining is a promising approach for synthesizing complex behaviors by sequentially combining previously learned skills. Yet, a naive composition of skills fails when a policy encounters a starting state never seen during its training. For successful skill chaining, prior approaches attempt to widen the policy's starting state distribution. However, these approaches require larger state distributions to be covered as more policies are sequenced, and thus are limited to short skill sequences. In this paper, we propose to chain multiple policies without excessively large initial state distributions by regularizing the terminal state distributions in an adversarial learning framework. We evaluate our approach on two complex long-horizon manipulation tasks of furniture assembly. Our results have shown that our method establishes the first model-free reinforcement learning algorithm to solve these tasks; whereas prior skill chaining approaches fail. The code and videos are available at https://clvrai.com/skill-chaining