# Compositional Servoing by Recombining Demonstrations

arXiv: https://arxiv.org/abs/2310.04271 · HTML: https://mlsanity.com/paper/2310.04271/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [KNN-Diffusion: Image Generation via Large-Scale Retrieval](https://mlsanity.com/paper/2204.02849/) (2022)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations](https://mlsanity.com/paper/2003.06085/) (2020)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration](https://mlsanity.com/paper/2201.12716/) (2022)
- [Latent Plans for Task-Agnostic Offline Reinforcement Learning](https://mlsanity.com/paper/2209.08959/) (2022)
- [Neural Task Programming: Learning to Generalize Across Hierarchical Tasks](https://mlsanity.com/paper/1710.01813/) (2017)
- [Watch and Match: Supercharging Imitation with Regularized Optimal Transport](https://mlsanity.com/paper/2206.15469/) (2022)
- [Memory-Augmented Reinforcement Learning for Image-Goal Navigation](https://mlsanity.com/paper/2101.05181/) (2021)
- [Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration](https://mlsanity.com/paper/1807.03480/) (2018)
- [Interactive Imitation Learning in Robotics: A Survey](https://mlsanity.com/paper/2211.00600/) (2022)

## Abstract (arXiv)

Learning-based manipulation policies from image inputs often show weak task transfer capabilities. In contrast, visual servoing methods allow efficient task transfer in high-precision scenarios while requiring only a few demonstrations. In this work, we present a framework that formulates the visual servoing task as graph traversal. Our method not only extends the robustness of visual servoing, but also enables multitask capability based on a few task-specific demonstrations. We construct demonstration graphs by splitting existing demonstrations and recombining them. In order to traverse the demonstration graph in the inference case, we utilize a similarity function that helps select the best demonstration for a specific task. This enables us to compute the shortest path through the graph. Ultimately, we show that recombining demonstrations leads to higher task-respective success. We present extensive simulation and real-world experimental results that demonstrate the efficacy of our approach.