# Learning from Demonstration with Weakly Supervised Disentanglement

arXiv: https://arxiv.org/abs/2006.09107 · HTML: https://mlsanity.com/paper/2006.09107/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 16 Jun 2020.

## Placed in

- Offline RL and imitation learning from data / [Robot learning from demonstration](https://mlsanity.com/offlinerl/robot-learning-from-demonstration/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [Semi-Supervised Learning with Ladder Networks](https://mlsanity.com/paper/1507.02672/) (2015)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [An Uncertain Future: Forecasting from Static Images using Variational Autoencoders](https://mlsanity.com/paper/1606.07873/) (2016)
- [Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objective](https://mlsanity.com/paper/1806.07772/) (2018)
- [Interpretable Latent Spaces for Learning from Demonstration](https://mlsanity.com/paper/1807.06583/) (2018)
- [Disentangled Relational Representations for Explaining and Learning from Demonstration](https://mlsanity.com/paper/1907.13627/) (2019)

## Built on by (most cited first, 8 of 8)

- [A Survey of Demonstration Learning](https://mlsanity.com/paper/2303.11191/) (2023)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [Out of Distribution Reasoning by Weakly-Supervised Disentangled Logic Variational Autoencoder](https://mlsanity.com/paper/2210.09959/) (2022)
- [Imitation Learning Based on Disentangled Representation Learning of Behavioral Characteristics](https://mlsanity.com/paper/2509.04737/) (2025)
- [Learning Time-Invariant Reward Functions through Model-Based Inverse Reinforcement Learning](https://mlsanity.com/paper/2107.03186/) (2021)
- [Perceive, Represent, Generate: Translating Multimodal Information to Robotic Motion Trajectories](https://mlsanity.com/paper/2204.03051/) (2022)
- [Learning Manner of Execution from Partial Corrections](https://mlsanity.com/paper/2302.03338/) (2023)
- [ConceptACT: Episode-Level Concepts for Sample-Efficient Robotic Imitation Learning](https://mlsanity.com/paper/2601.17135/) (2026)

## Abstract (arXiv)

Robotic manipulation tasks, such as wiping with a soft sponge, require control from multiple rich sensory modalities. Human-robot interaction, aimed at teaching robots, is difficult in this setting as there is potential for mismatch between human and machine comprehension of the rich data streams. We treat the task of interpretable learning from demonstration as an optimisation problem over a probabilistic generative model. To account for the high-dimensionality of the data, a high-capacity neural network is chosen to represent the model. The latent variables in this model are explicitly aligned with high-level notions and concepts that are manifested in a set of demonstrations. We show that such alignment is best achieved through the use of labels from the end user, in an appropriately restricted vocabulary, in contrast to the conventional approach of the designer picking a prior over the latent variables. Our approach is evaluated in the context of two table-top robot manipulation tasks performed by a PR2 robot -- that of dabbing liquids with a sponge (forcefully pressing a sponge and moving it along a surface) and pouring between different containers. The robot provides visual information, arm joint positions and arm joint efforts. We have made videos of the tasks and data available - see supplementary materials at: https://sites.google.com/view/weak-label-lfd.