# Learning a Multi-Modal Policy via Imitating Demonstrations with Mixed Behaviors

arXiv: https://arxiv.org/abs/1903.10304 · HTML: https://mlsanity.com/paper/1903.10304/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2019.

## Built on by (most cited first, 9 of 9)

- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [Bayesian Disturbance Injection: Robust Imitation Learning of Flexible Policies for Robot Manipulation](https://mlsanity.com/paper/2211.03393/) (2022)
- [Deep Adaptive Multi-Intention Inverse Reinforcement Learning](https://mlsanity.com/paper/2107.06692/) (2021)
- [FedEmbed: Personalized Private Federated Learning](https://mlsanity.com/paper/2202.09472/) (2022)
- [Towards Balancing Preference and Performance through Adaptive Personalized Explainability](https://mlsanity.com/paper/2504.13856/) (2025)
- [VGC-Bench: Towards Mastering Diverse Team Strategies in Competitive Pok'emon](https://mlsanity.com/paper/2506.10326/) (2025)
- [Generalization Guarantees for Imitation Learning](https://mlsanity.com/paper/2008.01913/) (2020)
- [FedPC: Federated Learning for Language Generation with Personal and Context Preference Embeddings](https://mlsanity.com/paper/2210.03766/) (2022)
- [Quantifying Potential Observation Missingness in Inverse Reinforcement Learning](https://mlsanity.com/paper/2605.12831/) (2026)

## Abstract (arXiv)

We propose a novel approach to train a multi-modal policy from mixed demonstrations without their behavior labels. We develop a method to discover the latent factors of variation in the demonstrations. Specifically, our method is based on the variational autoencoder with a categorical latent variable. The encoder infers discrete latent factors corresponding to different behaviors from demonstrations. The decoder, as a policy, performs the behaviors accordingly. Once learned, the policy is able to reproduce a specific behavior by simply conditioning on a categorical vector. We evaluate our method on three different tasks, including a challenging task with high-dimensional visual inputs. Experimental results show that our approach is better than various baseline methods and competitive with a multi-modal policy trained by ground truth behavior labels.