# Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations

arXiv: https://arxiv.org/abs/2408.04380 · HTML: https://mlsanity.com/paper/2408.04380/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 176 of its 230 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 176)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)

## Built on by (most cited first, 30 of 35)

- [Failure Prediction at Runtime for Generative Robot Policies](https://mlsanity.com/paper/2510.09459/) (2025)
- [Inference-Time Policy Steering through Human Interactions](https://mlsanity.com/paper/2411.16627/) (2024)
- [Diffusion Predictive Control with Constraints](https://mlsanity.com/paper/2412.09342/) (2024)
- [Motion Planning Diffusion: Learning and Adapting Robot Motion Planning with Diffusion Models](https://mlsanity.com/paper/2412.19948/) (2024)
- [Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards](https://mlsanity.com/paper/2410.23289/) (2024)
- [Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling](https://mlsanity.com/paper/2502.00622/) (2025)
- [Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction](https://mlsanity.com/paper/2508.05294/) (2025)
- [Fast and Robust Visuomotor Riemannian Flow Matching Policy](https://mlsanity.com/paper/2412.10855/) (2024)
- [CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansion](https://mlsanity.com/paper/2601.09512/) (2026)
- [FlowRAM: Grounding Flow Matching Policy with Region-Aware Mamba Framework for Robotic Manipulation](https://mlsanity.com/paper/2506.16201/) (2025)
- [TeLoGraF: Temporal Logic Planning via Graph-encoded Flow Matching](https://mlsanity.com/paper/2505.00562/) (2025)
- [Fast Flow-based Visuomotor Policies via Conditional Optimal Transport Couplings](https://mlsanity.com/paper/2505.01179/) (2025)
- [Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition](https://mlsanity.com/paper/2510.01068/) (2025)
- [XRoboToolkit: A Cross-Platform Framework for Robot Teleoperation](https://mlsanity.com/paper/2508.00097/) (2025)
- [VILP: Imitation Learning with Latent Video Planning](https://mlsanity.com/paper/2502.01784/) (2025)
- [CHD: Coupled Hierarchical Diffusion for Long-Horizon Tasks](https://mlsanity.com/paper/2505.07261/) (2025)
- [Grasp Diffusion Network: Learning Grasp Generators from Partial Point Clouds with Diffusion Models in SO(3)xR3](https://mlsanity.com/paper/2412.08398/) (2024)
- [On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lighting](https://mlsanity.com/paper/2504.13618/) (2025)
- [Interactive Imitation Learning for Dexterous Robotic Manipulation: Challenges and Perspectives -- A Survey](https://mlsanity.com/paper/2506.00098/) (2025)
- [From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedback](https://mlsanity.com/paper/2502.07645/) (2025)
- [EVE: A Generator-Verifier System for Generative Policies](https://mlsanity.com/paper/2512.21430/) (2025)
- [Discrete Contrastive Learning for Diffusion Policies in Autonomous Driving](https://mlsanity.com/paper/2503.05229/) (2025)
- [Human Centric General Physical Intelligence for Agile Manufacturing Automation](https://mlsanity.com/paper/2508.11960/) (2025)
- [ForeDiffusion: Foresight-Conditioned Diffusion Policy via Future View Construction for Robot Manipulation](https://mlsanity.com/paper/2601.12925/) (2026)
- [Encoding Predictability and Legibility for Style-Conditioned Diffusion Policy](https://mlsanity.com/paper/2603.16368/) (2026)
- [Modulating Reservoir Dynamics via Reinforcement Learning for Efficient Robot Skill Synthesis](https://mlsanity.com/paper/2411.10991/) (2024)
- [From Words to Workflows: Automating Business Processes](https://mlsanity.com/paper/2412.03446/) (2024)
- [Enhancing Path Planning Performance through Image Representation Learning of High-Dimensional Configuration Spaces](https://mlsanity.com/paper/2501.06639/) (2025)
- [Inverse RL Scene Dynamics Learning for Nonlinear Predictive Control in Autonomous Vehicles](https://mlsanity.com/paper/2504.01336/) (2025)
- [Analyzing the Impact of Multimodal Perception on Sample Complexity and Optimization Landscapes in Imitation Learning](https://mlsanity.com/paper/2508.05077/) (2025)

## Abstract (arXiv)

Learning from Demonstrations, the field that proposes to learn robot behavior models from data, is gaining popularity with the emergence of deep generative models. Although the problem has been studied for years under names such as Imitation Learning, Behavioral Cloning, or Inverse Reinforcement Learning, classical methods have relied on models that don't capture complex data distributions well or don't scale well to large numbers of demonstrations. In recent years, the robot learning community has shown increasing interest in using deep generative models to capture the complexity of large datasets. In this survey, we aim to provide a unified and comprehensive review of the last year's progress in the use of deep generative models in robotics. We present the different types of models that the community has explored, such as energy-based models, diffusion models, action value maps, or generative adversarial networks. We also present the different types of applications in which deep generative models have been used, from grasp generation to trajectory generation or cost learning. One of the most important elements of generative models is the generalization out of distributions. In our survey, we review the different decisions the community has made to improve the generalization of the learned models. Finally, we highlight the research challenges and propose a number of future directions for learning deep generative models in robotics.