mlsanityData to 14 Oct 2026

Few-Shot Task Learning through Inverse Generative Modeling

arXiv 2411.04987 ↗First posted 7 Nov 2024cs.AI cs.LG cs.ROcited by 4

Not on the map. Why?

Abstract (arXiv)

Learning the intents of an agent, defined by its goals or motion style, is often extremely challenging from just a few examples. We refer to this problem as task concept learning and present our approach, Few-Shot Task Learning through Inverse Generative Modeling (FTL-IGM), which learns new task concepts by leveraging invertible neural generative models. The core idea is to pretrain a generative model on a set of basic concepts and their demonstrations. Then, given a few demonstrations of a new concept (such as a new goal or a new action), our method learns the underlying concepts through backpropagation without updating the model weights, thanks to the invertibility of the generative model. We evaluate our method in five domains -- object rearrangement, goal-oriented navigation, motion caption of human actions, autonomous driving, and real-world table-top manipulation. Our experimental results demonstrate that via the pretrained generative model, we successfully learn novel concepts and generate agent plans or motion corresponding to these concepts in (1) unseen environments and (2) in composition with training concepts.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 69 of its 87 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

69 of 87 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  10. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  11. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2017Decoupled Weight Decay Regularizationcited by 1,831
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  7. 2022Human Motion Diffusion Modelcited by 1,080
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  10. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  11. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  13. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  14. 2021Implicit Behavioral Cloningcited by 452
  15. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  16. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  17. 2022Kubric: A scalable dataset generatorcited by 381
  18. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  19. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  20. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  21. 2017One-Shot Imitation Learningcited by 233
  22. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  23. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 220
  24. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  25. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 213
  26. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  27. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  28. 2019Goal-conditioned Imitation Learningcited by 162
  29. 2019Planning with Goal-Conditioned Policiescited by 149
  30. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  31. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  32. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 116
  33. 2022Learning from Few Examples: A Summary of Approaches to Few-Shot Learningcited by 86
  34. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  35. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  36. 2021Learning to Compose Visual Relationscited by 72
  37. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  38. 2020Online Bayesian Goal Inference for Boundedly-Rational Planning Agentscited by 68

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  2. 2025Compositional Scene Understanding through Inverse Generative Modelingcited by 3
  3. 2026Unsupervised Decomposition and Recombination with Discriminator-Driven Diffusion Modelscited by 3
  4. 2025Neural Force Field: Few-shot Learning of Generalized Physical Reasoningcited by 1