mlsanityData to 7 Oct 2026

Behavior Generation with Latent Actions

arXiv 2403.03181 ↗First posted 5 Mar 2024cs.LG cs.AI cs.ROcited by 159

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Generative modeling of complex behaviors from labeled datasets has been a longstanding problem in decision making. Unlike language or image generation, decision making requires modeling actions - continuous-valued vectors that are multimodal in their distribution, potentially drawn from uncurated sources, where generation errors can compound in sequential prediction. A recent class of models called Behavior Transformers (BeT) addresses this by discretizing actions using k-means clustering to capture different modes. However, k-means struggles to scale for high-dimensional action spaces or long sequences, and lacks gradient information, and thus BeT suffers in modeling long-range actions. In this work, we present Vector-Quantized Behavior Transformer (VQ-BeT), a versatile model for behavior generation that handles multimodal action prediction, conditional generation, and partial observations. VQ-BeT augments BeT by tokenizing continuous actions with a hierarchical vector quantization module. Across seven environments including simulated manipulation, autonomous driving, and robotics, VQ-BeT improves on state-of-the-art models such as BeT and Diffusion Policies. Importantly, we demonstrate VQ-BeT's improved ability to capture behavior modes while accelerating inference speed 5x over Diffusion Policies. Videos and code can be found https://sjlee.cc/vq-bet

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 48 of its 61 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

48 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play7 Dec
  2. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  4. 2023Imitating Human Behaviour with Diffusion Models25 Jan

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2016OpenAI Gymcited by 1,684
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  11. 2022Planning-oriented Autonomous Drivingcited by 1,068
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
32 more
  1. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  2. 2020Jukebox: A Generative Model for Musiccited by 535
  3. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  4. 2021Implicit Behavioral Cloningcited by 442
  5. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  6. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  7. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  8. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  9. 2019Learning Latent Plans from Playcited by 310
  10. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  11. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  12. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  13. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  14. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  15. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  16. 2023A Language Agent for Autonomous Drivingcited by 161
  17. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  18. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  19. 2023On Bringing Robots Homecited by 101
  20. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  21. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 90
  22. 2022Differentiable Raycasting for Self-supervised Occupancy Forecastingcited by 83
  23. 2015Deep Reinforcement Learning in Parameterized Action Spacecited by 65
  24. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 50
  25. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  26. 2021The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environmentscited by 39
  27. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
  28. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  29. 2020Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Drivingcited by 24
  30. 2021Continuous Control with Action Quantization from Demonstrationscited by 21
  31. 2022Automating Reinforcement Learning with Example-based Resetscited by 11
  32. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 58; 2026: 72.

159 papers.

Sort:

By area:

  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  3. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  4. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  5. 2024Latent Action Pretraining from Videoscited by 302
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  7. 2024Diffusion Policy Policy Optimizationcited by 255
  8. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
  9. 2025Real-Time Execution of Action Chunking Flow Policiescited by 199
  10. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  11. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 129
  12. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
38 more
  1. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  2. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  3. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  4. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  5. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 67
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  7. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  8. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  9. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 55
  10. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  11. 2024Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Samplingcited by 46
  12. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  13. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  14. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  15. 2025HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policycited by 42
  16. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 36
  17. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  18. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  19. 2024Inference-Time Policy Steering through Human Interactionscited by 35
  20. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulationcited by 35
  21. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34
  22. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  23. 2024CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Predictioncited by 26
  24. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 25
  25. 2025BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learningcited by 25
  26. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 25
  27. 2025GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Modelcited by 24
  28. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  29. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 22
  30. 2024TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learningcited by 21
  31. 2026ActionCodec: What Makes for Good Action Tokenizerscited by 17
  32. 2026G0.5: One Autoregressive Stream for Robot Reasoning and Actioncited by 17
  33. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  34. 2024P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policiescited by 16
  35. 2025Curating Demonstrations using Online Experiencecited by 16
  36. 2025Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learningcited by 16
  37. 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representationscited by 15
  38. 2025Generative Models in Decision Making: A Surveycited by 14

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusion
  2. 2025Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstraction
  3. 2026Large Discrete Policy: Advancing Explicit Behavior Modeling with Stochastic Iterative Scoring
  4. 2025HieroAction: Hierarchically Guided VLM for Fine-Grained Action Analysis
  5. 2023Hierarchical Imitation Learning with Vector Quantized Models
  6. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model
  7. 2025Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training
  8. 2026Learning Latent Action World Models In The Wild