mlsanityData to 7 Oct 2026

Continuous Control with Action Quantization from Demonstrations

arXiv 2110.10149 ↗First posted 19 Oct 2021cs.LG cs.AI cs.ROcited by 21

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel Reinforcement Learning (RL) framework for problems with continuous action spaces: Action Quantization from Demonstrations (AQuaDem). The proposed approach consists in learning a discretization of continuous action spaces from human demonstrations. This discretization returns a set of plausible actions (in light of the demonstrations) for each input state, thus capturing the priors of the demonstrator and their multimodal behavior. By discretizing the action space, any discrete action deep RL technique can be readily applied to the continuous control problem. Experiments show that the proposed approach outperforms state-of-the-art methods such as SAC in the RL setup, and GAIL in the Imitation Learning setup. We provide a website with interactive videos: https://google-research.github.io/aquadem/ and make the code available: https://github.com/google-research/google-research/tree/master/aquadem.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 50 of its 91 references that are in the map. First posted on arXiv 19 Oct 2021.

Builds on

50 of 91 references.

References · 50

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
38 more
  1. 2015Prioritized Experience Replaycited by 1,070
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2018Learning Dexterous In-Hand Manipulationcited by 643
  8. 2021Implicit Behavioral Cloningcited by 442
  9. 2016Input Convex Neural Networkscited by 407
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  11. 2019Learning Latent Plans from Playcited by 310
  12. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  13. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  14. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  15. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  16. 2019Goal-conditioned Imitation Learningcited by 159
  17. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  18. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  19. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  20. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  21. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  22. 2020Primal Wasserstein Imitation Learningcited by 120
  23. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 90
  24. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  25. 2021What Matters for Adversarial Imitation Learning?cited by 70
  26. 2020Munchausen Reinforcement Learningcited by 68
  27. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 65
  28. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  29. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 50
  30. 2017DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrationscited by 43
  31. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimationcited by 42
  32. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  33. 2019CAQL: Continuous Action Q-Learningcited by 22
  34. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 22
  35. 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 21
  36. 2019Q-Learning for Continuous Actions with Cross-Entropy Guided Policiescited by 17
  37. 2021Hyperparameter Selection for Imitation Learningcited by 17
  38. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 9; 2025: 2; 2026: 1.

21 papers.

Sort:

By area:

  1. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
  2. 2024Behavior Generation with Latent Actionscited by 159
  3. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  4. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  5. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
  6. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  7. 2023A Survey of Demonstration Learningcited by 12
  8. 2023Efficient Planning with Latent Diffusioncited by 11
  9. 2023No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learningcited by 6
  10. 2024Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusioncited by 6
  11. 2024The Impact of Quantization and Pruning on Deep Reinforcement Learning Modelscited by 6
  12. 2023Investigating the role of model-based learning in exploration and transfercited by 5
9 more
  1. 2024VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Trainingcited by 5
  2. 2024Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learningcited by 4
  3. 2023Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learningcited by 3
  4. 2024CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interactioncited by 3
  5. 2023A Statistical Guarantee for Representation Transfer in Multitask Imitation Learningcited by 1
  6. 2024Automated Design and Optimization of Distributed Filtering Circuits via Reinforcement Learningcited by 1
  7. 2025Learning Quantized Continuous Controllers for Integer Hardwarecited by 1
  8. 2025Distributed Detection of Adversarial Attacks in Multi-Agent Reinforcement Learning with Continuous Action Spacecited by 0
  9. 2026Understanding Behavior Cloning with Action Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training
  2. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
  3. 2025DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
  4. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
  5. 2023Hierarchical Imitation Learning with Vector Quantized Models
  6. 2025Action Chunking and Exploratory Data Collection Yield Exponential Improvements in Behavior Cloning for Continuous Control
  7. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  8. 2025Boosting Action-Information via a Variational Bottleneck on Unlabelled Robot Videos