mlsanityData to 7 Oct 2026

Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning

arXiv 2310.11731 ↗First posted 18 Oct 2023cs.AIcited by 34

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

The offline reinforcement learning (RL) paradigm provides a general recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. While policy constraints, conservatism, and other methods for mitigating distributional shifts have made offline reinforcement learning more effective, the continuous action setting often necessitates various approximations for applying these techniques. Many of these challenges are greatly alleviated in discrete action settings, where offline RL constraints and regularizers can often be computed more precisely or even exactly. In this paper, we propose an adaptive scheme for action quantization. We use a VQ-VAE to learn state-conditioned action quantization, avoiding the exponential blowup that comes with na"ive discretization of the action space. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme. We further validate our approach on a set of challenging long-horizon complex robotic manipulation tasks in the Robomimic environment, where our discretized offline RL algorithms are able to improve upon their continuous counterparts by 2-3x. Our project page is at https://saqrl.github.io/

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 32 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

31 of 32 references.

References · 31

  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
19 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  2. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  3. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  5. 2020Critic Regularized Regressioncited by 269
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  9. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  10. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  11. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 90
  12. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 65
  13. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 50
  14. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  15. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 22
  16. 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 21
  17. 2021Continuous Control with Action Quantization from Demonstrationscited by 21
  18. 2022Solving Continuous Control via Q-learningcited by 21
  19. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 12; 2026: 10.

34 papers.

Sort:

By area:

  1. 2024Behavior Generation with Latent Actionscited by 159
  2. 2024PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Controlcited by 21
  3. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  4. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  5. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  6. 2025STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantizationcited by 13
  7. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  8. 2023H-GAP: Humanoid Control with a Generalist Plannercited by 11
  9. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  10. 2025OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulationcited by 7
  11. 2024Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusioncited by 6
  12. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actionscited by 5
22 more
  1. 2025Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstractioncited by 4
  2. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
  3. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  4. 2025Behaviour Discovery and Attribution for Explainable Reinforcement Learningcited by 2
  5. 2025DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Rewardcited by 2
  6. 2025Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantizationcited by 2
  7. 2026Embodied Task Planning via Graph-Informed Action Generation with Large Language Modelscited by 2
  8. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithmscited by 1
  9. 2024Rethinking the Intermediate Features in Adversarial Attacks: Misleading Robotic Models via Adversarial Distillationcited by 1
  10. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learningcited by 1
  11. 2025Offline Reinforcement Learning with Discrete Diffusion Skillscited by 1
  12. 2025HiBerNAC: Hierarchical Brain-inspired Robotic Neural Agent Collective for Disentangling Complex Manipulationcited by 1
  13. 2026MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulationcited by 1
  14. 2025Mental Accounts for Actions: EWA-Inspired Attention in Decision Transformerscited by 0
  15. 2025D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Taskscited by 0
  16. 2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0
  17. 2026In-Context Planning with Latent Temporal Abstractionscited by 0
  18. 2026Temporal Action Representation Learning for Tactical Resource Control and Subsequent Maneuver Generationcited by 0
  19. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0
  20. 2026BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deploymentcited by 0
  21. 2026Probabilistic Recurrent Intention Switching Modelcited by 0
  22. 2026Learning Reliable Parking Policies via Offline Reinforcement Learning with Quantized Action Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
  2. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  3. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
  4. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
  5. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
  6. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models
  7. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
  8. 2021Continuous Doubly Constrained Batch Reinforcement Learning