mlsanityData to 7 Oct 2026

Discretizing Continuous Action Space for On-Policy Optimization

arXiv 1901.10500 ↗First posted 29 Jan 2019cs.LG cs.AIcited by 65

Not on the map. Why?

Abstract (arXiv)

In this work, we show that discretizing action space for continuous control is a simple yet powerful technique for on-policy optimization. The explosion in the number of discrete actions can be efficiently addressed by a policy with factorized distribution across action dimensions. We show that the discrete policy achieves significant performance gains with state-of-the-art on-policy optimization algorithms (PPO, TRPO, ACKTR) especially on high-dimensional tasks with complex dynamics. Additionally, we show that an ordinal parameterization of the discrete distribution can introduce the inductive bias that encodes the natural ordering between discrete actions. This ordinal architecture further significantly improves the performance of PPO/TRPO.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 7; 2023: 16; 2024: 15; 2025: 11; 2026: 5.

65 papers.

Sort:

By area:

  1. 2021Learning and Planning in Complex Action Spacescited by 76
  2. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 37
  3. 2023Neural Categorical Priors for Physics-Based Character Controlcited by 37
  4. 2021Learning Large Neighborhood Search Policy for Integer Programmingcited by 35
  5. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
  6. 2021Neural Algorithmic Reasoners are Implicit Plannerscited by 21
  7. 2021Continuous Control with Action Quantization from Demonstrationscited by 21
  8. 2022Solving Continuous Control via Q-learningcited by 21
  9. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  10. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  11. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  12. 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
38 more
  1. 2023H-GAP: Humanoid Control with a Generalist Plannercited by 11
  2. 2023Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learningcited by 10
  3. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9
  4. 2022Automated Reinforcement Learning: An Overviewcited by 9
  5. 2023SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agentscited by 9
  6. 2021Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matchingcited by 8
  7. 2022Training Discrete Deep Generative Models via Gapped Straight-Through Estimatorcited by 8
  8. 2024EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learningcited by 8
  9. 2021Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulnesscited by 7
  10. 2023Neural Episodic Control with State Abstractioncited by 7
  11. 2023OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environmentscited by 7
  12. 2021Fragment-based Sequential Translation for Molecular Optimizationcited by 6
  13. 2023Online Reinforcement Learning in Non-Stationary Context-Driven Environmentscited by 6
  14. 2023Nonprehensile Planar Manipulation through Reinforcement Learning with Multimodal Categorical Explorationcited by 6
  15. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  16. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 6
  17. 2024Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusioncited by 6
  18. 2024Stochastic Q-learning for Large Discrete Action Spacescited by 5
  19. 2024CALE: Continuous Arcade Learning Environmentcited by 5
  20. 2022Continuous Monte Carlo Graph Searchcited by 4
  21. 2022Continuous Neural Algorithmic Plannerscited by 4
  22. 2024Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learningcited by 4
  23. 2025Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstractioncited by 4
  24. 2022AdaCat: Adaptive Categorical Discretization for Autoregressive Modelscited by 3
  25. 2023When Learning Is Out of Reach, Reset: Generalization in Autonomous Visuomotor Reinforcement Learningcited by 3
  26. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
  27. 2021Soft Actor-Critic With Integer Actionscited by 2
  28. 2024Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learningcited by 2
  29. 2024Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scalescited by 2
  30. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimationcited by 2
  31. 2025Trajectory-Class-Aware Multi-Agent Reinforcement Learningcited by 2
  32. 2026Embodied Task Planning via Graph-Informed Action Generation with Large Language Modelscited by 2
  33. 2021Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learningcited by 1
  34. 2023V2N Service Scaling with Deep Reinforcement Learningcited by 1
  35. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learningcited by 1
  36. 2025Neural Path Guiding with Distribution Factorizationcited by 1
  37. 2025Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learningcited by 1
  38. 2025Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learningcited by 1

50 most cited of 65 All 65 (JSON)