mlsanityData to 14 Oct 2026

RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning

arXiv 2601.23075 ↗First posted 30 Jan 2026cs.LG cs.ROcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

On-policy Reinforcement Learning (RL) remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy, and policy updates must be conservative. In this paper, we revisit actor policy representation as a first-class design choice for on-policy RL. We study discretized categorical actors, which represent each action dimension as a distribution over discrete bins and induce a policy objective analogous to classification cross-entropy loss. Building on architectural advances from supervised learning, we further pair discretized categorical actors with regularized networks, yielding RN-D. Across diverse continuous-control benchmarks, we show that simply replacing the standard Gaussian actor with our proposed actor substantially improves performance, achieving state-of-the-art results within on-policy RL. We release our code at https://github.com/alwaysbyx/RND-RL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 42 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

29 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Improving Value Estimation Critically Enhances Vanilla Policy Gradient25 May
  2. 2024Simple Policy Optimization29 Jan

Other references · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Layer Normalizationcited by 5,669
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2016OpenAI Gymcited by 1,736
15 more
  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  2. 2020On Layer Normalization in the Transformer Architecturecited by 922
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  4. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  5. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  6. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  7. 2020Phasic Policy Gradientcited by 123
  8. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 121
  9. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  10. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  11. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  12. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 38
  13. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
  14. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  15. 2018PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
  2. 2026Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
  4. 2023Harnessing Discrete Representations For Continual Reinforcement Learning
  5. 2025Categorical Policies: Multimodal Policy Learning and Exploration in Continuous Control
  6. 2024AFU: Actor-Free critic Updates in off-policy RL for continuous control
  7. 2025Distributions as Actions: A Unified Framework for Diverse Action Spaces
  8. 2025SHAP-Guided Kernel Actor-Critic for Explainable Reinforcement Learning