mlsanityData to 14 Oct 2026

FERPO: Forward Entropy-Regularized Policy Optimization

arXiv 2610.02198 ↗First posted 1 Oct 2026cs.LG cs.AI cs.ROcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 37 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

25 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Relative Entropy Pathwise Policy Optimization15 Jul

Other references · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2018DeepMind Control Suitecited by 956
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  9. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  10. 2018Maximum a Posteriori Policy Optimisationcited by 328
  11. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  12. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
12 more
  1. 2021The Bayesian Learning Rulecited by 99
  2. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
  3. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 41
  4. 2021Recomposing the Reinforcement Learning Building Blocks with Hypernetworkscited by 35
  5. 2021On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributionscited by 23
  6. 2024Learning-based legged locomotion; state of the art and future perspectivescited by 23
  7. 2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 23
  8. 2020How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimizationcited by 16
  9. 2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 16
  10. 2024When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learningcited by 14
  11. 2025Learning Boltzmann Generators via Constrained Mass Transportcited by 9
  12. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning
  2. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning
  3. 2025Effective Reinforcement Learning Control using Conservative Soft Actor-Critic
  4. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
  5. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
  6. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
  7. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  8. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation