mlsanityData to 14 Oct 2026

Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learning

arXiv 2206.05357 ↗First posted 10 Jun 2022cs.LG math.OCcited by 15

Not on the map. Why?

Abstract (arXiv)

We study policy optimization for Markov decision processes (MDPs) with multiple reward value functions, which are to be jointly optimized according to given criteria such as proportional fairness (smooth concave scalarization), hard constraints (constrained MDP), and max-min trade-off. We propose an Anchor-changing Regularized Natural Policy Gradient (ARNPG) framework, which can systematically incorporate ideas from well-performing first-order methods into the design of policy optimization algorithms for multi-objective MDP problems. Theoretically, the designed algorithms based on the ARNPG framework achieve tildeO(1/T) global convergence with exact gradients. Empirically, the ARNPG-guided algorithms also demonstrate superior performance compared to some existing policy gradient-based approaches in both exact gradients and sample-based scenarios.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 23 of its 49 references that are in the map. First posted on arXiv 10 Jun 2022.

Builds on

23 of 49 references.

References · 23

  1. 2016OpenAI Gymcited by 1,736
  2. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  3. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  4. 2013Optimization, Learning, and Games with Predictable Sequencescited by 188
  5. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  6. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  7. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  8. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  9. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  10. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  11. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 80
  12. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
11 more
  1. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  2. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 34
  3. 2019Online Primal-Dual Mirror Descent under Stochastic Constraintscited by 31
  4. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 19
  5. 2019Reinforcement Learning for Joint Optimization of Multiple Rewardscited by 15
  6. 2021Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithmcited by 12
  7. 2020Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learningcited by 11
  8. 2022Towards Painless Policy Optimization for Constrained MDPscited by 8
  9. 2015A Simple Parallel Algorithm with an O(1/t) Convergence Rate for General Convex Programscited by 6
  10. 2017A Primal-Dual Parallel Method with O(1/ε) Convergence for Constrained Composite Convex Programscited by 3
  11. 2020Provable Multi-Objective Reinforcement Learning with Generative Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 3; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  2. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 28
  3. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 15
  4. 2023Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Gamescited by 13
  5. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 7
  6. 2023Adapting Static Fairness to Sequential Decision-Making: Bias Mitigation Strategies towards Equal Long-term Benefit Ratecited by 6
  7. 2024Multi-objective Deep Learning: Taxonomy and Survey of the State of the Artcited by 6
  8. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  9. 2024C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Frontcited by 4
  10. 2025Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learningcited by 2
  11. 2026Breaking the Bias Barrier in Concave Multi-Objective Reinforcement Learningcited by 2
  12. 2024Linear Convergence of Independent Natural Policy Gradient in Games with Entropy Regularizationcited by 1
3 more
  1. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  2. 2026Step-level Denoising-time Diffusion Alignment with Multiple Objectivescited by 0
  3. 2026Bias-Controlled Primal-Dual Natural Actor-Critic: Optimal Rates for Constrained Multi-Objective Average-Reward RLcited by 0