mlsanityData to 7 Oct 2026

Natural Policy Gradient and Actor Critic Methods for Constrained Multi-Task Reinforcement Learning

arXiv 2405.02456 ↗First posted 3 May 2024math.OC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-task reinforcement learning (RL) aims to find a single policy that effectively solves multiple tasks at the same time. This paper presents a constrained formulation for multi-task RL where the goal is to maximize the average performance of the policy across tasks subject to bounds on the performance in each task. We consider solving this problem both in the centralized setting, where information for all tasks is accessible to a single server, and in the decentralized setting, where a network of agents, each given one task and observing local information, cooperate to find the solution of the globally constrained objective using local communication. We first propose a primal-dual algorithm that provably converges to the globally optimal solution of this constrained formulation under exact gradient evaluations. When the gradient is unknown, we further develop a sampled-based actor-critic algorithm that finds the optimal policy using online samples of state, action, and reward. Finally, we study the extension of the algorithm to the linear function approximation setting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

42 of 56 references.

References · 42

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2015Policy Distillationcited by 414
  3. 2018Reward Constrained Policy Optimizationcited by 397
  4. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  6. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  7. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  9. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  10. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  11. 2013On the Convergence of Decentralized Gradient Descentcited by 158
  12. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
30 more
  1. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  2. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  3. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  4. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 120
  5. 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 84
  6. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 78
  7. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 55
  8. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 46
  9. 2020Non-asymptotic Convergence Analysis of Two Time-scale (Natural) Actor-Critic Algorithmscited by 43
  10. 2024Sharing Knowledge in Multi-Task Deep Reinforcement Learningcited by 42
  11. 2021Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithmcited by 39
  12. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 35
  13. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 33
  14. 2020Learning with Safety Constraints: Sample Complexity of Reinforcement Learning for Constrained MDPscited by 32
  15. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  16. 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
  17. 2020A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learningcited by 24
  18. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  19. 2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 22
  20. 2021A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learningcited by 20
  21. 2021Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processescited by 14
  22. 2021Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcited by 13
  23. 2019Policy Distillation and Value Matching in Multiagent Reinforcement Learningcited by 10
  24. 2020Finite-Time Convergence Rates of Decentralized Stochastic Approximation with Applications in Multi-Agent and Multi-Task Learningcited by 10
  25. 2022Decentralized Natural Policy Gradient with Variance Reduction for Collaborative Multi-Agent Reinforcement Learningcited by 8
  26. 2021A Stochastic Newton Algorithm for Distributed Convex Optimizationcited by 6
  27. 2021MDPGT: Momentum-based Decentralized Policy Gradient Trackingcited by 6
  28. 2013Distributed Constrained Optimization by Consensus-Based Primal-Dual Perturbation Methodcited by 5
  29. 2022Learning Action Translator for Meta Reinforcement Learning on Sparse-Reward Taskscited by 5
  30. 2015Primal-Dual Algorithm for Distributed Constrained Optimizationcited by 1

Built on by

Not cited yet.