mlsanityData to 14 Oct 2026

Actor-Critic based Improper Reinforcement Learning

arXiv 2207.09090 ↗First posted 19 Jul 2022cs.LG cs.AI cs.SYcited by 3

Not on the map. Why?

Abstract (arXiv)

We consider an improper reinforcement learning setting where a learner is given M base controllers for an unknown Markov decision process, and wishes to combine them optimally to produce a potentially new controller that can outperform each of the base ones. This can be useful in tuning across controllers, learnt possibly in mismatched or simulated environments, to obtain a good controller for a given target environment with relatively few trials. Towards this, we propose two algorithms: (1) a Policy Gradient-based approach; and (2) an algorithm that can switch between a simple Actor-Critic (AC) based scheme and a Natural Actor-Critic (NAC) scheme depending on the available information. Both algorithms operate over a class of improper mixtures of the given controllers. For the first case, we derive convergence rate guarantees assuming access to a gradient oracle. For the AC-based approach we provide convergence rate guarantees to a stationary point in the basic AC case and to a global optimum in the NAC case. Numerical results on (i) the standard control theoretic benchmark of stabilizing an cartpole; and (ii) a constrained queueing task show that our improper policy optimization algorithm can stabilize the system even when the base policies at its disposal are unstable.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 53 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

21 of 53 references.

References · 21

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  2. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  3. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  4. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  5. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  6. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  7. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  8. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 159
  9. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  10. 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 142
  11. 2015Explore no more: Improved high-probability regret bounds for non-stochastic banditscited by 120
  12. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
9 more
  1. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  2. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
  3. 2020Black-Box Control for Linear Dynamical Systemscited by 43
  4. 2013From average case complexity to improper learning complexitycited by 16
  5. 2021Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcited by 14
  6. 2013Efficient Reinforcement Learning for High Dimensional Linear Quadratic Systemscited by 13
  7. 2019Boosting for Control of Dynamical Systemscited by 7
  8. 2020Regret Analysis of a Markov Policy Gradient Algorithm for Multi-arm Banditscited by 6
  9. 2013More data speeds up training time in learning halfspaces over sparse vectorscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 1.

3 papers.

  1. 2022Hierarchical Policy Blending as Inference for Reactive Robot Controlcited by 8
  2. 2022Hierarchical Policy Blending As Optimal Transportcited by 4
  3. 2024Oracle-Efficient Reinforcement Learning for Max Value Ensemblescited by 1