mlsanityData to 7 Oct 2026

Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and model

arXiv 2102.11513 ↗First posted 23 Feb 2021cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) shows great potential in sequential decision-making. At present, mainstream RL algorithms are data-driven, which usually yield better asymptotic performance but much slower convergence compared with model-driven methods. This paper proposes mixed policy gradient (MPG) algorithm, which fuses the empirical data and the transition model in policy gradient (PG) to accelerate convergence without performance degradation. Formally, MPG is constructed as a weighted average of the data-driven and model-driven PGs, where the former is the derivative of the learned Q-value function, and the latter is that of the model-predictive return. To guide the weight design, we analyze and compare the upper bound of each PG error. Relying on that, a rule-based method is employed to heuristically adjust the weights. In particular, to get a better PG, the weight of the data-driven PG is designed to grow along the learning process while the other to decrease. Simulation results show that the MPG method achieves the best asymptotic performance and convergence speed compared with other baseline algorithms.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 23 Feb 2021.

Builds on

19 of 34 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  10. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  11. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  12. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
7 more
  1. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  2. 2017Revisiting Distributed Synchronous SGDcited by 142
  3. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  4. 2018Neural Lander: Stable Drone Landing Control using Learned Dynamicscited by 82
  5. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  6. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  7. 2019Direct and indirect reinforcement learningcited by 11

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 4; 2023: 1.

8 papers.

  1. 2022Reachability Constrained Reinforcement Learningcited by 52
  2. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  3. 2023Aligning Language Models with Human Preferences via a Bayesian Approachcited by 27
  4. 2021Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligencecited by 25
  5. 2021Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learningcited by 4
  6. 2022Improve Generalization of Driving Policy at Signalized Intersections with Adversarial Learningcited by 2
  7. 2022Integrated Decision and Control for High-Level Automated Vehicles by Mixed Policy Gradient and Its Experiment Verificationcited by 0
  8. 2022Smoothing Policy Iteration for Zero-sum Markov Gamescited by 0