Data to 7 Oct 2026

Model-Augmented Actor-Critic: Backpropagating through Paths

arXiv 2005.08068 ↗First posted 16 May 2020cs.LG cs.AI stat.MLcited by 61

Not on the map. Why?

Abstract (arXiv)

Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 19; 2023: 10; 2024: 10; 2025: 3; 2026: 5.

61 papers.

Sort:

By area:

2022Temporal Difference Learning for Model Predictive Controlcited by 375
2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
2021Gradients are Not All You Needcited by 82
2022A Survey on Model-based Reinforcement Learningcited by 65
2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
2021Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvaturecited by 32
2025CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Drivingcited by 31
2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
2022Deep Interactive Motion Prediction and Planning: Playing Games with Motion Prediction Modelscited by 27

38 more

2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
2021Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rolloutscited by 24
2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
2021Contrastive Active Inferencecited by 21
2023Deep Learning in Deterministic Computational Mechanicscited by 21
2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
2022Physics-Informed Model-Based Reinforcement Learningcited by 17
2023Simplified Temporal Consistency Reinforcement Learningcited by 17
2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14
2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 13
2024Do Transformer World Models Give Better Policy Gradients?cited by 10
2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learningcited by 10
2023Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithmscited by 7
2021Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimizationcited by 6
2021On-Policy Model Errors in Reinforcement Learningcited by 6
2022Integrating Symmetry into Differentiable Planning with Steerable Convolutionscited by 6
2022Model-based Reinforcement Learning with Multi-step Plan Value Estimationcited by 6
2022Scaling up and Stabilizing Differentiable Planning with Implicit Differentiationcited by 6
2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimizationcited by 6
2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
2021OffCon3: What is state of the art anyway?cited by 5
2022Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratiocited by 5
2022Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learningcited by 5
2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 5
2023Learning an Inventory Control Policy with General Inventory Arrival Dynamicscited by 5
2024Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial Statescited by 4
2025Differentiable Information Enhanced Model-Based Reinforcement Learningcited by 3
2025First Order Model-Based RL through Decoupled Backpropagationcited by 3
2026WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Controlcited by 3
2022Imitation Learning via Differentiable Physicscited by 2
2022Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rolloutscited by 2
2023Hint assisted reinforcement learning: an application in radio astronomycited by 2
2024Physics-Informed Model and Hybrid Planning for Efficient Dyna-Style Reinforcement Learningcited by 2
2024Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimationcited by 2

50 most cited of 61 All 61 (JSON)