mlsanityData to 7 Oct 2026

Taylor TD-learning

arXiv 2302.14182 ↗First posted 27 Feb 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Many reinforcement learning approaches rely on temporal-difference (TD) learning to learn a critic. However, TD-learning updates can be high variance. Here, we introduce a model-based RL framework, Taylor TD, which reduces this variance in continuous state-action settings. Taylor TD uses a first-order Taylor series expansion of TD updates. This expansion allows Taylor TD to analytically integrate over stochasticity in the action-choice, and some stochasticity in the state distribution for the initial state and action of each TD update. We include theoretical and empirical evidence that Taylor TD updates are indeed lower variance than standard TD updates. Additionally, we show Taylor TD has the same stable learning guarantees as standard TD-learning with linear function approximation under a reasonable assumption. Next, we combine Taylor TD with the TD3 algorithm, forming TaTD3. We show TaTD3 performs as well, if not better, than several state-of-the art model-free and model-based baseline algorithms on a set of standard benchmark tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

32 of 52 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2016OpenAI Gymcited by 1,684
  9. 2015Automatic differentiation in machine learning: a surveycited by 846
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  12. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
20 more
  1. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  2. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  3. 2017Sobolev Training for Neural Networkscited by 136
  4. 2013Risk-sensitive Reinforcement Learningcited by 129
  5. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  6. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  7. 2017Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networkscited by 58
  8. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  9. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  10. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  11. 2020How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimizationcited by 16
  12. 2017Expected Policy Gradientscited by 14
  13. 2021Variance Penalized On-Policy and Off-Policy Actor-Criticcited by 10
  14. 2020Taylor Expansion Policy Optimizationcited by 9
  15. 2017Mean Actor Criticcited by 8
  16. 2015Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policiescited by 5
  17. 2019All-Action Policy Gradient Methods: A Numerical Integration Approachcited by 4
  18. 2022Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimizationcited by 4
  19. 2018Fourier Policy Gradientscited by 3
  20. 2015An Empirical Evaluation of True Online TD(λ)cited by 1

Built on by

Not cited yet.