mlsanityData to 7 Oct 2026

Learning Continuous Control Policies by Stochastic Value Gradients

arXiv 1510.09142 ↗First posted 30 Oct 2015cs.LG cs.NEcited by 156

Not on the map. Why?

Abstract (arXiv)

We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment in- stead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Oct 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 2; 2020: 3; 2021: 42; 2022: 30; 2023: 29; 2024: 21; 2025: 17; 2026: 11.

156 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  4. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  5. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  6. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  7. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  8. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  9. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  10. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  11. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  12. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
38 more
  1. 2022Tutorial on amortized optimizationcited by 83
  2. 2022A Survey on Model-based Reinforcement Learningcited by 65
  3. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  4. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  5. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  6. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  7. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  8. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimationcited by 42
  9. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  10. 2022Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregationcited by 34
  11. 2023Stochastic Generative Flow Networkscited by 32
  12. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  13. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  14. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
  15. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLcited by 24
  16. 2023A Generalist Dynamics Model for Controlcited by 24
  17. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  18. 2022Bellman Meets Hawkes: Model-Based Reinforcement Learning via Temporal Point Processescited by 23
  19. 2021Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Controlcited by 22
  20. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  21. 2023Deep Learning in Deterministic Computational Mechanicscited by 21
  22. 2024Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attributioncited by 21
  23. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  24. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  25. 2021Improved Regret Bound and Experience Replay in Regularized Policy Iterationcited by 18
  26. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  27. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  28. 2022Physics-Informed Model-Based Reinforcement Learningcited by 17
  29. 2023Simplified Temporal Consistency Reinforcement Learningcited by 17
  30. 2021Bayesian Bellman Operatorscited by 16
  31. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 16
  32. 2022A Meta Reinforcement Learning Approach for Predictive Autoscaling in the Cloudcited by 16
  33. 2022Training Efficient Controllers via Analytic Policy Gradientcited by 16
  34. 2021Discovering Diverse Athletic Jumping Strategiescited by 15
  35. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  36. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  37. 2022Latent Variable Representation for Reinforcement Learningcited by 14
  38. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14

50 most cited of 156 All 156 (JSON)