mlsanityData to 14 Oct 2026

Policy Gradient With Serial Markov Chain Reasoning

arXiv 2210.06766 ↗First posted 13 Oct 2022cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We introduce a new framework that performs decision-making in reinforcement learning (RL) as an iterative reasoning process. We model agent behavior as the steady-state distribution of a parameterized reasoning Markov chain (RMC), optimized with a new tractable estimate of the policy gradient. We perform action selection by simulating the RMC for enough reasoning steps to approach its steady-state distribution. We show our framework has several useful properties that are inherently missing from traditional RL. For instance, it allows agent behavior to approximate any continuous distribution over actions by parameterizing the RMC with a simple Gaussian transition function. Moreover, the number of reasoning steps to reach convergence can scale adaptively with the difficulty of each action selection decision and can be accelerated by re-using past solutions. Our resulting algorithm achieves state-of-the-art performance in popular Mujoco and DeepMind Control benchmarks, both for proprioceptive and pixel-based tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 101 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

48 of 101 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  11. 2016Density estimation using Real NVPcited by 2,042
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
35 more
  1. 2018DeepMind Control Suitecited by 956
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  3. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  7. 2019Solving Rubik's Cube with a Robot Handcited by 734
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  9. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  13. 2018Maximum a Posteriori Policy Optimisationcited by 328
  14. 2019Challenges of Real-World Reinforcement Learningcited by 291
  15. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  16. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  17. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  18. 2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 148
  19. 2020Phasic Policy Gradientcited by 123
  20. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
  21. 2018Iterative Amortized Inferencecited by 94
  22. 2017On Nesting Monte Carlo Estimatorscited by 83
  23. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 71
  24. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  25. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
  26. 2017Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Netcited by 30
  27. 2020Temporally-Extended ε-Greedy Explorationcited by 29
  28. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 28
  29. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  30. 2018Revisiting the Gelman-Rubin Diagnosticcited by 18
  31. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  32. 2017Learning to Generate Samples from Noise through Infusion Trainingcited by 15
  33. 2020Iterative Amortized Policy Optimizationcited by 13
  34. 2018Boosting Trust Region Policy Optimization by Normalizing Flows Policycited by 11
  35. 2015GSNs : Generative Stochastic Networkscited by 9

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  2. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes
  2. 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
  3. 2026Bayesian policy gradient and actor-critic algorithms
  4. 2023Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes
  5. 2025Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions
  6. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
  7. 2025Taxonomy and Trends in Reinforcement Learning for Robotics and Control Systems: A Structured Review
  8. 2026Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs