# Risk-Sensitive Stochastic Optimal Control as Rao-Blackwellized Markovian Score Climbing

arXiv: https://arxiv.org/abs/2312.14000 · HTML: https://mlsanity.com/paper/2312.14000/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 49 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [Filtering Variational Objectives](https://mlsanity.com/paper/1705.09279/) (2017)
- [Differentiable Particle Filtering via Entropy-Regularized Optimal Transport](https://mlsanity.com/paper/2102.07850/) (2021)
- [MushroomRL: Simplifying Reinforcement Learning Research](https://mlsanity.com/paper/2001.01102/) (2020)
- [On Particle Methods for Parameter Estimation in State-Space Models](https://mlsanity.com/paper/1412.8695/) (2014)
- [Markovian Score Climbing: Variational Inference with KL(p||q)](https://mlsanity.com/paper/2003.10374/) (2020)
- [Stochastic Optimal Control as Approximate Input Inference](https://mlsanity.com/paper/1910.03003/) (2019)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [On backward smoothing algorithms](https://mlsanity.com/paper/2207.00976/) (2022)
- [State and parameter learning with PaRIS particle Gibbs](https://mlsanity.com/paper/2301.00900/) (2023)

## Built on by (most cited first, 2 of 2)

- [Nesting Particle Filters for Experimental Design in Dynamical Systems](https://mlsanity.com/paper/2402.07868/) (2024)
- [Recursive Nested Filtering for Efficient Amortized Bayesian Experimental Design](https://mlsanity.com/paper/2409.05354/) (2024)

## Abstract (arXiv)

Stochastic optimal control of dynamical systems is a crucial challenge in sequential decision-making. Recently, control-as-inference approaches have had considerable success, providing a viable risk-sensitive framework to address the exploration-exploitation dilemma. Nonetheless, a majority of these techniques only invoke the inference-control duality to derive a modified risk objective that is then addressed within a reinforcement learning framework. This paper introduces a novel perspective by framing risk-sensitive stochastic control as Markovian score climbing under samples drawn from a conditional particle filter. Our approach, while purely inference-centric, provides asymptotically unbiased estimates for gradient-based policy optimization with optimal importance weighting and no explicit value function learning. To validate our methodology, we apply it to the task of learning neural non-Gaussian feedback policies, showcasing its efficacy on numerical benchmarks of stochastic dynamical systems.