# Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching

arXiv: https://arxiv.org/abs/2404.17939 · HTML: https://mlsanity.com/paper/2404.17939/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 6 of its 25 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 6 of 6)

- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.02409/) (2022)
- [Actor critic learning algorithms for mean-field control with moment neural networks](https://mlsanity.com/paper/2309.04317/) (2023)
- [Reservoir optimization and Machine Learning methods](https://mlsanity.com/paper/2106.08097/) (2021)

## Built on by (most cited first, 8 of 8)

- [Exploratory Optimal Stopping: A Singular Control Formulation](https://mlsanity.com/paper/2408.09335/) (2024)
- [Continuous-time reinforcement learning for optimal switching over multiple regimes](https://mlsanity.com/paper/2512.04697/) (2025)
- [Reinforcement Learning for Jump-Diffusions, with Financial Applications](https://mlsanity.com/paper/2405.16449/) (2024)
- [Robust Reinforcement Learning under Diffusion Models for Data with Jumps](https://mlsanity.com/paper/2411.11697/) (2024)
- [Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergence](https://mlsanity.com/paper/2506.08121/) (2025)
- [Reinforcement Learning in Real Option Models](https://mlsanity.com/paper/2602.15643/) (2026)
- [An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows](https://mlsanity.com/paper/2604.05398/) (2026)
- [When Should a Human Take Back Control? Optimal Delegation under Turbulent AI Risk](https://mlsanity.com/paper/2609.32083/) (2026)

## Abstract (arXiv)

We propose a comprehensive framework for policy gradient methods tailored to continuous time reinforcement learning. This is based on the connection between stochastic control problems and randomised problems, enabling applications across various classes of Markovian continuous time control problems, beyond diffusion models, including e.g. regular, impulse and optimal stopping/switching problems. By utilizing change of measure in the control randomisation technique, we derive a new policy gradient representation for these randomised problems, featuring parametrised intensity policies. We further develop actor-critic algorithms specifically designed to address general Markovian stochastic control issues. Our framework is demonstrated through its application to optimal switching problems, with two numerical case studies in the energy sector focusing on real options.