# Consensus Based Stochastic Control

arXiv: https://arxiv.org/abs/2501.17801 · HTML: https://mlsanity.com/paper/2501.17801/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 40 of its 73 references that are in the map.
- First posted on arXiv 29 Jan 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 40)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space](https://mlsanity.com/paper/2005.05409/) (2020)
- [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019)
- [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020)
- [Nonparametric inference of interaction laws in systems of agents from trajectory data](https://mlsanity.com/paper/1812.06003/) (2018)
- [In value-based deep reinforcement learning, a pruned network is a good network](https://mlsanity.com/paper/2402.12479/) (2024)
- [Stochastic Optimal Control for Collective Variable Free Sampling of Molecular Transition Paths](https://mlsanity.com/paper/2207.02149/) (2022)
- [Q-value Regularized Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.17098/) (2024)
- [Recent Developments in Machine Learning Methods for Stochastic Control and Games](https://mlsanity.com/paper/2303.10257/) (2023)
- [Mean Field Game of Controls and An Application To Trade Crowding](https://mlsanity.com/paper/1610.09904/) (2016)
- [Neural networks-based backward scheme for fully nonlinear PDEs](https://mlsanity.com/paper/1908.00412/) (2019)
- [Consensus-Based Optimization Methods Converge Globally](https://mlsanity.com/paper/2103.15130/) (2021)
- [The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm](https://mlsanity.com/paper/2406.07826/) (2024)
- [Mean-field neural networks-based algorithms for McKean-Vlasov control problems *](https://mlsanity.com/paper/2212.11518/) (2022)
- [Model-based Reinforcement Learning for Parameterized Action Spaces](https://mlsanity.com/paper/2404.03037/) (2024)
- [Constrained Reinforcement Learning Under Model Mismatch](https://mlsanity.com/paper/2405.01327/) (2024)
- [On Bellman equations for continuous-time policy evaluation I: discretization and approximation](https://mlsanity.com/paper/2407.05966/) (2024)
- [Efficiency of the Price Formation Process in Presence of High Frequency Participants: a Mean Field Game analysis](https://mlsanity.com/paper/1305.6323/) (2013)
- [Mean-field type modeling of nonlocal crowd aversion in pedestrian crowd dynamics](https://mlsanity.com/paper/1701.09118/) (2017)
- [rm E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.11842/) (2023)

## Abstract (arXiv)

We propose a gradient-free deep reinforcement learning algorithm to solve high-dimensional, finite-horizon stochastic control problems. Although the recently developed deep reinforcement learning framework has achieved great success in solving these problems, direct estimation of policy gradients from Monte Carlo sampling often suffers from high variance. To address this, we introduce the Momentum Consensus-Based Optimization (M-CBO) and Adaptive Momentum Consensus-Based Optimization (Adam-CBO) frameworks. These methods optimize policies using Monte Carlo estimates of the value function, rather than its gradients. Adjustable Gaussian noise supports efficient exploration, helping the algorithm converge to optimal policies in complex, nonconvex environments. Numerical results confirm the accuracy and scalability of our approach across various problem dimensions and show the potential for extension to mean-field control problems. Theoretically, we prove that M-CBO can converge to the optimal policy under some assumptions.