# SDE Guided Monte Carlo Reinforcement Learning: A Stochastic Maximum Principle Approach for Robust Decision Making in Noisy Environments

arXiv: https://arxiv.org/abs/2607.22541 · HTML: https://mlsanity.com/paper/2607.22541/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 26 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 9 of 9)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems](https://mlsanity.com/paper/2309.00983/) (2023)
- [Stochastic Operator Network: A Stochastic Maximum Principle Based Approach to Operator Learning](https://mlsanity.com/paper/2507.10401/) (2025)
- [An efficient numerical algorithm for solving data driven feedback control problems](https://mlsanity.com/paper/2006.03047/) (2020)
- [An Online Algorithm for Solving Feedback Optimal Control Problems with Partial Observations](https://mlsanity.com/paper/2404.05734/) (2024)
- [Batch Sample-wise Stochastic Optimal Control via Stochastic Maximum Principle](https://mlsanity.com/paper/2505.02688/) (2025)

## Abstract (arXiv)

This work investigates whether the qualitative optimality conditions of the SMP can serve as guiding heuristics for designing robust tabular RL algorithms. We propose the SDE-MC-AC framework, which establishes a set of SMP-inspired correspondences: the potential field gradient is mapped to potential-based reward shaping, the diffusion coefficient to a softmax temperature, and the value function gradient magnitude to the episode-averaged absolute temporal-difference (TD) error. Crucially, we derive an SMP-motivated adaptive temperature schedule that scales exploration stochastically in response to local value uncertainty. The resulting Monte Carlo Actor-Critic agent integrates potential-shaped rewards, entropy regularization, and adaptive temperature control within episodic updates. We conduct a set of five principled experiments in a minimal, fixed maze testbed under three canonical noise regimes (perceptual, dynamic goal, action) to test five falsifiable hypotheses derived from the SMP. Our results provide evidence for (i) an inverted-U optimal stochasticity curve, (ii) the robust failure prevention of the adaptive schedule under non-stationary noise, (iii) convergence acceleration by potential field guidance, (iv) cross-noise generalization, and (v) the indispensability of the combined SDE components through systematic ablation. Trajectory visualizations reveal a characteristic ``macroscopically deterministic, microscopically stochastic'' navigation pattern consistent with the SDE model. The study demonstrates that even a heuristic transposition of the SMP into a discrete MDP can yield significant empirical gains, thereby opening a bridge between rigorous stochastic optimal control and practical robust reinforcement learning.