# Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies

arXiv: https://arxiv.org/abs/2607.19928 · HTML: https://mlsanity.com/paper/2607.19928/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 59 references that are in the map.
- First posted on arXiv 22 Jul 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 19 of 19)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023)
- [Exploratory LQG Mean Field Games with Entropy Regularization](https://mlsanity.com/paper/2011.12946/) (2020)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2407.17226/) (2024)
- [Regret of exploratory policy improvement and q-learning](https://mlsanity.com/paper/2411.01302/) (2024)
- [Continuous-time reinforcement learning for optimal switching over multiple regimes](https://mlsanity.com/paper/2512.04697/) (2025)
- [Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations](https://mlsanity.com/paper/2604.27372/) (2026)
- [Entropy-Regularized Stochastic Games](https://mlsanity.com/paper/1907.11543/) (2019)
- [Square-root regret bounds for continuous-time episodic Markov decision processes](https://mlsanity.com/paper/2210.00832/) (2022)
- [Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2507.00358/) (2025)
- [Learning Distributed Equilibria in Linear-Quadratic Stochastic Differential Games: An α-Potential Approach](https://mlsanity.com/paper/2602.16555/) (2026)
- [A Coupled Optimization Framework for Correlated Equilibria in Normal-Form Game](https://mlsanity.com/paper/2403.16223/) (2024)
- [Entropy Regularized Reinforcement Learning for Zero-Sum Stochastic Differential Games in a Regime-Switching Jump-Diffusion Process](https://mlsanity.com/paper/2606.28669/) (2026)
- [Entropy-Regularized Reinforcement Learning for Linear-Quadratic Stackelberg Differential Games in Regime-Switching Diffusion Models](https://mlsanity.com/paper/2606.28671/) (2026)

## Abstract (arXiv)

We study entropy-regularized exploratory control in finite N-player stochastic differential games under a response model in which each player conditions on the opponents' currently realized actions and evaluates continuation with that profile frozen. The resulting Gibbs best responses form a system of full conditional densities, which need not admit a common joint law. We characterize joint realizability by a cross-partial condition on the entropy-scaled Hamiltonian gradients and, on simply connected action domains, by an equivalent entropy-weighted potential structure. When compatibility fails, a coordinate-path construction yields a joint density whose full conditionals satisfy explicit quadratic Kullback--Leibler bounds. We extend the analysis to stationary discounted problems and derive martingale and policy-improvement characterizations for learning the frozen response maps. A two-player linear-quadratic example illustrates the compatibility criterion and the associated learning procedure.