# Learning-based primal-dual optimal control of discrete-time stochastic systems with multiplicative noise

arXiv: https://arxiv.org/abs/2506.02613 · HTML: https://mlsanity.com/paper/2506.02613/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 11 of 11)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint](https://mlsanity.com/paper/1812.03565/) (2018)
- [Finite-time Analysis of Approximate Policy Iteration for the Linear Quadratic Regulator](https://mlsanity.com/paper/1905.12842/) (2019)
- [Model-Free Linear Quadratic Control via Reduction to Expert Prediction](https://mlsanity.com/paper/1804.06021/) (2018)
- [Can Primal Methods Outperform Primal-dual Methods in Decentralized Dynamic Optimization?](https://mlsanity.com/paper/2003.00816/) (2020)
- [Model-free optimal control of discrete-time systems with additive and multiplicative noises](https://mlsanity.com/paper/2008.08734/) (2020)
- [Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems](https://mlsanity.com/paper/2107.07788/) (2021)
- [On stabilizability and exact observability of stochastic systems with their applications](https://mlsanity.com/paper/2307.08915/) (2023)
- [Bounded Synthesis and Reinforcement Learning of Supervisors for Stochastic Discrete Event Systems with LTL Specifications](https://mlsanity.com/paper/2105.03081/) (2021)
- [Data-Driven Output Prediction and Control of Stochastic Systems: An Innovation-Based Approach](https://mlsanity.com/paper/2209.08995/) (2022)
- [Inverse linear-quadratic discrete-time finite-horizon optimal control for indistinguishable homogeneous agents: a convex optimization approach](https://mlsanity.com/paper/2109.01040/) (2021)

## Abstract (arXiv)

Reinforcement learning (RL) is an effective approach for solving optimal control problems without knowing the exact information of the system model. However, the classical Q-learning method, a model-free RL algorithm, has its limitations, such as lack of strict theoretical analysis and the need for artificial disturbances during implementation. This paper explores the partially model-free stochastic linear quadratic regulator (SLQR) problem for a system with multiplicative noise from the primal-dual perspective to address these challenges. This approach lays a strong theoretical foundation for understanding the intrinsic mechanisms of classical RL algorithms. We reformulate the SLQR into a non-convex primal-dual optimization problem and derive a strong duality result, which enables us to provide model-based and model-free algorithms for SLQR optimal policy design based on the Karush-Kuhn-Tucker (KKT) conditions. An illustrative example demonstrates the proposed model-free algorithm's validity, showcasing the central nervous system's learning mechanism in human arm movement.