# Actor-Critic Learning for Risk-Constrained Linear Quadratic Regulation

arXiv: https://arxiv.org/abs/2510.22267 · HTML: https://mlsanity.com/paper/2510.22267/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 25 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 18 of 18)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Optimizing Static Linear Feedback: Gradient Method](https://mlsanity.com/paper/2004.09875/) (2020)
- [A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning](https://mlsanity.com/paper/2109.14756/) (2021)
- [Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems](https://mlsanity.com/paper/1809.05149/) (2018)
- [Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games](https://mlsanity.com/paper/1910.07498/) (2019)
- [Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs](https://mlsanity.com/paper/2104.04901/) (2021)
- [A Risk-Sensitive Finite-Time Reachability Approach for Safety of Stochastic Dynamic Systems](https://mlsanity.com/paper/1902.11277/) (2019)
- [Risk-sensitive safety analysis using Conditional Value-at-Risk](https://mlsanity.com/paper/2101.12086/) (2021)
- [On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control](https://mlsanity.com/paper/2201.09598/) (2022)
- [Separation of Learning and Control for Cyber-Physical Systems](https://mlsanity.com/paper/2107.06379/) (2021)
- [Risk-Constrained Linear-Quadratic Regulators](https://mlsanity.com/paper/2004.04685/) (2020)
- [Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Cost](https://mlsanity.com/paper/2103.15363/) (2021)
- [Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control](https://mlsanity.com/paper/2212.02072/) (2022)
- [Risk-Aware Finite-Horizon Social Optimal Control of Mean-Field Coupled Linear-Quadratic Subsystems](https://mlsanity.com/paper/2406.05239/) (2024)
- [Combined Learning and Control: A New Paradigm for Optimal Control with Unknown Dynamics](https://mlsanity.com/paper/2510.00308/) (2025)

## Abstract (arXiv)

In this paper, we investigate the infinite-horizon risk-constrained linear quadratic regulator problem (RC-QR), which augments the classical LQR formulation with a statistical constraint on the variability of the system state to incorporate risk awareness, a key requirement in safety-critical control applications. We propose an actor-critic learning algorithm that jointly performs policy evaluation and policy improvement in a model-free and online manner. The RC-QR problem is first reformulated as a max-min optimization problem, from which we develop a multi-time-scale stochastic approximation scheme. The critic employs temporal-difference learning to estimate the action-value function, the actor updates the policy parameters via a policy gradient step, and the dual variable is adapted through gradient ascent to enforce the risk constraint.