# Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls

arXiv: https://arxiv.org/abs/2104.09311 · HTML: https://mlsanity.com/paper/2104.09311/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 47 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 13 of 13)

- [Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning](https://mlsanity.com/paper/1703.07710/) (2017)
- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator](https://mlsanity.com/paper/1805.09388/) (2018)
- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression](https://mlsanity.com/paper/1804.02605/) (2018)
- [Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon](https://mlsanity.com/paper/2006.15316/) (2020)
- [Exploration versus exploitation in reinforcement learning: a stochastic control approach](https://mlsanity.com/paper/1812.01552/) (2018)
- [Regularity and stability of feedback relaxed controls](https://mlsanity.com/paper/2001.03148/) (2020)
- [Gradient Flows for Regularized Stochastic Control Problems](https://mlsanity.com/paper/2006.05956/) (2020)
- [Robustness to incorrect system models in stochastic control](https://mlsanity.com/paper/1803.06046/) (2018)
- [Continuity of Utility Maximization under Weak Convergence](https://mlsanity.com/paper/1811.01420/) (2018)
- [A convex penalty for switching control of partial differential equations](https://mlsanity.com/paper/1702.07505/) (2017)
- [Sparse solutions in optimal control of PDEs with uncertain parameters: the linear case](https://mlsanity.com/paper/1804.05678/) (2018)

## Built on by (most cited first, 24 of 24)

- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Policy Optimization for Continuous Reinforcement Learning](https://mlsanity.com/paper/2305.18901/) (2023)
- [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023)
- [Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models](https://mlsanity.com/paper/2112.10264/) (2021)
- [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025)
- [Entropy annealing for policy mirror descent in continuous time and space](https://mlsanity.com/paper/2405.20250/) (2024)
- [Exploratory Optimal Stopping: A Singular Control Formulation](https://mlsanity.com/paper/2408.09335/) (2024)
- [Logarithmic regret bounds for continuous-time average-reward Markov decision processes](https://mlsanity.com/paper/2205.11168/) (2022)
- [Linear convergence of a policy gradient method for some finite horizon continuous time control problems](https://mlsanity.com/paper/2203.11758/) (2022)
- [Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learning](https://mlsanity.com/paper/2208.04466/) (2022)
- [Square-root regret bounds for continuous-time episodic Markov decision processes](https://mlsanity.com/paper/2210.00832/) (2022)
- [Statistical Learning with Sublinear Regret of Propagator Models](https://mlsanity.com/paper/2301.05157/) (2023)
- [Data-driven rules for multidimensional reflection problems](https://mlsanity.com/paper/2311.06639/) (2023)
- [An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations](https://mlsanity.com/paper/2407.10065/) (2024)
- [Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control](https://mlsanity.com/paper/2510.17122/) (2025)
- [Reinforcement Learning for Jump-Diffusions, with Financial Applications](https://mlsanity.com/paper/2405.16449/) (2024)
- [Robust Reinforcement Learning under Diffusion Models for Data with Jumps](https://mlsanity.com/paper/2411.11697/) (2024)
- [Discretization error from regularized Reinforcement Learning to continuous-time stochastic control](https://mlsanity.com/paper/2604.21179/) (2026)
- [ε-Policy Gradient for Online Pricing](https://mlsanity.com/paper/2405.03624/) (2024)
- [Learning to steer with Brownian noise](https://mlsanity.com/paper/2410.03221/) (2024)
- [Learning Optimal Search Strategies](https://mlsanity.com/paper/2603.02356/) (2026)
- [An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows](https://mlsanity.com/paper/2604.05398/) (2026)
- [Learning to Bid in Repeated Second-Price Auctions with Dynamic Values and Aggregated Feedback](https://mlsanity.com/paper/2605.28133/) (2026)
- [Inverse Optimal Control for Linear Quadratic Problem with Poisson Jumps: Model-Free Inverse Reinforcement Learning Approaches](https://mlsanity.com/paper/2607.03169/) (2026)

## Abstract (arXiv)

We study finite-time horizon continuous-time linear-convex reinforcement learning problems in an episodic setting. In this problem, the unknown linear jump-diffusion process is controlled subject to nonsmooth convex costs. We show that the associated linear-convex control problems admit Lipchitz continuous optimal feedback controls and further prove the Lipschitz stability of the feedback controls, i.e., the performance gap between applying feedback controls for an incorrect model and for the true model depends Lipschitz-continuously on the magnitude of perturbations in the model coefficients; the proof relies on a stability analysis of the associated forward-backward stochastic differential equation. We then propose a novel least-squares algorithm which achieves a regret of the order O(√Nln N) on linear-convex learning problems with jumps, where N is the number of learning episodes; the analysis leverages the Lipschitz stability of feedback controls and concentration properties of sub-Weibull random variables. Numerical experiment confirms the convergence and the robustness of the proposed algorithm.