# Operator-Theoretic Foundations and Policy Gradient Methods for General MDPs with Unbounded Costs

arXiv: https://arxiv.org/abs/2603.17875 · HTML: https://mlsanity.com/paper/2603.17875/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 71 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies](https://mlsanity.com/paper/1906.08383/) (2019)
- [Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence](https://mlsanity.com/paper/2105.11066/) (2021)
- [Unified Reinforcement Q-Learning for Mean Field Game and Control Problems](https://mlsanity.com/paper/2006.13912/) (2020)
- [Near Optimality of Finite Memory Feedback Policies in Partially Observed Markov Decision Processes](https://mlsanity.com/paper/2010.07452/) (2020)
- [Policy Optimization over General State and Action Spaces](https://mlsanity.com/paper/2211.16715/) (2022)
- [Convergence of Finite Memory Q-Learning for POMDPs and Near Optimality of Learned Policies under Filter Stability](https://mlsanity.com/paper/2103.12158/) (2021)
- [Model Based Control of Soft Robots: A Survey of the State of the Art and Open Challenges](https://mlsanity.com/paper/2110.01358/) (2021)
- [LMI Properties and Applications in Systems, Stability, and Control Theory](https://mlsanity.com/paper/1903.08599/) (2019)
- [Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.00661/) (2024)
- [Team Optimal Control of Coupled Subsystems with Mean-Field Sharing](https://mlsanity.com/paper/2012.01418/) (2020)
- [A Class of Dimension-free Metrics for the Convergence of Empirical Measures](https://mlsanity.com/paper/2104.12036/) (2021)
- [PPO in the Fisher-Rao geometry](https://mlsanity.com/paper/2506.03757/) (2025)
- [PDE-based Dynamic Control and Estimation of Soft Robotic Arms](https://mlsanity.com/paper/2203.13724/) (2022)
- [Deep Reinforcement Learning-Based Motion Planning and PDE Control for Flexible Manipulators](https://mlsanity.com/paper/2506.08639/) (2025)
- [Lambda-Policy Iteration: A Review and a New Implementation](https://mlsanity.com/paper/1507.01029/) (2015)
- [Stochastic Setup-Cost Inventory Model with Backorders and Quasiconvex Cost Functions](https://mlsanity.com/paper/1705.06814/) (2017)
- [Stochastic Comparative Statics in Markov Decision Processes](https://mlsanity.com/paper/1904.05481/) (2019)
- [Model-Based Reinforcement Learning Control of Reaction-Diffusion Problems](https://mlsanity.com/paper/2402.14446/) (2024)
- [Towards Active Flow Control Strategies Through Deep Reinforcement Learning](https://mlsanity.com/paper/2411.05536/) (2024)
- [Average-Cost MDPs with Infinite State and Action Sets: New Sufficient Conditions for Optimality Inequalities and Equations](https://mlsanity.com/paper/2412.01594/) (2024)

## Built on by (most cited first, 1 of 1)

- [Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework](https://mlsanity.com/paper/2605.10671/) (2026)

## Abstract (arXiv)

Markov decision processes (MDPs) is viewed as an optimization of an objective function over certain linear operators over general function spaces. A new existence result is established for the existence of optimal policies in general MDPs, which differs from the existence result derived previously in the literature. Using the well-established perturbation theory of linear operators, policy difference lemma is established for general MDPs and the Gauteaux derivative of the objective function as a function of the policy operator is derived. By upper bounding the policy difference via the theory of integral probability metric, a new majorization-minimization type policy gradient algorithm for general MDPs is derived. This leads to generalization of many well-known algorithms in reinforcement learning to cases with general state and action spaces. Further, by taking the integral probability metric as maximum mean discrepancy, a low-complexity policy gradient algorithm is derived for finite MDPs. The new algorithm, called MM-RKHS, appears to be superior to PPO algorithm due to low computational complexity, low sample complexity, and faster convergence.