mlsanityData to 14 Oct 2026

Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies

arXiv 2602.23811 ↗First posted 27 Feb 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

We investigate the theoretical aspects of offline reinforcement learning (RL) under general function approximation. While prior works (e.g., Xie et al., 2021) have established the theoretical foundations of learning a good policy from offline data via pessimism, existing algorithms that are computationally tractable (often in an oracle-efficient sense), such as PSPI, only apply to finite and small action spaces. Moreover, these algorithms rely on state-wise mirror descent and require actors to be implicitly induced from the critic functions, failing to accommodate standalone policy parameterization which is ubiquitous in practice. In this work, we address these limitations and extend the theoretical guarantees to parameterized policy classes over large or continuous action spaces. When extending mirror descent to parameterized policies, we identify contextual coupling as the core difficulty, and show how connecting mirror descent to natural policy gradient leads to novel analyses, guarantees, and algorithmic insights, including a surprising unification between offline RL and imitation learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 53 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

28 of 53 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees5 Oct
  2. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees14 Nov
  3. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance4 Sep
  4. 2023Offline Reinforcement Learning with Additional Covering Distributions22 May
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  6. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Introduction to Online Convex Optimizationcited by 803
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  6. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  7. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  9. 2024Diffusion Policy Policy Optimizationcited by 280
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  11. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  12. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
6 more
  1. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  2. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  3. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  4. 2025The Pitfalls of Imitation Learning when Actions are Continuouscited by 26
  5. 2020Statistically Efficient Off-Policy Policy Gradientscited by 25
  6. 2023Low-Rank MDPs with Continuous Action Spacescited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
  2. 2024Functional Acceleration for Policy Mirror Descent
  3. 2024MoMA: Model-based Mirror Ascent for Offline Reinforcement Learning
  4. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  5. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  6. 2021Offline RL Without Off-Policy Evaluation
  7. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
  8. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning