mlsanityData to 7 Oct 2026

MoMA: Model-based Mirror Ascent for Offline Reinforcement Learning

arXiv 2401.11380 ↗First posted 21 Jan 2024cs.LG math.ST stat.MEcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based offline reinforcement learning methods (RL) have achieved state-of-the-art performance in many decision-making problems thanks to their sample efficiency and generalizability. Despite these advancements, existing model-based offline RL approaches either focus on theoretical studies without developing practical algorithms or rely on a restricted parametric policy space, thus not fully leveraging the advantages of an unrestricted policy space inherent to model-based methods. To address this limitation, we develop MoMA, a model-based mirror ascent algorithm with general function approximations under partial coverage of offline data. MoMA distinguishes itself from existing literature by employing an unrestricted policy class. In each iteration, MoMA conservatively estimates the value function by a minimization procedure within a confidence set of transition models in the policy evaluation step, then updates the policy with general function approximations instead of commonly-used parametric policy classes in the policy improvement step. Under some mild assumptions, we establish theoretical guarantees of MoMA by proving an upper bound on the suboptimality of the returned policy. We also provide a practically implementable, approximate version of the algorithm. The effectiveness of MoMA is demonstrated via numerical studies.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 55 references that are in the map. First posted on arXiv 21 Jan 2024.

Builds on

35 of 55 references.

Closest ancestors · 24

Same direction, newest first

  1. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  2. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
  3. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  4. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  5. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  6. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  7. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  8. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
12 more
  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  5. 2020Batch Value-function Approximation with Only Realizability11 Aug
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 11

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  9. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
  10. 2018Deep Reinforcement Learning for General Video Game AIcited by 25
  11. 2022Policy Optimization over General State and Action Spacescited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
  2. 2023A Survey on Offline Model-Based Reinforcement Learning
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  4. 2024Offline Model-Based Reinforcement Learning with Anti-Exploration
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  7. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning
  8. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning