mlsanityData to 14 Oct 2026

Dual Alignment Maximin Optimization for Offline Model-based RL

arXiv 2502.00850 ↗First posted 2 Feb 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning agents face significant deployment challenges due to the synthetic-to-real distribution mismatch. While most prior research has focused on improving the fidelity of synthetic sampling and incorporating off-policy mechanisms, the directly integrated paradigm often fails to ensure consistent policy behavior in biased models and underlying environmental dynamics, which inherently arise from discrepancies between behavior and learning policies. In this paper, we first shift the focus from model reliability to policy discrepancies while optimizing for expected returns, and then self-consistently incorporate synthetic data, deriving a novel actor-critic paradigm, Dual Alignment Maximin Optimization (DAMO). It is a unified framework to ensure both model-environment policy consistency and synthetic and offline data compatibility. The inner minimization performs dual conservative value estimation, aligning policies and trajectories to avoid out-of-distribution states and actions, while the outer maximization ensures that policy improvements remain consistent with inner value estimates. Empirical evaluations demonstrate that DAMO effectively ensures model and policy alignments, achieving competitive performance across diverse benchmark tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 53 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

34 of 53 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression25 Oct
  2. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update1 Feb
  3. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  4. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  5. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  6. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  8. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  9. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
7 more
  1. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2021Convex optimizationcited by 2,515
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  8. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  9. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  10. 2019Meta Reinforcement Learning for Sim-to-real Domain Adaptationcited by 49
  11. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 36
  12. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 25
3 more
  1. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetescited by 23
  2. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  3. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  2. 2020MOPO: Model-based Offline Policy Optimization
  3. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
  4. 2026Offline Policy Optimization with Posterior Sampling
  5. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximization
  6. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  7. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  8. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets