mlsanityData to 14 Oct 2026

MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator

arXiv 2312.03991 ↗First posted 7 Dec 2023cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) faces a significant challenge of distribution shift. Model-free offline RL penalizes the Q value for out-of-distribution (OOD) data or constrains the policy closed to the behavior policy to tackle this problem, but this inhibits the exploration of the OOD region. Model-based offline RL, which uses the trained environment model to generate more OOD data and performs conservative policy optimization within that model, has become an effective method for this problem. However, the current model-based algorithms rarely consider agent robustness when incorporating conservatism into policy. Therefore, the new model-based offline algorithm with a conservative Bellman operator (MICRO) is proposed. This method trades off performance and robustness via introducing the robust Bellman operator into the algorithm. Compared with previous model-based algorithms with robust adversarial models, MICRO can significantly reduce the computation cost by only choosing the minimal Q value in the state uncertainty set. Extensive experiments demonstrate that MICRO outperforms prior RL algorithms in offline RL benchmark and is considerably robust to adversarial perturbations.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 43 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

36 of 43 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023CROP: Conservative Reward for Model-based Offline Policy Optimization26 Oct
  2. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning16 Sep
  3. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
  4. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  5. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  6. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  8. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  9. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  12. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
7 more
  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2017Robust Adversarial Reinforcement Learningcited by 451
  7. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  9. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  10. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  11. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  12. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
5 more
  1. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  2. 2022Robust Reinforcement Learning using Offline Datacited by 96
  3. 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
  4. 2020Robust Reinforcement Learning with Wasserstein Constraintcited by 21
  5. 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimizationcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 1.

9 papers.

  1. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6
  2. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  3. 2025DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Rewardcited by 2
  4. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  5. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  6. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiencycited by 0
  7. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RLcited by 0
  8. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Controlcited by 0
  9. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021DROMO: Distributionally Robust Offline Model-based Policy Optimization
  2. 2026Residuals-based Offline Reinforcement Learning
  3. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning
  4. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning
  5. 2020MOReL : Model-Based Offline Reinforcement Learning
  6. 2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood
  7. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  8. 2023Model-based Offline Policy Optimization with Adversarial Network