mlsanityData to 14 Oct 2026

SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning

arXiv 2408.12970 ↗First posted 23 Aug 2024cs.LGcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The performance of offline reinforcement learning (RL) suffers from the limited size and quality of static datasets. Model-based offline RL addresses this issue by generating synthetic samples through a dynamics model to enhance overall performance. To evaluate the reliability of the generated samples, uncertainty estimation methods are often employed. However, model ensemble, the most commonly used uncertainty estimation method, is not always the best choice. In this paper, we propose a Search-based Uncertainty estimation method for Model-based Offline RL (SUMO) as an alternative. SUMO characterizes the uncertainty of synthetic samples by measuring their cross entropy against the in-distribution dataset samples, and uses an efficient search-based method for implementation. In this way, SUMO can achieve trustworthy uncertainty estimation. We integrate SUMO into several model-based offline RL algorithms including MOPO and Adapted MOReL (AMOReL), and provide theoretical analysis for them. Extensive experimental results on D4RL datasets demonstrate that SUMO can provide more accurate uncertainty estimation and boost the performance of base algorithms. These indicate that SUMO could be a better uncertainty estimator for model-based offline RL when used in either reward penalty or trajectory truncation. Our code is available and will be open-source for further research and development.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 26 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

19 of 26 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
  2. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  3. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  4. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 8

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017Billion-scale similarity search with GPUscited by 2,497
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  7. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  8. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 6.

12 papers.

  1. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  2. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robotscited by 17
  3. 2025Unraveling the Effects of Synthetic Data on End-to-End Autonomous Drivingcited by 12
  4. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  5. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6
  6. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 3
  7. 2026Robust Regularized Policy Iteration under Transition Uncertaintycited by 2
  8. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  9. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
  10. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  11. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0
  12. 2026In-Distribution Imagination for Model-Based Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  2. 2024Offline Model-Based Reinforcement Learning with Anti-Exploration
  3. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
  4. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning
  5. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  6. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
  7. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
  8. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness