SOMBRL: Scalable and Optimistic Model-Based RL

arXiv 2511.20066 ↗First posted 25 Nov 2025cs.LGcited by 7

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We address the challenge of efficient exploration in model-based reinforcement learning (MBRL), where the system dynamics are unknown and the RL agent must learn directly from online interactions. We propose Scalable and Optimistic MBRL (SOMBRL), an approach based on the principle of optimism in the face of uncertainty. SOMBRL learns an uncertainty-aware dynamics model and greedily maximizes a weighted sum of the extrinsic reward and the agent's epistemic uncertainty. SOMBRL is compatible with any policy optimizers or planners, and under common regularity assumptions on the system, we show that SOMBRL has sublinear regret for nonlinear dynamics in the (i) finite-horizon, (ii) discounted infinite-horizon, and (iii) non-episodic settings. Additionally, SOMBRL offers a flexible and scalable solution for principled exploration. We evaluate SOMBRL on state-based and visual-control environments, where it displays strong performance across all tasks and baselines. We also evaluate SOMBRL on a dynamic RC car hardware and show SOMBRL outperforms the state-of-the-art, illustrating the benefits of principled exploration for MBRL.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 60 of its 87 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

60 of 87 references.

References · 60

2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2016OpenAI Gymcited by 1,736
2023Mastering Diverse Domains through World Modelscited by 1,253
2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
2018Exploration by Random Network Distillationcited by 993
2018DeepMind Control Suitecited by 956
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752

38 more

2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
2017A Tutorial on Thompson Samplingcited by 433
2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
2022DayDreamer: World Models for Physical Robot Learningcited by 400
2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
2020Planning to Explore via Self-Supervised World Modelscited by 368
2017On Kernelized Multi-armed Banditscited by 352
2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
2019Self-Supervised Exploration via Disagreementcited by 265
2018Visual Reinforcement Learning with Imagined Goalscited by 251
2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 192
2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 159
2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
2021Discovering and Achieving Goals via World Modelscited by 131
2020Naive Exploration is Optimal for Online LQRcited by 116
2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 95
2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 90
2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
2020Active Learning for Nonlinear System Identification with Guaranteescited by 80
2019Learning Linear-Quadratic Regulators Efficiently with only √T Regretcited by 70
2019Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activitycited by 70
2017Boltzmann Exploration Done Rightcited by 69
2013Empowerment -- an Introductioncited by 68
2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 62
2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 52
2016A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patientscited by 38
2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
2023Foundations of Reinforcement Learning and Interactive Decision Makingcited by 33
2023Optimistic Active Exploration of Dynamical Systemscited by 32
2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
2019AMZ Driverless: The Full Autonomous Racing Systemcited by 29
2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
2019Actively Learning Gaussian Process Dynamicscited by 26
2022Model-based Causal Bayesian Optimizationcited by 26

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

2026Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learningcited by 2
2026HaM-World: Soft-Hamiltonian World Models with Selective Memory for Planningcited by 2
2026Model-Based Reinforcement Learning for Control under Time-Varying Dynamicscited by 1
2026RAY-TOLD: Ray-Based Latent Dynamics for Dense Dynamic Obstacle Avoidance with TDMPCcited by 0
2026Sampling-Based Safe Reinforcement Learningcited by 0
2026Deep Epistemic Value Functions for Optimistic Explorationcited by 0
2026Tactile Curiosity Drives Robot Interactioncited by 0

Similar, not cited

Close in wording, no citation either way