Data to 7 Oct 2026

RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning

arXiv 2204.12581 ↗First posted 26 Apr 2022cs.LG cs.AIcited by 115

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) aims to find performant policies from logged data without further environment interaction. Model-based algorithms, which learn a model of the environment from the dataset and perform conservative policy optimisation within that model, have emerged as a promising approach to this problem. In this work, we present Robust Adversarial Model-Based Offline RL (RAMBO), a novel approach to model-based offline RL. We formulate the problem as a two-player zero sum game against an adversarial environment model. The model is trained to minimise the value function while still accurately predicting the transitions in the dataset, forcing the policy to act conservatively in areas not covered by the dataset. To approximately solve the two-player game, we alternate between optimising the policy and adversarially optimising the model. The problem formulation that we address is theoretically grounded, resulting in a probably approximately correct (PAC) performance guarantee and a pessimistic value function which lower bounds the value function in the true environment. We evaluate our approach on widely studied offline RL benchmarks, and demonstrate that it outperforms existing state-of-the-art baselines.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 91 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

62 of 91 references.

Closest ancestors · 24

Same direction, newest first

12 more

Other references · 38

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2018Recurrent World Models Facilitate Policy Evolutioncited by 761
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
2019Solving Rubik's Cube with a Robot Handcited by 699
2021A Survey of Uncertainty in Deep Neural Networkscited by 675

26 more

2017Robust Adversarial Reinforcement Learningcited by 439
2017Adversarial Attacks on Neural Network Policiescited by 314
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
2020Model-Based Offline Planningcited by 126
2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
2020Robust Deep Reinforcement Learning through Adversarial Losscited by 73
2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
2019Risk Averse Robust Adversarial Reinforcement Learningcited by 54
2017Reinforcement Learning under Model Mismatchcited by 52
2019Learning When-to-Treat Policiescited by 49
2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
2021Robust Policy Gradient against Strong Data Corruptioncited by 30
2020Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learningcited by 23
2021Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learningcited by 14
2020Pessimism About Unknown Unknowns Inspires Conservatismcited by 13
2017Batch Policy Gradient Methods for Improving Neural Conversation Modelscited by 10
2020Minimax Regret Optimisation for Robust Planning in Uncertain Markov Decision Processescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 45; 2024: 28; 2025: 25; 2026: 12.

115 papers.

Sort:

By area:

2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
2023Provable Offline Preference-Based Reinforcement Learningcited by 55
2023A Survey on Causal Reinforcement Learningcited by 43
2024Policy-Guided Diffusioncited by 40
2024Diffusion Model Predictive Controlcited by 35
2023Adversarial Model for Offline Reinforcement Learningcited by 31
2025Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failurescited by 30
2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
2023Quantification of Uncertainty with Adversarial Modelscited by 28
2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26

38 more

2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
2025RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbationscited by 23
2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2023DITTO: Offline Imitation Learning with World Modelscited by 20
2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
2023Survival Instinct in Offline Reinforcement Learningcited by 20
2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
2023TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfercited by 18
2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 18
2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learningcited by 17
2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimizationcited by 16
2023Model-based Offline Reinforcement Learning with Count-based Conservatismcited by 16
2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
2023Reward-Free Curricula for Training Robust World Modelscited by 13
2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learningcited by 12
2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12
2025Scalable Offline Model-Based RL with Action Chunkscited by 12
2022User-Interactive Offline Reinforcement Learningcited by 11
2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
2023Value function estimation using conditional diffusion models for controlcited by 10
2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learningcited by 10
2023Conservative State Value Estimation for Offline Reinforcement Learningcited by 9
2023More for Less: Safe Policy Improvement With Stronger Performance Guaranteescited by 9
2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learningcited by 8
2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
2025VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Drivingcited by 8

50 most cited of 115 All 115 (JSON)

Similar, not cited

Close in wording, no citation either way