mlsanityData to 7 Oct 2026

Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning

arXiv 2102.11327 ↗First posted 22 Feb 2021cs.LG cs.AIcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based offline reinforcement learning approaches generally rely on bounds of model error. Estimating these bounds is usually achieved through uncertainty estimation methods. In this work, we combine parametric and nonparametric methods for uncertainty estimation through a novel latent space based metric. In particular, we build upon recent advances in Riemannian geometry of generative models to construct a pullback metric of an encoder-decoder based forward model. Our proposed metric measures both the quality of out-of-distribution samples as well as the discrepancy of examples in the data. We leverage our method for uncertainty estimation in a pessimistic model-based framework, showing a significant improvement upon contemporary model-based offline approaches on continuous control and autonomous driving benchmarks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 81 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

47 of 81 references.

Closest ancestors · 7

Same direction, newest first

  1. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  2. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  3. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  11. 2020Mastering Atari with Discrete World Modelscited by 990
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
28 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  3. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  4. 2016Variational Lossy Autoencodercited by 175
  5. 2016A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialoguescited by 174
  6. 2020Revisiting Fundamentals of Experience Replaycited by 171
  7. 2017Latent Space Oddity: on the Curvature of Deep Generative Modelscited by 169
  8. 2017Optimizing the Latent Space of Generative Networkscited by 157
  9. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  10. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  11. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  12. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  13. 2019Learning Action Representations for Reinforcement Learningcited by 75
  14. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  15. 2020Guided Variational Autoencoder for Disentanglement Learningcited by 57
  16. 2020Simple and Effective VAE Training with Calibrated Decoderscited by 57
  17. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  18. 2017Metrics for Deep Generative Modelscited by 53
  19. 2019Learning Hierarchical Priors in VAEscited by 53
  20. 2020Geometrically Enriched Latent Spacescited by 45
  21. 2017Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Modelscited by 39
  22. 2020Variational Autoencoders with Riemannian Brownian Motion Priorscited by 34
  23. 2020Learning Flat Latent Manifolds with VAEscited by 33
  24. 2020Masked Contrastive Representation Learning for Reinforcement Learningcited by 32
  25. 2019The Natural Language of Actionscited by 28
  26. 2018Fast Approximate Geodesics for Deep Generative Modelscited by 13
  27. 2019Multi-View Reinforcement Learningcited by 12
  28. 2019Disentangled Skill Embeddings for Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 1; 2025: 3; 2026: 2.

10 papers.

  1. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  2. 2024Embedding-Aligned Language Modelscited by 9
  3. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  4. 2025Riemann2: Learning Riemannian Submanifolds from Riemannian Datacited by 5
  5. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
  6. 2023Representation-Driven Reinforcement Learningcited by 4
  7. 2025Emergent Riemannian geometry over learning discrete computations on continuous manifoldscited by 3
  8. 2026Geometry-aware similarity metrics for neural representations on Riemannian and statistical manifoldscited by 3
  9. 2023Short and Straight: Geodesics on Differentiable Manifoldscited by 0
  10. 2026World-Task Factorization for Robot Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning from Images with Latent Space Models
  2. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
  3. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  4. 2024Offline Model-Based Reinforcement Learning with Anti-Exploration
  5. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning
  6. 2026Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitions
  7. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching
  8. 2026Guided Riemannian Optimization (GuRO): Bridging Model Predictive Control and Decision Transformers