mlsanityData to 7 Oct 2026

OGBench: Benchmarking Offline Goal-Conditioned RL

arXiv 2410.20092 ↗First posted 26 Oct 2024cs.LG cs.AIcited by 216

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline goal-conditioned reinforcement learning (GCRL) is a major problem in reinforcement learning (RL) because it provides a simple, unsupervised, and domain-agnostic way to acquire diverse behaviors and representations from unlabeled data without rewards. Despite the importance of this setting, we lack a standard benchmark that can systematically evaluate the capabilities of offline GCRL algorithms. In this work, we propose OGBench, a new, high-quality benchmark for algorithms research in offline goal-conditioned RL. OGBench consists of 8 types of environments, 85 datasets, and reference implementations of 6 representative offline GCRL algorithms. We have designed these challenging and realistic environments and datasets to directly probe different capabilities of algorithms, such as stitching, long-horizon reasoning, and the ability to handle high-dimensional inputs and stochasticity. While representative algorithms may rank similarly on prior benchmarks, our experiments reveal stark strengths and weaknesses in these different capabilities, providing a strong foundation for building new algorithms. Project page: https://seohong.me/projects/ogbench

Key facts. Cited by 216 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 70 references that are in the map. First posted on arXiv 26 Oct 2024.

Builds on

54 of 70 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Diffused Task-Agnostic Milestone Planner6 Dec
  2. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning3 Nov
  3. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models30 Oct
  4. 2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learning7 Jul
  5. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?30 May
  6. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression7 Jun
  7. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning24 May
  8. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2017Hindsight Experience Replaycited by 1,096
34 more
  1. 2018DeepMind Control Suitecited by 935
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  6. 2019Learning Latent Plans from Playcited by 310
  7. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  8. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  9. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 209
  10. 2018Semi-parametric Topological Memory for Navigationcited by 203
  11. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  12. 2019Planning with Goal-Conditioned Policiescited by 144
  13. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  14. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  15. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  16. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  17. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  18. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  19. 2024Foundation Policies with Hilbert Representationscited by 89
  20. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  21. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  22. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  23. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  24. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  25. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  26. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  27. 2023Contrastive Difference Predictive Codingcited by 35
  28. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  29. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  30. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  31. 2022Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddingscited by 22
  32. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
  33. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  34. 2022Powderworld: A Platform for Understanding Generalization via Rich Task Distributionscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 63; 2026: 152.

216 papers.

Sort:

By area:

  1. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 237
  2. 2025Flow Q-Learningcited by 162
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  4. 2025Reinforcement Learning with Action Chunkingcited by 97
  5. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  6. 2025Horizon Reduction Makes RL Scalablecited by 55
  7. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  8. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
  9. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  10. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  11. 2026Hierarchical Planning with Latent World Modelscited by 34
  12. 2025Monte Carlo Tree Diffusion for System 2 Planningcited by 33
38 more
  1. 2026Q-learning with Adjoint Matchingcited by 33
  2. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  3. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  4. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  5. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  6. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
  7. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 17
  8. 2026Predictive but Not Plannable: RC-aux for Latent World Modelscited by 16
  9. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  10. 2026Sub-JEPA: Subspace Gaussian Regularization for Stable End-to-End World Modelscited by 15
  11. 2026Sensorimotor World Models: Perception for Action via Inverse Dynamicscited by 14
  12. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  13. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 13
  14. 2025Dual Goal Representationscited by 13
  15. 2026stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluationcited by 13
  16. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  17. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 12
  18. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  19. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  20. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  21. 2026Fast LeWorldModelcited by 12
  22. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
  23. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  24. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  25. 2026Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Modelscited by 11
  26. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
  27. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learningcited by 9
  28. 2025CREPE: Controlling Diffusion with Replica Exchangecited by 9
  29. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  30. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  31. 2025Flattening Hierarchies with Policy Bootstrappingcited by 8
  32. 2025Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planningcited by 8
  33. 2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
  34. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  35. 2025Semantic World Modelscited by 8
  36. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 8
  37. 2026stable-worldmodel-v1: Reproducible World Modeling Research and Evaluationcited by 8
  38. 2025Intention-Conditioned Flow Occupancy Modelscited by 7

50 most cited of 216 All 216 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
  2. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  3. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
  4. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  5. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios
  6. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  7. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  8. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL