# OGBench: Benchmarking Offline Goal-Conditioned RL

arXiv: https://arxiv.org/abs/2410.20092 · HTML: https://mlsanity.com/paper/2410.20092/

## Key facts

- Cited by 238 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 54 of its 70 references that are in the map.
- First posted on arXiv 26 Oct 2024.

## Placed in

- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 30 of 54)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Physics of Language Models: Part 3.2, Knowledge Manipulation](https://mlsanity.com/paper/2309.14402/) (2023)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning](https://mlsanity.com/paper/2304.01203/) (2023)
- [CORL: Research-oriented Deep Offline Reinforcement Learning Library](https://mlsanity.com/paper/2210.07105/) (2022)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)

## Built on by (most cited first, 30 of 238)

- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [Flow Q-Learning](https://mlsanity.com/paper/2502.02538/) (2025)
- [Steering Your Diffusion Policy with Latent Space Reinforcement Learning](https://mlsanity.com/paper/2506.15799/) (2025)
- [Reinforcement Learning with Action Chunking](https://mlsanity.com/paper/2507.07969/) (2025)
- [Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models](https://mlsanity.com/paper/2502.14819/) (2025)
- [Diffusion Guidance Is a Controllable Policy Improvement Operator](https://mlsanity.com/paper/2505.23458/) (2025)
- [Horizon Reduction Makes RL Scalable](https://mlsanity.com/paper/2506.04168/) (2025)
- [Inference-time Scaling of Diffusion Models through Classical Search](https://mlsanity.com/paper/2505.23614/) (2025)
- [1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities](https://mlsanity.com/paper/2503.14858/) (2025)
- [Hierarchical Planning with Latent World Models](https://mlsanity.com/paper/2604.03208/) (2026)
- [Generative Trajectory Stitching through Diffusion Composition](https://mlsanity.com/paper/2503.05153/) (2025)
- [Q-learning with Adjoint Matching](https://mlsanity.com/paper/2601.14234/) (2026)
- [Monte Carlo Tree Diffusion for System 2 Planning](https://mlsanity.com/paper/2502.07202/) (2025)
- [SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling](https://mlsanity.com/paper/2509.25756/) (2025)
- [Scaling Offline RL via Efficient and Expressive Shortcut Models](https://mlsanity.com/paper/2505.22866/) (2025)
- [Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations](https://mlsanity.com/paper/2509.20478/) (2025)
- [Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.12737/) (2025)
- [floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL](https://mlsanity.com/paper/2509.06863/) (2025)
- [TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.00739/) (2025)
- [stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation](https://mlsanity.com/paper/2605.21800/) (2026)
- [Sensorimotor World Models: Perception for Action via Inverse Dynamics](https://mlsanity.com/paper/2606.20104/) (2026)
- [Predictive but Not Plannable: RC-aux for Latent World Models](https://mlsanity.com/paper/2605.07278/) (2026)
- [Dual Goal Representations](https://mlsanity.com/paper/2510.06714/) (2025)
- [Sub-JEPA: Subspace Gaussian Regularization for Stable End-to-End World Models](https://mlsanity.com/paper/2605.09241/) (2026)
- [Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2509.06782/) (2025)
- [Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling](https://mlsanity.com/paper/2603.04553/) (2026)
- [Fast LeWorldModel](https://mlsanity.com/paper/2606.26217/) (2026)
- [Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following](https://mlsanity.com/paper/2502.05454/) (2025)
- [Normalizing Flows are Capable Models for Continuous Control](https://mlsanity.com/paper/2505.23527/) (2025)
- [Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning](https://mlsanity.com/paper/2506.21427/) (2025)

## Abstract (arXiv)

Offline goal-conditioned reinforcement learning (GCRL) is a major problem in reinforcement learning (RL) because it provides a simple, unsupervised, and domain-agnostic way to acquire diverse behaviors and representations from unlabeled data without rewards. Despite the importance of this setting, we lack a standard benchmark that can systematically evaluate the capabilities of offline GCRL algorithms. In this work, we propose OGBench, a new, high-quality benchmark for algorithms research in offline goal-conditioned RL. OGBench consists of 8 types of environments, 85 datasets, and reference implementations of 6 representative offline GCRL algorithms. We have designed these challenging and realistic environments and datasets to directly probe different capabilities of algorithms, such as stitching, long-horizon reasoning, and the ability to handle high-dimensional inputs and stochasticity. While representative algorithms may rank similarly on prior benchmarks, our experiments reveal stark strengths and weaknesses in these different capabilities, providing a strong foundation for building new algorithms. Project page: https://seohong.me/projects/ogbench