# Accelerating Goal-Conditioned RL Algorithms and Research

arXiv: https://arxiv.org/abs/2408.11052 · HTML: https://mlsanity.com/paper/2408.11052/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- First posted on arXiv 20 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Built on by (most cited first, 19 of 19)

- [1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities](https://mlsanity.com/paper/2503.14858/) (2025)
- [Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations](https://mlsanity.com/paper/2509.20478/) (2025)
- [Horizon Generalization in Reinforcement Learning](https://mlsanity.com/paper/2501.02709/) (2025)
- [DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning](https://mlsanity.com/paper/2505.19850/) (2025)
- [Contrastive Representations for Temporal Reasoning](https://mlsanity.com/paper/2508.13113/) (2025)
- [Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration](https://mlsanity.com/paper/2509.10656/) (2025)
- [Automaton Constrained Q-Learning](https://mlsanity.com/paper/2510.05061/) (2025)
- [Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization](https://mlsanity.com/paper/2605.06145/) (2026)
- [Survival Reinforcement Learning: Toward Scalable Self-Supervised RL](https://mlsanity.com/paper/2605.31273/) (2026)
- [NiceWebRL: a Python library for human subject experiments with reinforcement learning environments](https://mlsanity.com/paper/2508.15693/) (2025)
- [Consistent Zero-Shot Imitation with Contrastive Goal Inference](https://mlsanity.com/paper/2510.17059/) (2025)
- [Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration](https://mlsanity.com/paper/2603.22273/) (2026)
- [Inverting the Bellman Equation: From Q-Values to World Models](https://mlsanity.com/paper/2606.21173/) (2026)
- [Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning](https://mlsanity.com/paper/2510.25311/) (2025)
- [TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.01665/) (2026)
- [Beyond Success Rates: Trainability and Extractability for Offline GCRL](https://mlsanity.com/paper/2602.05459/) (2026)
- [Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation](https://mlsanity.com/paper/2605.13554/) (2026)
- [Goal-Conditioned Agents that Learn Everything All at Once](https://mlsanity.com/paper/2605.23551/) (2026)
- [Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2606.16515/) (2026)

## Abstract (arXiv)

Self-supervision has the potential to transform reinforcement learning (RL), paralleling the breakthroughs it has enabled in other areas of machine learning. While self-supervised learning in other domains aims to find patterns in a fixed dataset, self-supervised goal-conditioned reinforcement learning (GCRL) agents discover new behaviors by learning from the goals achieved during unstructured interaction with the environment. However, these methods have failed to see similar success, both due to a lack of data from slow environment simulations as well as a lack of stable algorithms. We take a step toward addressing both of these issues by releasing a high-performance codebase and benchmark (JaxGCRL) for self-supervised GCRL, enabling researchers to train agents for millions of environment steps in minutes on a single GPU. By utilizing GPU-accelerated replay buffers, environments, and a stable contrastive RL algorithm, we reduce training time by up to 22×. Additionally, we assess key design choices in contrastive RL, identifying those that most effectively stabilize and enhance training performance. With this approach, we provide a foundation for future research in self-supervised GCRL, enabling researchers to quickly iterate on new ideas and evaluate them in diverse and challenging environments. Website + Code: https://github.com/MichalBortkiewicz/JaxGCRL