mlsanityData to 7 Oct 2026

Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spaces

arXiv 2402.10820 ↗First posted 16 Feb 2024cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study the problem of learning optimal behavior from sub-optimal datasets for goal-conditioned offline reinforcement learning under sparse rewards, invertible actions and deterministic transitions. To mitigate the effects of distribution shift, we propose MetricRL, a method that combines metric learning for value function approximation with weighted imitation learning for policy estimation. MetricRL avoids conservative or behavior-cloning constraints, enabling effective learning even in severely sub-optimal regimes. We introduce distance monotonicity as a key property linking metric representations to optimality and design an objective that explicitly promotes it. Empirically, MetricRL consistently outperforms prior state-of-the-art goal-conditioned RL methods in recovering near-optimal behavior from sub-optimal offline data.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 59 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

44 of 59 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Supported Trust Region Optimization for Offline Reinforcement Learning15 Nov
  2. 2022Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning9 Jun
  3. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  4. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
1 more
  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 31

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  8. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  9. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  10. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  11. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  12. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
19 more
  1. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  2. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  3. 2019Unsupervised State Representation Learning in Ataricited by 144
  4. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  5. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  6. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  7. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  8. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  9. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  10. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  11. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  12. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  13. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 28
  14. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  15. 2020Plan2Vec: Unsupervised Representation Learning by Latent Planscited by 21
  16. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  17. 2019Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learningcited by 18
  18. 2022State Representation Learning for Goal-Conditioned Reinforcement Learningcited by 11
  19. 2023Contrastive Example-Based Controlcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 2.

3 papers.

  1. 2024QuasiNav: Asymmetric Cost-Aware Navigation Planning with Constrained Quasimetric Reinforcement Learningcited by 0
  2. 2026Geometry of Uncertainty: Learning Metric Spaces for Multimodal State Estimation in RLcited by 0
  3. 2026World Models in Pieces: Structural Certification for General Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization
  2. 2023Distance Weighted Supervised Learning for Offline Interaction Data
  3. 2021Behavior Constraining in Weight Space for Offline Reinforcement Learning
  4. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioning
  5. 2023Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
  6. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
  7. 2025Behavior Preference Regression for Offline Reinforcement Learning
  8. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learning