mlsanityData to 7 Oct 2026

Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping

arXiv 2301.02099 ↗First posted 5 Jan 2023cs.RO cs.AI cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem in robotics, where online interaction with the environment is extremely time-consuming. Moreover, manually designing reward functions for every single desired skill is prohibitive. Prior works targeted these challenges by learning goal-conditioned policies from offline datasets without manually specified rewards, through hindsight relabelling. These methods suffer from the issue of sparsity of rewards, and fail at long-horizon tasks. In this work, we propose a novel self-supervised learning phase on the pre-collected dataset to understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. We evaluate our method on three continuous control tasks, and show that our model significantly outperforms existing approaches, especially on tasks that involve long-term planning.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

31 of 36 references.

References · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2018DeepMind Control Suitecited by 935
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  8. 2019Self-Supervised Exploration via Disagreementcited by 263
  9. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  10. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  11. 2021Reinforcement Learning with Prototypical Representationscited by 179
  12. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
19 more
  1. 2019Planning with Goal-Conditioned Policiescited by 144
  2. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  3. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  4. 2021Discovering and Achieving Goals via World Modelscited by 131
  5. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  6. 2018Episodic Curiosity through Reachabilitycited by 109
  7. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  8. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  9. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  10. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  11. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  12. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  13. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  14. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  15. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  16. 2018Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learningcited by 23
  17. 2019Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learningcited by 18
  18. 2021Efficient Self-Supervised Data Collection for Offline Robot Learningcited by 8
  19. 2022Walk the Random Walk: Learning to Discover and Reach Goals Without Supervisioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 5; 2025: 7; 2026: 7.

25 papers.

Sort:

By area:

  1. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  2. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  3. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  4. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  5. 2023ReRoGCRL: Representation-based Robustness in Goal-Conditioned Reinforcement Learningcited by 6
  6. 2025Automatic Reward Shaping from Confounded Offline Datacited by 6
  7. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
  8. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5
  9. 2023PcLast: Discovering Plannable Continuous Latent Statescited by 4
  10. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioningcited by 3
  11. 2023Goal-conditioned Offline Planning from Curious Explorationcited by 3
  12. 2026Confounding Robust Continuous Control via Automatic Reward Shapingcited by 2
13 more
  1. 2026Survival Reinforcement Learning: Toward Scalable Self-Supervised RLcited by 2
  2. 2024Single-Shot Learning of Stable Dynamical Systems for Long-Horizon Manipulation Taskscited by 1
  3. 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrationscited by 1
  4. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  5. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  6. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  7. 2024QuasiNav: Asymmetric Cost-Aware Navigation Planning with Constrained Quasimetric Reinforcement Learningcited by 0
  8. 2025Proposing Hierarchical Goal-Conditioned Policy Planning in Multi-Goal Reinforcement Learningcited by 0
  9. 2025Towards Fully Automated Decision-Making Systems for Greenhouse Control: Challenges and Opportunitiescited by 0
  10. 2025Reachability Weighted Offline Goal-conditioned Resamplingcited by 0
  11. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Ordercited by 0
  12. 2026Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamicscited by 0
  13. 2026Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulationcited by 0