Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?
Not on the map. Why?
Abstract (arXiv)
Goal-conditioned reinforcement learning (GCRL) relies heavily on how target goals are represented to the policy. While recent methods encode goals via temporal distance, occupancy, or controllability, it remains unclear how much downstream performance actually depends on representation quality. We study this in offline GCRL by constructing an exact temporal-distance goal representation in deterministic mazes. We then systematically corrupt its geometric quality while keeping the downstream learner fixed. Across OGBench navigation tasks and two algorithms, large changes in goal-representation quality produce almost no change in performance. However, applying the same interventions to the agent's current state more than doubles success, revealing the state pathway as the true bottleneck. Building on this insight, we show that simple random Fourier positional encodings substantially improve performance on the hardest navigation tasks without map information or objective modifications. Overall, our findings suggest that in state-based offline navigation, improving how the agent's current state is represented matters far more than refining the goal representation. Code will be released soon.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
29 of 39 references.
References · 29
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
- 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
- 2018On the Spectral Bias of Neural Networkscited by 1,321
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2019Designing and Interpreting Probes with Control Taskscited by 608
- 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
17 more
- 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
- 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
- 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
- 2018Semi-parametric Topological Memory for Navigationcited by 203
- 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
- 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
- 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
- 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
- 2024Foundation Policies with Hilbert Representationscited by 89
- 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 56
- 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
- 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
- 2022On the Learning and Learnability of Quasimetricscited by 15
- 2025Horizon Generalization in Reinforcement Learningcited by 15
- 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
- 2025Dual Goal Representationscited by 13
Built on by
Not cited yet.