mlsanityData to 14 Oct 2026

Goal-Conditioned Q-Learning as Knowledge Distillation

arXiv 2208.13298 ↗First posted 28 Aug 2022cs.LGcited by 1

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

Many applications of reinforcement learning can be formalized as goal-conditioned environments, where, in each episode, there is a "goal" that affects the rewards obtained during that episode but does not affect the dynamics. Various techniques have been proposed to improve performance in goal-conditioned environments, such as automatic curriculum generation and goal relabeling. In this work, we explore a connection between off-policy reinforcement learning in goal-conditioned settings and knowledge distillation. In particular: the current Q-value function and the target Q-value estimate are both functions of the goal, and we would like to train the Q-value function to match its target for all goals. We therefore apply Gradient-Based Attention Transfer (Zagoruyko and Komodakis 2017), a knowledge distillation technique, to the Q-function update. We empirically show that this can improve the performance of goal-conditioned off-policy reinforcement learning when the space of goals is high-dimensional. We also show that this technique can be adapted to allow for efficient learning in the case of multiple simultaneous sparse goals, where the agent can attain a reward by achieving any one of a large set of objectives, all specified at test time. Finally, to provide theoretical support, we give examples of classes of environments where (under some assumptions) standard off-policy algorithms such as DDPG require at least O(d^2) replay buffer transitions to learn an optimal policy, while our proposed technique requires only O(d) transitions, where d is the dimensionality of the goal and state space. Code is available at https://github.com/alevine0/ReenGAGE.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 30 of its 46 references that are in the map. First posted on arXiv 28 Aug 2022.

Builds on

30 of 46 references.

References · 30

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2017Deep Setscited by 1,421
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  12. 2015Policy Distillationcited by 440
18 more
  1. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  2. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  4. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  5. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
  6. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  7. 2021Learning One Representation to Optimize All Rewardscited by 115
  8. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  9. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 98
  10. 2020Automatic Curriculum Learning through Value Disagreementcited by 87
  11. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  12. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  13. 2019Reinforcement Learning with Attention that Works: A Self-Supervised Approachcited by 19
  14. 2021MHER: Model-based Hindsight Experience Replaycited by 15
  15. 2020Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learningcited by 12
  16. 2017Multi-focus Attention Network for Efficient Deep Reinforcement Learningcited by 10
  17. 2019A Closer Look at Double Backpropagationcited by 7
  18. 2022A Closer Look at Knowledge Distillation with Features, Logits, and Gradientscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Knowledge Boundary Discovery for Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation
  2. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  3. 2026X-KD: General Experiential Knowledge Distillation for Large Language Models
  4. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learning
  5. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learning
  6. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching
  7. 2024Q-WSL: Optimizing Goal-Conditioned RL with Weighted Supervised Learning via Dynamic Programming
  8. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL