Reward-Predictive Clustering
Not on the map. Why?
Abstract (arXiv)
Recent advances in reinforcement-learning research have demonstrated impressive results in building algorithms that can out-perform humans in complex tasks. Nevertheless, creating reinforcement-learning systems that can build abstractions of their experience to accelerate learning in new contexts still remains an active area of research. Previous work showed that reward-predictive state abstractions fulfill this goal, but have only be applied to tabular settings. Here, we provide a clustering algorithm that enables the application of such state abstractions to deep learning settings, providing compressed representations of an agent's inputs that preserve the ability to predict sequences of reward. A convergence theorem and simulations show that the resulting reward-predictive deep network maximally compresses the agent's inputs, significantly speeding up learning in high dimensional visual control tasks. Furthermore, we present different generalization experiments and analyze under which conditions a pre-trained reward-predictive representation network can be re-used without re-training to accelerate learning -- a form of systematic out-of-distribution transfer.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 54 references that are in the map. First posted on arXiv 7 Nov 2022.
Builds on
31 of 54 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2018World Modelscited by 1,444
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
- 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
- 2015Policy Distillationcited by 414
19 more
- 2016Successor Features for Transfer in Reinforcement Learningcited by 331
- 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
- 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
- 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
- 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
- 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
- 2017Value Prediction Networkcited by 117
- 2016The Predictron: End-To-End Learning and Planningcited by 92
- 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
- 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
- 2016Deep Successor Reinforcement Learningcited by 72
- 2017Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physicscited by 57
- 2016Deep Reinforcement Learning with Successor Features for Navigation across Similar Environmentscited by 50
- 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 40
- 2018Measuring and Characterizing Generalization in Deep Reinforcement Learningcited by 20
- 2017Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learningcited by 10
- 2016A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Gamescited by 9
- 2018Learning the Reward Function for a Misspecified Modelcited by 4
- 2018Transfer with Model Features in Reinforcement Learningcited by 2
Built on by
Not cited yet.