Graph Backup: Data Efficient Backup Exploiting Markovian Transitions
Not on the map. Why?
Abstract (arXiv)
The successes of deep Reinforcement Learning (RL) are limited to settings where we have a large stream of online experiences, but applying RL in the data-efficient setting with limited access to online interactions is still challenging. A key to data-efficient RL is good value estimation, but current methods in this space fail to fully utilise the structure of the trajectory data gathered from the environment. In this paper, we treat the transition data of the MDP as a graph, and define a novel backup operator, Graph Backup, which exploits this graph structure for better value estimation. Compared to multi-step backup methods such as n-step Q-Learning and TD(λ), Graph Backup can perform counterfactual credit assignment and gives stable value estimates for a state regardless of which trajectory the state is sampled from. Our method, when combined with popular value-based methods, provides improved performance over one-step and multi-step methods on a suite of data-efficient RL benchmarks including MiniGrid, Minatar and Atari100K. We further analyse the reasons for this performance boost through a novel visualisation of the transition graphs of Atari games.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 31 May 2022.
Builds on
26 of 40 references.
References · 26
- 2017Neural Discrete Representation Learningcited by 4,712
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
- 2015Prioritized Experience Replaycited by 1,099
- 2020Mastering Atari with Discrete World Modelscited by 1,007
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
14 more
- 2019Model-Based Reinforcement Learning for Ataricited by 502
- 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
- 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
- 2017Noisy Networks for Explorationcited by 266
- 2020First return, then explorecited by 266
- 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
- 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
- 2017Learning Discrete Representations via Information Maximizing Self-Augmented Trainingcited by 149
- 2019When to use parametric models in reinforcement learning?cited by 118
- 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
- 2019Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Targetcited by 31
- 2020Expected Eligibility Tracescited by 27
- 2022Topological Experience Replaycited by 17
- 2017Convergent Tree Backup and Retrace with Function Approximationcited by 11
Built on by
Citing papers per year: 2022: 1.
1 paper.