The Impact of Data Distribution on Q-learning with Function Approximation
Not on the map. Why?
Abstract (arXiv)
We study the interplay between the data distribution and Q-learning-based algorithms with function approximation. We provide a unified theoretical and empirical analysis as to how different properties of the data distribution influence the performance of Q-learning-based algorithms. We connect different lines of research, as well as validate and extend previous results. We start by reviewing theoretical bounds on the performance of approximate dynamic programming algorithms. We then introduce a novel four-state MDP specifically tailored to highlight the impact of the data distribution in the performance of Q-learning-based algorithms with function approximation, both online and offline. Finally, we experimentally assess the impact of the data distribution properties on the performance of two offline Q-learning-based algorithms under different environments. According to our results: (i) high entropy data distributions are well-suited for learning in an offline manner; and (ii) a certain degree of data diversity (data coverage) and data quality (closeness to optimal policy) are jointly desirable for offline learning.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 68 references that are in the map. First posted on arXiv 23 Nov 2021.
Builds on
28 of 68 references.
References · 28
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2021Convex optimizationcited by 2,479
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
- 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
- 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
- 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
- 2018Provably Efficient Maximum Entropy Explorationcited by 240
- 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
16 more
- 2019A Theoretical Analysis of Deep Q-Learningcited by 212
- 2019Efficient Exploration via State Marginal Matchingcited by 186
- 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
- 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
- 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
- 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
- 2020Batch Value-function Approximation with Only Realizabilitycited by 103
- 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
- 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
- 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
- 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
- 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
- 2021Breaking the Deadly Triad with a Target Networkcited by 47
- 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
- 2020A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Settingcited by 24
- 2018The Utility of Sparse Representations for Control in Reinforcement Learningcited by 23
Built on by
Citing papers per year: 2022: 1.
1 paper.