mlsanityData to 7 Oct 2026

On the Limited Representational Power of Value Functions and its Links to Statistical (In)Efficiency

arXiv 2403.07136 ↗First posted 11 Mar 2024cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Identifying the trade-offs between model-based and model-free methods is a central question in reinforcement learning. Value-based methods offer substantial computational advantages and are sometimes just as statistically efficient as model-based methods. However, focusing on the core problem of policy evaluation, we show information about the transition dynamics may be impossible to represent in the space of value functions. We explore this through a series of case studies focused on structures that arises in many important problems. In several, there is no information loss and value-based methods are as statistically efficient as model based ones. In other closely-related examples, information loss is severe and value-based methods are severely outperformed. A deeper investigation points to the limitations of the representational power as the driver of the inefficiency, as opposed to failure in algorithm design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 34 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

20 of 34 references.

References · 20

  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  3. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  4. 2019Batch Policy Learning under Constraintscited by 242
  5. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  6. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  7. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 129
  8. 2023Synthetic Experience Replaycited by 100
  9. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  10. 2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
  11. 2018The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpointcited by 36
  12. 2020Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimensioncited by 36
8 more
  1. 2017Least-Squares Temporal Difference Learning for the Linear Quadratic Regulatorcited by 27
  2. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  3. 2021Infinite-Horizon Offline Reinforcement Learning with Linear Function Approximation: Curse of Dimensionality and Algorithmcited by 10
  4. 2023Policy Optimization for Personalized Interventions in Behavioral Healthcited by 9
  5. 2023Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspectivecited by 8
  6. 2023On Representation Complexity of Model-based and Model-free Reinforcement Learningcited by 6
  7. 2023Provably Efficient Reinforcement Learning via Surprise Boundcited by 4
  8. 2021Uniformly Conservative Exploration in Reinforcement Learningcited by 3

Built on by

Not cited yet.