mlsanityData to 7 Oct 2026

Proper Value Equivalence

arXiv 2106.10316 ↗First posted 18 Jun 2021cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

One of the main challenges in model-based reinforcement learning (RL) is to decide which aspects of the environment should be modeled. The value-equivalence (VE) principle proposes a simple answer to this question: a model should capture the aspects of the environment that are relevant for value-based planning. Technically, VE distinguishes models based on a set of policies and a set of functions: a model is said to be VE to the environment if the Bellman operators it induces for the policies yield the correct result when applied to the functions. As the number of policies and functions increase, the set of VE models shrinks, eventually collapsing to a single point corresponding to a perfect model. A fundamental question underlying the VE principle is thus how to select the smallest sets of policies and functions that are sufficient for planning. In this paper we take an important step towards answering this question. We start by generalizing the concept of VE to order-k counterparts defined with respect to k applications of the Bellman operator. This leads to a family of VE classes that increase in size as k rightarrow ∞. In the limit, all functions become value functions, and we have a special instantiation of VE which we call proper VE or simply PVE. Unlike VE, the PVE class may contain multiple models even in the limit when all value functions are used. Crucially, all these models are sufficient for planning, meaning that they will yield an optimal policy despite the fact that they may ignore many aspects of the environment. We construct a loss function for learning PVE models and argue that popular algorithms such as MuZero can be understood as minimizing an upper bound for this loss. We leverage this connection to propose a modification to MuZero and show that it can lead to improved performance in practice.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 22 of its 52 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

22 of 52 references.

References · 22

  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  4. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  5. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  6. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  7. 2016Value Iteration Networkscited by 158
  8. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  9. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  10. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  11. 2017Value Prediction Networkcited by 117
  12. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
10 more
  1. 2016The Predictron: End-To-End Learning and Planningcited by 92
  2. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  3. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  4. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  5. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  6. 2018Combined Reinforcement Learning via Abstract Representationscited by 37
  7. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  8. 2019Learning Causal State Representations of Partially Observable Environmentscited by 30
  9. 2018Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulationcited by 20
  10. 2020Learning Discrete State Abstractions With Deep Variational Inferencecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 12; 2024: 3; 2025: 3; 2026: 6.

33 papers.

Sort:

By area:

  1. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 46
  2. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  3. 2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 24
  4. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  5. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  6. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  7. 2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9
  8. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  9. 2026How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Positioncited by 9
  10. 2023Policy Gradient Methods in the Presence of Symmetries and State Abstractionscited by 8
  11. 2023What model does MuZero learn?cited by 7
  12. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
21 more
  1. 2022Integrating Symmetry into Differentiable Planning with Steerable Convolutionscited by 6
  2. 2022Scaling up and Stabilizing Differentiable Planning with Implicit Differentiationcited by 6
  3. 2022On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learningcited by 6
  4. 2022A Look at Value-Based Decision-Time vs. Background Planning Methods Across Different Settingscited by 5
  5. 2023Bayesian Reinforcement Learning with Limited Cognitive Loadcited by 4
  6. 2026Imperfect World Models are Exploitablecited by 4
  7. 2023The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learningcited by 3
  8. 2022Between Rate-Distortion Theory & Value Equivalence in Model-Based Reinforcement Learningcited by 2
  9. 2023Minimal Value-Equivalent Partial Models for Scalable and Robust Planning in Lifelong Reinforcement Learningcited by 2
  10. 2024Provably Efficient Information-Directed Sampling Algorithms for Multi-Agent Reinforcement Learningcited by 2
  11. 2025Calibrated Value-Aware Model Learning with Probabilistic Environment Modelscited by 2
  12. 2026Operator-on-F complements value-equivalence: a planning-time diagnostic for latent world modelscited by 2
  13. 2026The Intervention Gap in Latent World Modelscited by 2
  14. 2023λ-models: Effective Decision-Aware Reinforcement Learning with Latent Modelscited by 1
  15. 2025Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimalitycited by 1
  16. 2026Inverting the Bellman Equation: From Q-Values to World Modelscited by 1
  17. 2023Goal-oriented inference of environment from redundant observationscited by 0
  18. 2023Pixel State Value Network for Combined Prediction and Planning in Interactive Environmentscited by 0
  19. 2024Meta-Gradient Search Control: A Method for Improving the Efficiency of Dyna-style Planningcited by 0
  20. 2025CORL: Reinforcement Learning of MILP Policies Solved via Branch and Boundcited by 0
  21. 2026What a World Model Represents Is Three Questionscited by 0