mlsanityData to 7 Oct 2026

Self-Consistent Models and Values

arXiv 2110.12840 ↗First posted 25 Oct 2021cs.LG cs.AI stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. In particular, models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions. In this work, we investigate a way of augmenting model-based RL, by additionally encouraging a learned model and value function to be jointly self-consistent. Our approach differs from classic planning methods such as Dyna, which only update values to be consistent with the model. We propose multiple self-consistency updates, evaluate these in both tabular and function approximation settings, and find that, with appropriate choices, self-consistency helps both policy evaluation and control.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 34 of its 68 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

34 of 68 references.

References · 34

  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2018World Modelscited by 1,444
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2020Mastering Atari with Discrete World Modelscited by 990
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2019Model-Based Reinforcement Learning for Ataricited by 495
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  10. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  11. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  12. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
22 more
  1. 2016Value Iteration Networkscited by 158
  2. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  3. 2019When to use parametric models in reinforcement learning?cited by 118
  4. 2017Value Prediction Networkcited by 117
  5. 2016The Predictron: End-To-End Learning and Planningcited by 92
  6. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  7. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  8. 2017Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigationcited by 74
  9. 2019Shaping Belief States with Generative Environment Models for RLcited by 65
  10. 2020On the role of planning in model-based deep reinforcement learningcited by 57
  11. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  12. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  13. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  14. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  15. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  16. 2020Causally Correct Partial Models for Reinforcement Learningcited by 24
  17. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  18. 2019Deep Residual Reinforcement Learningcited by 21
  19. 2020Forethought and Hindsight in Credit Assignmentcited by 16
  20. 2018The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spacescited by 15
  21. 2020Frequency-based Search-control in Dynacited by 7
  22. 2020Value-driven Hindsight Modellingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 2; 2024: 1; 2025: 0; 2026: 2.

7 papers.

  1. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  2. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  3. 2023TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matchingcited by 10
  4. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  5. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
  6. 2026Scaling Tasks, Not Samples: Mastering Humanoid Control through Multi-Task Model-Based Reinforcement Learningcited by 0
  7. 2026DT2: Decision-Targeted Digital Twinscited by 0