mlsanityData to 7 Oct 2026

Value Prediction Network

arXiv 1707.03497 ↗First posted 11 Jul 2017cs.AI cs.LGcited by 117

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

This paper proposes a novel deep reinforcement learning (RL) architecture, called Value Prediction Network (VPN), which integrates model-free and model-based RL methods into a single neural network. In contrast to typical model-based RL methods, VPN learns a dynamics model whose abstract states are trained to make option-conditional predictions of future values (discounted sum of rewards) rather than of future observations. Our experimental results show that VPN has several advantages over both model-free and model-based baselines in a stochastic environment where careful planning is required but building an accurate observation-prediction model is difficult. Furthermore, VPN outperforms Deep Q-Network (DQN) on several Atari games even with short-lookahead planning, demonstrating its potential as a new way of learning a good state representation.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 0 of its 40 references that are in the map. First posted on arXiv 11 Jul 2017.

Builds on

0 of 40 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 2; 2020: 3; 2021: 32; 2022: 26; 2023: 24; 2024: 15; 2025: 4; 2026: 9.

117 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  4. 2020Mastering Atari with Discrete World Modelscited by 990
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  8. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  9. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  10. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  11. 2021Learning to drive from a world on railscited by 114
  12. 2025Reinforcement Learning with Action Chunkingcited by 97
38 more
  1. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  2. 2022A Survey on Model-based Reinforcement Learningcited by 65
  3. 2021Learning Task Informed Abstractionscited by 58
  4. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  5. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  6. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  7. 2021Vector Quantized Models for Planningcited by 47
  8. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  9. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  10. 2021Differentiable SLAM-net: Learning Particle SLAM for Visual Navigationcited by 34
  11. 2021Proper Value Equivalencecited by 33
  12. 2023Adaptive Online Replanning with Diffusion Modelscited by 33
  13. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  14. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  15. 2023Rethinking Closed-loop Training for Autonomous Drivingcited by 29
  16. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  17. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  18. 2021Differentiable Spatial Planning using Transformerscited by 25
  19. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  20. 2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
  21. 2021Model-Based Opponent Modelingcited by 21
  22. 2021Neural Algorithmic Reasoners are Implicit Plannerscited by 21
  23. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  24. 2024Offline Model-Based Optimization via Policy-Guided Gradient Searchcited by 19
  25. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  26. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  27. 2023Simplified Temporal Consistency Reinforcement Learningcited by 17
  28. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  29. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  30. 2021High-Accuracy Model-Based Reinforcement Learning, a Surveycited by 13
  31. 2023Thinker: Learning to Plan and Actcited by 13
  32. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
  33. 2024Prioritized Generative Replaycited by 13
  34. 2022Goal-Space Planning with Subgoal Modelscited by 12
  35. 2021Neuro-algorithmic Policies enable Fast Combinatorial Generalizationcited by 11
  36. 2021Temporally Abstract Partial Modelscited by 11
  37. 2021Hierarchies of Planning and Reinforcement Learning for Robot Navigationcited by 11
  38. 2023RTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernelscited by 11

50 most cited of 117 All 117 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
  2. 2018Reward learning from human preferences and demonstrations in Atari
  3. 2025VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning
  4. 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rollouts
  5. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimization
  6. 2025Outcome-based Reinforcement Learning to Predict the Future
  7. 2022Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning
  8. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training