mlsanityData to 7 Oct 2026

The Predictron: End-To-End Learning and Planning

arXiv 1612.08810 ↗First posted 28 Dec 2016cs.LG cs.AI cs.NEcited by 92

Not on the map. Why?

Abstract (arXiv)

One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. The predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple "imagined" planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. The predictron yielded significantly more accurate predictions than conventional deep neural network architectures.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 1; 2021: 18; 2022: 20; 2023: 18; 2024: 13; 2025: 7; 2026: 13.

92 papers.

Sort:

By area:

  1. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  2. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  3. 2021Towards Causal Representation Learningcited by 207
  4. 2024Video as the New Language for Real-World Decision Makingcited by 86
  5. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  6. 2021Learning Task Informed Abstractionscited by 58
  7. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  8. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  9. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  10. 2021Proper Value Equivalencecited by 33
  11. 2021A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learningcited by 27
  12. 2021Differentiable Spatial Planning using Transformerscited by 25
38 more
  1. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  2. 2022Learning to Predict Graphs with Fused Gromov-Wasserstein Barycenterscited by 22
  3. 2018Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learningcited by 21
  4. 2021Neural Algorithmic Reasoners are Implicit Plannerscited by 21
  5. 2025Learning Transformer-based World Models with Contrastive Predictive Codingcited by 20
  6. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  7. 2023Simplified Temporal Consistency Reinforcement Learningcited by 17
  8. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  9. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  10. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  11. 2021High-Accuracy Model-Based Reinforcement Learning, a Surveycited by 13
  12. 2024MuDreamer: Learning Predictive World Models without Reconstructioncited by 13
  13. 2022Goal-Space Planning with Subgoal Modelscited by 12
  14. 2021Neuro-algorithmic Policies enable Fast Combinatorial Generalizationcited by 11
  15. 2021Learning cortical representations through perturbed and adversarial dreamingcited by 10
  16. 2022Generative Planning for Temporally Coordinated Exploration in Reinforcement Learningcited by 10
  17. 2021Visualizing MuZero Modelscited by 9
  18. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  19. 2022Operator Splitting Value Iterationcited by 8
  20. 2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
  21. 2021Self-Consistent Models and Valuescited by 7
  22. 2023What model does MuZero learn?cited by 7
  23. 2025World Modeling with Probabilistic Structure Integrationcited by 7
  24. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  25. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
  26. 2022On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learningcited by 6
  27. 2023On Representation Complexity of Model-based and Model-free Reinforcement Learningcited by 6
  28. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  29. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 5
  30. 2018Learning the Reward Function for a Misspecified Modelcited by 4
  31. 2021Combining Off and On-Policy Training in Model-Based Reinforcement Learningcited by 4
  32. 2023Bayesian Reinforcement Learning with Limited Cognitive Loadcited by 4
  33. 2023Co-Learning Empirical Games and World Modelscited by 4
  34. 2024FACTS: A Factored State-Space Framework For World Modellingcited by 4
  35. 2022A model-based approach to meta-Reinforcement Learning: Transformers and tree searchcited by 3
  36. 2023Decision-Aware Actor-Critic with Function Approximation and Theoretical Guaranteescited by 3
  37. 2024OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequencescited by 3
  38. 2024Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planningcited by 3

50 most cited of 92 All 92 (JSON)