mlsanityData to 7 Oct 2026

Model-Based Reinforcement Learning via Latent-Space Collocation

arXiv 2106.13229 ↗First posted 24 Jun 2021cs.LG cs.AI cs.ROcited by 27

Not on the map. Why?

Abstract (arXiv)

The ability to plan into the future while utilizing only raw high-dimensional observations, such as images, can provide autonomous agents with broad capabilities. Visual model-based reinforcement learning (RL) methods that plan future actions directly have shown impressive results on tasks that require only short-horizon reasoning, however, these methods struggle on temporally extended tasks. We argue that it is easier to solve long-horizon tasks by planning sequences of states rather than just actions, as the effects of actions greatly compound over time and are harder to optimize. To achieve this, we draw on the idea of collocation, which has shown good results on long-horizon tasks in optimal control literature, and adapt it to the image-based setting by utilizing learned latent state space models. The resulting latent collocation method (LatCo) optimizes trajectories of latent states, which improves over previously proposed shooting methods for visual model-based RL on tasks with sparse rewards and long-term goals. Videos and code at https://orybkin.github.io/latco/.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 36 of its 68 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

36 of 68 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  12. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
24 more
  1. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  2. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  3. 2018Stochastic Video Generation with a Learned Priorcited by 244
  4. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  5. 2018Semi-parametric Topological Memory for Navigationcited by 203
  6. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  7. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  8. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  9. 2019Planning with Goal-Conditioned Policiescited by 144
  10. 2016Sequential Neural Models with Stochastic Layerscited by 115
  11. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  12. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  13. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  14. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  15. 2017Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundarycited by 58
  16. 2018Learning Plannable Representations with Causal InfoGANcited by 56
  17. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  18. 2015From Pixels to Torques: Policy Learning with Deep Dynamical Modelscited by 44
  19. 2019Model Based Planning with Energy Based Modelscited by 32
  20. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  21. 2020Hallucinative Topological Memory for Zero-Shot Visual Planningcited by 25
  22. 2019Keyframing the Future: Keyframe Discovery for Visual Prediction and Planningcited by 20
  23. 2020Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planningcited by 19
  24. 2016Learning Quadrotor Dynamics Using Neural Network for Flight Controlcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 9; 2024: 3; 2025: 5; 2026: 2.

27 papers.

Sort:

By area:

  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  3. 2022Masked World Models for Visual Controlcited by 169
  4. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  5. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  6. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  7. 2023Planning Goals for Explorationcited by 28
  8. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  9. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matchingcited by 17
  10. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 17
  11. 2025Horizon Generalization in Reinforcement Learningcited by 15
  12. 2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 11
15 more
  1. 2023Self-Supervised Reinforcement Learning that Transfers using Random Featurescited by 10
  2. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8
  3. 2025Semantic World Modelscited by 8
  4. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  5. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  6. 2026Parallel Stochastic Gradient-Based Planning for World Modelscited by 6
  7. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
  8. 2022Just Round: Quantized Observation Spaces Enable Memory Efficient Learning of Dynamic Locomotioncited by 3
  9. 2023Learning Concept-Based Causal Transition and Symbolic Reasoning for Visual Planningcited by 2
  10. 2024Context-aware Diversity Enhancement for Neural Multi-Objective Combinatorial Optimizationcited by 2
  11. 2023Deep Generative Models for Decision-Making and Controlcited by 1
  12. 2026Grounding Generated Videos in Feasible Plans via World Modelscited by 1
  13. 2024Off-Policy Selection for Initiating Human-Centric Experimental Designcited by 0
  14. 2025MInCo: Mitigating Information Conflicts in Distracted Visual Model-based Reinforcement Learningcited by 0
  15. 2025Wavelet Predictive Representations for Non-Stationary Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond a single latent space: a dual-latent world model for long-horizon planning
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models
  3. 2026LaWM: Least Action World Models for Long-Horizon Physical Consistency from Visual Observations
  4. 2025Reinforcement Learning for Large Model: A Survey
  5. 2020Planning from Pixels using Inverse Dynamics Models
  6. 2021Temporal Predictive Coding For Model-Based Planning In Latent Space
  7. 2019Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
  8. 2026Beyond the Next Step: Variable-Length Latent World Models for Long-Horizon Planning