mlsanityData to 7 Oct 2026

Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

arXiv 1811.01848 ↗First posted 5 Nov 2018cs.LG cs.AI cs.ROcited by 104

Not on the map. Why?

Abstract (arXiv)

We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning. Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions. Finally, we also demonstrate how trajectory optimization can be used to perform temporally coordinated exploration in conjunction with estimating uncertainty in value function approximation. This exploration is critical for fast and stable learning of the value function. Combining these components enable solutions to complex simulated control tasks, like humanoid locomotion and dexterous in-hand manipulation, in the equivalent of a few minutes of experience in the real world.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 28; 2022: 20; 2023: 13; 2024: 15; 2025: 18; 2026: 8.

104 papers.

Sort:

By area:

  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  2. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  4. 2022Tutorial on amortized optimizationcited by 83
  5. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  6. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  7. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
  8. 2022Skill-based Model-based Reinforcement Learningcited by 50
  9. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  10. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  11. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  12. 2022Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Graspscited by 27
38 more
  1. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  2. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  3. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  4. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLcited by 24
  5. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  6. 2023ALAN: Autonomously Exploring Robotic Agents in the Real Worldcited by 23
  7. 2021Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrianscited by 20
  8. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 20
  9. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 20
  10. 2022BayesFormer: Transformer with Uncertainty Estimationcited by 19
  11. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  12. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  13. 2022Cross-Domain Transfer via Semantic Skill Imitationcited by 17
  14. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 15
  15. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  16. 2025Horizon Generalization in Reinforcement Learningcited by 15
  17. 2021Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Explorationcited by 14
  18. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  19. 2025Bootstrapped Model Predictive Controlcited by 13
  20. 2021Optimal Cost Design for Model Predictive Controlcited by 12
  21. 2022Generative Planning for Temporally Coordinated Exploration in Reinforcement Learningcited by 10
  22. 2022Learning Sampling Distributions for Model Predictive Controlcited by 10
  23. 2022Learning to Optimize in Model Predictive Controlcited by 10
  24. 2023Efficient Dynamics Modeling in Interactive Environments with Koopman Theorycited by 10
  25. 2021Reinforcement Learning of the Prediction Horizon in Model Predictive Controlcited by 9
  26. 2021LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Taskscited by 9
  27. 2022S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learningcited by 9
  28. 2025BodyGen: Advancing Towards Efficient Embodiment Co-Designcited by 9
  29. 2021The Value of Planning for Infinite-Horizon Model Predictive Controlcited by 8
  30. 2021Continuous Control With Ensemble Deep Deterministic Policy Gradientscited by 8
  31. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8
  32. 2021Optimization of the Model Predictive Control Meta-Parameters Through Reinforcement Learningcited by 7
  33. 2023What model does MuZero learn?cited by 7
  34. 2023Deep Model Predictive Optimizationcited by 7
  35. 2024Model-based Reinforcement Learning for Parameterized Action Spacescited by 7
  36. 2021Safe Model-based Off-policy Reinforcement Learning for Eco-Driving in Connected and Automated Hybrid Electric Vehiclescited by 6
  37. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  38. 2022Learning Forward Dynamics Model and Informed Trajectory Sampler for Safe Quadruped Navigationcited by 6

50 most cited of 104 All 104 (JSON)