mlsanityData to 14 Oct 2026

Chunk-wise Model-based Policy Optimization

arXiv 2610.14564 ↗First posted 10 Oct 2026cs.RO cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: https://song-wx.github.io/cmpo.github.io/

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 41 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

38 of 41 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  10. 2023Mastering Diverse Domains through World Modelscited by 1,253
  11. 2020Mastering Atari with Discrete World Modelscited by 1,007
  12. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
26 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  2. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  3. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  4. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  6. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  7. 2021Mastering Atari Games with Limited Datacited by 237
  8. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  9. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  10. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  11. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  12. 2023ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robotscited by 152
  13. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  14. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  15. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  16. 2025Reinforcement Learning with Action Chunkingcited by 107
  17. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63
  18. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  19. 2026Adaptive Action Chunking at Inference-time for Vision-Language-Action Modelscited by 28
  20. 2020PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inferencecited by 23
  21. 2025Decoupled Q-Chunkingcited by 20
  22. 2025Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulationcited by 17
  23. 2025Bootstrapped Model Predictive Controlcited by 16
  24. 2025TD-M(PC)2: Improving Temporal Difference MPC Through Policy Constraintcited by 14
  25. 2025Scalable Offline Model-Based RL with Action Chunkscited by 13
  26. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Action Chunking Proximal Policy Optimization with Feedback Correction
  2. 2026MARCH: Model-Assisted Reinforcement Learning for the Perceptive Control of Humanoids over Sparse Footholds
  3. 2026Efficiently Learning Robust Torque-based Locomotion Through Reinforcement with Model-Based Supervision
  4. 2026Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control
  5. 2025DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion
  6. 2026Inference Time Policy Optimization for Offline RL with Differentiable World Models
  7. 2026Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization
  8. 2026ChunkFlow: Towards Continuity-Consistent Chunked Policy Learning