# Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL

arXiv: https://arxiv.org/abs/2608.12063 · HTML: https://mlsanity.com/paper/2608.12063/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Placed in

- Offline RL and imitation learning from data / [Real-world robot offline RL](https://mlsanity.com/offlinerl/real-world-robot-offline-rl/)

## Builds on (most cited first, 21 of 21)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Efficient Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2302.02948/) (2023)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity](https://mlsanity.com/paper/1902.05605/) (2019)
- [Imitation Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2311.02198/) (2023)
- [Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo](https://mlsanity.com/paper/2212.00541/) (2022)
- [FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control](https://mlsanity.com/paper/2505.22642/) (2025)
- [ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learning](https://mlsanity.com/paper/2510.05070/) (2025)
- [Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation](https://mlsanity.com/paper/2409.20514/) (2024)
- [Sim-to-Real Learning for Humanoid Box Loco-Manipulation](https://mlsanity.com/paper/2310.03191/) (2023)
- [mjlab: A Lightweight Framework for GPU-Accelerated Robot Learning](https://mlsanity.com/paper/2601.22074/) (2026)
- [Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation](https://mlsanity.com/paper/2408.01258/) (2024)
- [Hierarchical Optimization-based Control for Whole-body Loco-manipulation of Heavy Objects](https://mlsanity.com/paper/2311.00112/) (2023)
- [Contact Optimization for Non-Prehensile Loco-Manipulation via Hierarchical Model Predictive Control](https://mlsanity.com/paper/2210.03442/) (2022)
- [Versatile Loco-Manipulation through Flexible Interlimb Coordination](https://mlsanity.com/paper/2506.07876/) (2025)
- [Sample-Efficient Learning of Nonprehensile Manipulation Policies via Physics-Based Informed State Distributions](https://mlsanity.com/paper/1810.10654/) (2018)
- [Generative Models From and For Sampling-Based MPC: A Bootstrapped Approach For Adaptive Contact-Rich Manipulation](https://mlsanity.com/paper/2510.14643/) (2025)
- [A Nonlinear MPC Framework for Loco-Manipulation of Quadrupedal Robots with Non-Negligible Manipulator Dynamics](https://mlsanity.com/paper/2507.22042/) (2025)
- [Whole-Body Inverse Dynamics MPC for Legged Loco-Manipulation](https://mlsanity.com/paper/2511.19709/) (2025)

## Abstract (arXiv)

Integrating locomotion and manipulation is essential for robot autonomy, but scaling standard Reinforcement Learning (RL) to complex tasks is severely bottlenecked by the slow, manual process of dense reward shaping. To bypass this limitation, we leverage Sample-based Model Predictive Control (SMPC) entirely in simulation as an automated, rapidly tunable expert to generate massive offline datasets. Because this data solves the fundamental exploration problem, we can train an off-policy RL agent using purely sparse task rewards, drastically reducing the time required to learn new skills and eliminating the need for manual tuning. Integrating this high-level agent with a low-level dynamic stability controller yields more optimal behaviors that strictly align with true task objectives, ultimately allowing the learned policies to surpass the original optimal control teacher. We validate the robustness of this sim-to-real framework by successfully deploying complex loco-manipulation skills across different morphologies, including an arm-equipped Spot quadruped and a G1 humanoid.