# Dynamic Non-Prehensile Object Transport via Model-Predictive Reinforcement Learning

arXiv: https://arxiv.org/abs/2412.00086 · HTML: https://mlsanity.com/paper/2412.00086/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 29 of 29)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [Bellman-consistent Pessimism for Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06926/) (2021)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [ALOHA Unleashed: A Simple Recipe for Robot Dexterity](https://mlsanity.com/paper/2410.13126/) (2024)
- [Model-Based Offline Planning](https://mlsanity.com/paper/2008.05556/) (2020)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Human-in-the-Loop Imitation Learning using Remote Teleoperation](https://mlsanity.com/paper/2012.06733/) (2020)
- [Adversarially Trained Actor Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2202.02446/) (2022)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Learning Off-Policy with Online Planning](https://mlsanity.com/paper/2008.10066/) (2020)
- [STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation](https://mlsanity.com/paper/2104.13542/) (2021)
- [A Unified MPC Framework for Whole-Body Dynamic Locomotion and Manipulation](https://mlsanity.com/paper/2103.00946/) (2021)
- [Blending MPC & Value Function Approximation for Efficient Reinforcement Learning](https://mlsanity.com/paper/2012.05909/) (2020)
- [Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.13842/) (2021)
- [Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size](https://mlsanity.com/paper/2211.11092/) (2022)
- [RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstration](https://mlsanity.com/paper/1906.07372/) (2019)
- [MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations](https://mlsanity.com/paper/2303.17156/) (2023)
- [Internal Model from Observations for Reward Shaping](https://mlsanity.com/paper/1806.01267/) (2018)
- [VP-STO: Via-point-based Stochastic Trajectory Optimization for Reactive Robot Behavior](https://mlsanity.com/paper/2210.04067/) (2022)
- [Keep it Upright: Model Predictive Control for Nonprehensile Object Transportation with Obstacle Avoidance on a Mobile Manipulator](https://mlsanity.com/paper/2305.17484/) (2023)

## Built on by (most cited first, 2 of 2)

- [Model Predictive Adversarial Imitation Learning for Planning from Observation](https://mlsanity.com/paper/2507.21533/) (2025)
- [Grasp-MPC: Closed-Loop Visual Grasping via Value-Guided Model Predictive Control](https://mlsanity.com/paper/2509.06201/) (2025)

## Abstract (arXiv)

We investigate the problem of teaching a robot manipulator to perform dynamic non-prehensile object transport, also known as the `robot waiter' task, from a limited set of real-world demonstrations. We propose an approach that combines batch reinforcement learning (RL) with model-predictive control (MPC) by pretraining an ensemble of value functions from demonstration data, and utilizing them online within an uncertainty-aware MPC scheme to ensure robustness to limited data coverage. Our approach is straightforward to integrate with off-the-shelf MPC frameworks and enables learning solely from task space demonstrations with sparsely labeled transitions, while leveraging MPC to ensure smooth joint space motions and constraint satisfaction. We validate the proposed approach through extensive simulated and real-world experiments on a Franka Panda robot performing the robot waiter task and demonstrate robust deployment of value functions learned from 50-100 demonstrations. Furthermore, our approach enables generalization to novel objects not seen during training and can improve upon suboptimal demonstrations. We believe that such a framework can reduce the burden of providing extensive demonstrations and facilitate rapid training of robot manipulators to perform non-prehensile manipulation tasks. Project videos and supplementary material can be found at: https://sites.google.com/view/cvmpc.