# STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation

arXiv: https://arxiv.org/abs/2104.13542 · HTML: https://mlsanity.com/paper/2104.13542/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2021.

## Built on by (most cited first, 30 of 47)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion](https://mlsanity.com/paper/2209.03855/) (2022)
- [LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning](https://mlsanity.com/paper/2309.06440/) (2023)
- [Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning](https://mlsanity.com/paper/2407.03162/) (2024)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [Motion Policy Networks](https://mlsanity.com/paper/2210.12209/) (2022)
- [EDMP: Ensemble-of-costs-guided Diffusion for Motion Planning](https://mlsanity.com/paper/2309.11414/) (2023)
- [VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation](https://mlsanity.com/paper/2407.09829/) (2024)
- [Learning Implicit Priors for Motion Optimization](https://mlsanity.com/paper/2204.05369/) (2022)
- [Real World Offline Reinforcement Learning with Realistic Data Source](https://mlsanity.com/paper/2210.06479/) (2022)
- [Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning](https://mlsanity.com/paper/2105.04962/) (2021)
- [A Memory System of a Robot Cognitive Architecture and its Implementation in ArmarX](https://mlsanity.com/paper/2206.02241/) (2022)
- [Learning to Optimize in Model Predictive Control](https://mlsanity.com/paper/2212.02603/) (2022)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control](https://mlsanity.com/paper/2210.11668/) (2022)
- [Learning Sampling Distributions for Model Predictive Control](https://mlsanity.com/paper/2212.02587/) (2022)
- [Model Predictive Control for Fluid Human-to-Robot Handovers](https://mlsanity.com/paper/2204.00134/) (2022)
- [Deep Reactive Policy: Learning Reactive Manipulator Motion Planning for Dynamic Environments](https://mlsanity.com/paper/2509.06953/) (2025)
- [Hierarchical Policy Blending as Inference for Reactive Robot Control](https://mlsanity.com/paper/2210.07890/) (2022)
- [Constrained Stein Variational Trajectory Optimization](https://mlsanity.com/paper/2308.12110/) (2023)
- [USA-Net: Unified Semantic and Affordance Representations for Robot Memory](https://mlsanity.com/paper/2304.12164/) (2023)
- [Hierarchical Policy Blending As Optimal Transport](https://mlsanity.com/paper/2212.01938/) (2022)
- [Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generation](https://mlsanity.com/paper/2504.05225/) (2025)
- [Bayesian Optimisation for Robust Model Predictive Control under Model Parameter Uncertainty](https://mlsanity.com/paper/2203.00551/) (2022)
- [Neural Motion Fields: Encoding Grasp Trajectories as Implicit Value Functions](https://mlsanity.com/paper/2206.14854/) (2022)
- [Generative Models From and For Sampling-Based MPC: A Bootstrapped Approach For Adaptive Contact-Rich Manipulation](https://mlsanity.com/paper/2510.14643/) (2025)
- [Differentiable and Learnable Robot Models](https://mlsanity.com/paper/2202.11217/) (2022)
- [Bi-Level Optimization Augmented with Conditional Variational Autoencoder for Autonomous Driving in Dense Traffic](https://mlsanity.com/paper/2212.02224/) (2022)
- [End-to-End Learning of Behavioural Inputs for Autonomous Driving in Dense Traffic](https://mlsanity.com/paper/2310.14766/) (2023)

## Abstract (arXiv)

Sampling-based model-predictive control (MPC) is a promising tool for feedback control of robots with complex, non-smooth dynamics, and cost functions. However, the computationally demanding nature of sampling-based MPC algorithms has been a key bottleneck in their application to high-dimensional robotic manipulation problems in the real world. Previous methods have addressed this issue by running MPC in the task space while relying on a low-level operational space controller for joint control. However, by not using the joint space of the robot in the MPC formulation, existing methods cannot directly account for non-task space related constraints such as avoiding joint limits, singular configurations, and link collisions. In this paper, we develop a system for fast, joint space sampling-based MPC for manipulators that is efficiently parallelized using GPUs. Our approach can handle task and joint space constraints while taking less than 8ms~(125Hz) to compute the next control command. Further, our method can tightly integrate perception into the control problem by utilizing learned cost functions from raw sensor data. We validate our approach by deploying it on a Franka Panda robot for a variety of dynamic manipulation tasks. We study the effect of different cost formulations and MPC parameters on the synthesized behavior and provide key insights that pave the way for the application of sampling-based MPC for manipulators in a principled manner. We also provide highly optimized, open-source code to be used by the wider robot learning and control community. Videos of experiments can be found at: https://sites.google.com/view/manipulation-mpc