# A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation

arXiv: https://arxiv.org/abs/2111.05318 · HTML: https://mlsanity.com/paper/2111.05318/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 9 Nov 2021.

## Builds on (most cited first, 30 of 31)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Learning to Poke by Poking: Experiential Learning of Intuitive Physics](https://mlsanity.com/paper/1606.07419/) (2016)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Scalable Differentiable Physics for Learning and Control](https://mlsanity.com/paper/2007.02168/) (2020)
- [DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions](https://mlsanity.com/paper/1906.03853/) (2019)
- [Model-Based Inverse Reinforcement Learning from Visual Demonstrations](https://mlsanity.com/paper/2010.09034/) (2020)
- [Manipulation by Feel: Touch-Based Control with Deep Predictive Models](https://mlsanity.com/paper/1903.04128/) (2019)
- [The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation](https://mlsanity.com/paper/2002.09093/) (2020)
- [Learning Mixed-Integer Convex Optimization Strategies for Robot Planning and Control](https://mlsanity.com/paper/2004.03736/) (2020)
- [Contact Mode Guided Sampling-Based Planning for Quasistatic Dexterous Manipulation in 2D](https://mlsanity.com/paper/2011.01454/) (2020)
- [In-Hand Manipulation via Motion Cones](https://mlsanity.com/paper/1810.00219/) (2018)
- [Sampling-based Planning of In-Hand Manipulation with External Pushes](https://mlsanity.com/paper/1707.00318/) (2017)
- [Robust Execution of Contact-Rich Motion Plans by Hybrid Force-Velocity Control](https://mlsanity.com/paper/1903.02715/) (2019)
- [LVIS: Learning from Value Function Intervals for Contact-Aware Robot Controllers](https://mlsanity.com/paper/1809.05802/) (2018)
- [Accurate Vision-based Manipulation through Contact Reasoning](https://mlsanity.com/paper/1911.03112/) (2019)
- [A Convex Polynomial Force-Motion Model for Planar Sliding: Identification and Application](https://mlsanity.com/paper/1602.06056/) (2016)
- [Reactive Planar Manipulation with Convex Hybrid MPC](https://mlsanity.com/paper/1710.05724/) (2017)
- [Certified Grasping](https://mlsanity.com/paper/1909.03985/) (2019)
- [Omnipush: accurate, diverse, real-world dataset of pushing dynamics with RGB-D video](https://mlsanity.com/paper/1910.00618/) (2019)

## Built on by (most cited first, 2 of 2)

- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [Imitation Learning for Nonprehensile Manipulation through Self-Supervised Learning Considering Motion Speed](https://mlsanity.com/paper/2206.10289/) (2022)

## Abstract (arXiv)

Specifying tasks with videos is a powerful technique towards acquiring novel and general robot skills. However, reasoning over mechanics and dexterous interactions can make it challenging to scale learning contact-rich manipulation. In this work, we focus on the problem of visual non-prehensile planar manipulation: given a video of an object in planar motion, find contact-aware robot actions that reproduce the same object motion. We propose a novel architecture, Differentiable Learning for Manipulation (ours), that combines video decoding neural models with priors from contact mechanics by leveraging differentiable optimization and finite difference based simulation. Through extensive simulated experiments, we investigate the interplay between traditional model-based techniques and modern deep learning approaches. We find that our modular and fully differentiable architecture performs better than learning-only methods on unseen objects and motions. urlhttps://github.com/baceituno/dlm.