# The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation

arXiv: https://arxiv.org/abs/2002.09093 · HTML: https://mlsanity.com/paper/2002.09093/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 25 of 25)

- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [3D Neural Scene Representations for Visuomotor Control](https://mlsanity.com/paper/2107.04004/) (2021)
- [Offline Reinforcement Learning from Images with Latent Space Models](https://mlsanity.com/paper/2012.11547/) (2020)
- [Learning Multi-Object Dynamics with Compositional Neural Radiance Fields](https://mlsanity.com/paper/2202.11855/) (2022)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [Noether Networks: Meta-Learning Useful Conserved Quantities](https://mlsanity.com/paper/2112.03321/) (2021)
- [Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data](https://mlsanity.com/paper/2306.03346/) (2023)
- [Dynamic-Resolution Model Learning for Object Pile Manipulation](https://mlsanity.com/paper/2306.16700/) (2023)
- [Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching](https://mlsanity.com/paper/2306.14079/) (2023)
- [Model-Based Control with Sparse Neural Dynamics](https://mlsanity.com/paper/2312.12791/) (2023)
- [Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding](https://mlsanity.com/paper/2211.14648/) (2022)
- [Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks](https://mlsanity.com/paper/2109.10312/) (2021)
- [3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes](https://mlsanity.com/paper/2304.11470/) (2023)
- [Robotic Table Wiping via Reinforcement Learning and Whole-body Trajectory Optimization](https://mlsanity.com/paper/2210.10865/) (2022)
- [Transporters with Visual Foresight for Solving Unseen Rearrangement Tasks](https://mlsanity.com/paper/2202.10765/) (2022)
- [Neural Field Dynamics Model for Granular Object Piles Manipulation](https://mlsanity.com/paper/2311.00802/) (2023)
- [Learning Sequential Acquisition Policies for Robot-Assisted Feeding](https://mlsanity.com/paper/2309.05197/) (2023)
- [An Efficient Image-to-Image Translation HourGlass-based Architecture for Object Pushing Policy Learning](https://mlsanity.com/paper/2108.01034/) (2021)
- [A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation](https://mlsanity.com/paper/2111.05318/) (2021)
- [Localized Graph-Based Neural Dynamics Models for Terrain Manipulation](https://mlsanity.com/paper/2503.23270/) (2025)
- [Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control](https://mlsanity.com/paper/2505.16249/) (2025)
- [Few-shot Scooping Under Domain Shift via Simulated Maximal Deployment Gaps](https://mlsanity.com/paper/2408.02949/) (2024)
- [Planning-Query-Guided Model Generation for Model-Based Deformable Object Manipulation](https://mlsanity.com/paper/2508.19199/) (2025)
- [IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction](https://mlsanity.com/paper/2603.29315/) (2026)

## Abstract (arXiv)

In this paper, we tackle the problem of pushing piles of small objects into a desired target set using visual feedback. Unlike conventional single-object manipulation pipelines, which estimate the state of the system parametrized by pose, the underlying physical state of this system is difficult to observe from images. Thus, we take the approach of reasoning directly in the space of images, and acquire the dynamics of visual measurements in order to synthesize a visual-feedback policy. We present a simple controller using an image-space Lyapunov function, and evaluate the closed-loop performance using three different class of models for image prediction: deep-learning-based models for image-to-image translation, an object-centric model obtained from treating each pixel as a particle, and a switched-linear system where an action-dependent linear map is used. Through results in simulation and experiment, we show that for this task, a linear model works surprisingly well -- achieving better prediction error, downstream task performance, and generalization to new environments than the deep models we trained on the same amount of data. We believe these results provide an interesting example in the spectrum of models that are most useful for vision-based feedback in manipulation, considering both the quality of visual prediction, as well as compatibility with rigorous methods for control design and analysis. Project site: https://sites.google.com/view/linear-visual-foresight/home