# Dynamic-Resolution Model Learning for Object Pile Manipulation

arXiv: https://arxiv.org/abs/2306.16700 · HTML: https://mlsanity.com/paper/2306.16700/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 75 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 30 of 45)

- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [3D Neural Scene Representations for Visuomotor Control](https://mlsanity.com/paper/2107.04004/) (2021)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [Unsupervised Learning of Object Structure and Dynamics from Videos](https://mlsanity.com/paper/1906.07889/) (2019)
- [Flexible Neural Representation for Physics Prediction](https://mlsanity.com/paper/1806.08047/) (2018)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2009.05085/) (2020)
- [Propagation Networks for Model-Based Control Under Partial Observation](https://mlsanity.com/paper/1809.11169/) (2018)
- [Learning Multi-Object Dynamics with Compositional Neural Radiance Fields](https://mlsanity.com/paper/2202.11855/) (2022)
- [Learning Visible Connectivity Dynamics for Cloth Smoothing](https://mlsanity.com/paper/2105.10389/) (2021)
- [Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Models](https://mlsanity.com/paper/2206.10787/) (2022)
- [Learning Long-term Visual Dynamics with Region Proposal Interaction Networks](https://mlsanity.com/paper/2008.02265/) (2020)
- [Compositional Video Prediction](https://mlsanity.com/paper/1908.08522/) (2019)
- [Planning with Learned Object Importance in Large Problem Instances using Graph Neural Networks](https://mlsanity.com/paper/2009.05613/) (2020)

## Built on by (most cited first, 22 of 22)

- [Causal World Modeling for Robot Control](https://mlsanity.com/paper/2601.21998/) (2026)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation](https://mlsanity.com/paper/2407.07889/) (2024)
- [Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling](https://mlsanity.com/paper/2410.18912/) (2024)
- [NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos](https://mlsanity.com/paper/2510.08568/) (2025)
- [Native Video-Action Pretraining for Generalizable Robot Control](https://mlsanity.com/paper/2607.08639/) (2026)
- [D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement](https://mlsanity.com/paper/2309.16118/) (2023)
- [GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy](https://mlsanity.com/paper/2410.17488/) (2024)
- [Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos](https://mlsanity.com/paper/2506.15680/) (2025)
- [Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation](https://mlsanity.com/paper/2412.12089/) (2024)
- [RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing](https://mlsanity.com/paper/2407.01418/) (2024)
- [PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesis](https://mlsanity.com/paper/2510.21447/) (2025)
- [Learning Coordinated Bimanual Manipulation Policies using State Diffusion and Inverse Dynamics Models](https://mlsanity.com/paper/2503.23271/) (2025)
- [Neural Field Dynamics Model for Granular Object Piles Manipulation](https://mlsanity.com/paper/2311.00802/) (2023)
- [Localized Graph-Based Neural Dynamics Models for Terrain Manipulation](https://mlsanity.com/paper/2503.23270/) (2025)
- [Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control](https://mlsanity.com/paper/2505.16249/) (2025)
- [Parallel Differentiable Reachability for Learning and Planning with Certified Neural Dynamics and Controllers](https://mlsanity.com/paper/2605.25346/) (2026)
- [PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics](https://mlsanity.com/paper/2607.20653/) (2026)
- [Planning-Query-Guided Model Generation for Model-Based Deformable Object Manipulation](https://mlsanity.com/paper/2508.19199/) (2025)
- [DexKnot: Generalizable Visuomotor Policy Learning for Dexterous Bag-Knotting Manipulation](https://mlsanity.com/paper/2603.07136/) (2026)
- [IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction](https://mlsanity.com/paper/2603.29315/) (2026)
- [TRACE: A spatiotemporal contact memory graph network simulator for granular dynamics](https://mlsanity.com/paper/2609.02991/) (2026)

## Abstract (arXiv)

Dynamics models learned from visual observations have shown to be effective in various robotic manipulation tasks. One of the key questions for learning such dynamics models is what scene representation to use. Prior works typically assume representation at a fixed dimension or resolution, which may be inefficient for simple tasks and ineffective for more complicated tasks. In this work, we investigate how to learn dynamic and adaptive representations at different levels of abstraction to achieve the optimal trade-off between efficiency and effectiveness. Specifically, we construct dynamic-resolution particle representations of the environment and learn a unified dynamics model using graph neural networks (GNNs) that allows continuous selection of the abstraction level. During test time, the agent can adaptively determine the optimal resolution at each model-predictive control (MPC) step. We evaluate our method in object pile manipulation, a task we commonly encounter in cooking, agriculture, manufacturing, and pharmaceutical applications. Through comprehensive evaluations both in the simulation and the real world, we show that our method achieves significantly better performance than state-of-the-art fixed-resolution baselines at the gathering, sorting, and redistribution of granular object piles made with various instances like coffee beans, almonds, corn, etc.