# Planning-Query-Guided Model Generation for Model-Based Deformable Object Manipulation

arXiv: https://arxiv.org/abs/2508.19199 · HTML: https://mlsanity.com/paper/2508.19199/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [MultiScale MeshGraphNets](https://mlsanity.com/paper/2210.00612/) (2022)
- [HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics](https://mlsanity.com/paper/2212.07242/) (2022)
- [Eagle: Large-Scale Learning of Turbulent Fluid Dynamics with Mesh Transformers](https://mlsanity.com/paper/2302.10803/) (2023)
- [AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation](https://mlsanity.com/paper/2407.07889/) (2024)
- [Efficient Learning of Mesh-Based Physical Simulation with BSMS-GNN](https://mlsanity.com/paper/2210.02573/) (2022)
- [Learning rigid dynamics with face interaction graph networks](https://mlsanity.com/paper/2212.03574/) (2022)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation](https://mlsanity.com/paper/2002.09093/) (2020)
- [Dynamic-Resolution Model Learning for Object Pile Manipulation](https://mlsanity.com/paper/2306.16700/) (2023)
- [Learning Controllable Adaptive Simulation for Multi-resolution Physics](https://mlsanity.com/paper/2305.01122/) (2023)
- [Multiscale graph neural networks with adaptive mesh refinement for accelerating mesh-based simulations](https://mlsanity.com/paper/2402.08863/) (2024)
- [Differentiable Particles for General-Purpose Deformable Object Manipulation](https://mlsanity.com/paper/2405.01044/) (2024)
- [Linear vs Nonlinear MPC for Trajectory Tracking Applied to Rotary Wing Micro Aerial Vehicles](https://mlsanity.com/paper/1611.09240/) (2016)
- [GFN: A graph feedforward network for resolution-invariant reduced operator learning in multifidelity applications](https://mlsanity.com/paper/2406.03569/) (2024)
- [Neural Field Dynamics Model for Granular Object Piles Manipulation](https://mlsanity.com/paper/2311.00802/) (2023)
- [Planning with Selective Physics-based Simulation for Manipulation Among Movable Objects](https://mlsanity.com/paper/2003.06743/) (2020)
- [Learning to Simulate Tree-Branch Dynamics for Manipulation](https://mlsanity.com/paper/2306.03410/) (2023)
- [Towards Robotic Tree Manipulation: Leveraging Graph Representations](https://mlsanity.com/paper/2311.07479/) (2023)
- [Online state vector reduction during model predictive control with gradient-based trajectory optimisation](https://mlsanity.com/paper/2408.11665/) (2024)

## Abstract (arXiv)

Efficient planning in high-dimensional spaces, such as those involving deformable objects, requires computationally tractable yet sufficiently expressive dynamics models. This paper introduces a method that automatically generates task-specific, spatially adaptive dynamics models by learning which regions of the object require high-resolution modeling to achieve good task performance for a given planning query. Task performance depends on the complex interplay between the dynamics model, world dynamics, control, and task requirements. Our proposed diffusion-based model generator predicts per-region model resolutions based on start and goal pointclouds that define the planning query. To efficiently collect the data for learning this mapping, a two-stage process optimizes resolution using predictive dynamics as a prior before directly optimizing using closed-loop performance. On a tree-manipulation task, our method doubles planning speed with only a small decrease in task performance over using a full-resolution model. This approach informs a path towards using previous planning and control data to generate computationally efficient yet sufficiently expressive dynamics models for new tasks.