# Graph-based Task-specific Prediction Models for Interactions between Deformable and Rigid Objects

arXiv: https://arxiv.org/abs/2103.02932 · HTML: https://mlsanity.com/paper/2103.02932/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2021.

## Built on by (most cited first, 10 of 10)

- [Swarm Reinforcement Learning For Adaptive Mesh Refinement](https://mlsanity.com/paper/2304.00818/) (2023)
- [Towards Multi-Layered 3D Garments Animation](https://mlsanity.com/paper/2305.10418/) (2023)
- [Transformer with Implicit Edges for Particle-based Physics Simulation](https://mlsanity.com/paper/2207.10860/) (2022)
- [Grounding Graph Network Simulators using Physical Sensor Observations](https://mlsanity.com/paper/2302.11864/) (2023)
- [EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamics](https://mlsanity.com/paper/2209.08996/) (2022)
- [Context-aware Learned Mesh-based Simulation via Trajectory-Level Meta-Learning](https://mlsanity.com/paper/2511.05234/) (2025)
- [Context-Aware Composition of Agent Policies by Markov Decision Process Entity Embeddings and Agent Ensembles](https://mlsanity.com/paper/2308.14521/) (2023)
- [Latent Task-Specific Graph Network Simulators](https://mlsanity.com/paper/2311.05256/) (2023)
- [Dynamic Manipulation of Deformable Objects using Imitation Learning with Adaptation to Hardware Constraints](https://mlsanity.com/paper/2403.12685/) (2024)
- [Learning Goal-oriented Bimanual Dough Rolling Using Dynamic Heterogeneous Graph Based on Human Demonstration](https://mlsanity.com/paper/2410.22355/) (2024)

## Abstract (arXiv)

Capturing scene dynamics and predicting the future scene state is challenging but essential for robotic manipulation tasks, especially when the scene contains both rigid and deformable objects. In this work, we contribute a simulation environment and generate a novel dataset for task-specific manipulation, involving interactions between rigid objects and a deformable bag. The dataset incorporates a rich variety of scenarios including different object sizes, object numbers and manipulation actions. We approach dynamics learning by proposing an object-centric graph representation and two modules which are Active Prediction Module (APM) and Position Prediction Module (PPM) based on graph neural networks with an encode-process-decode architecture. At the inference stage, we build a two-stage model based on the learned modules for single time step prediction. We combine modules with different prediction horizons into a mixed-horizon model which addresses long-term prediction. In an ablation study, we show the benefits of the two-stage model for single time step prediction and the effectiveness of the mixed-horizon model for long-term prediction tasks. Supplementary material is available at https://github.com/wengzehang/deformable_rigid_interaction_prediction