# Learning Contact Dynamics through Touching: Action-conditional Graph Neural Networks for Robotic Peg Insertion

arXiv: https://arxiv.org/abs/2509.12151 · HTML: https://mlsanity.com/paper/2509.12151/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [Clifford Neural Layers for PDE Modeling](https://mlsanity.com/paper/2209.04934/) (2022)
- [Propagation Networks for Model-Based Control Under Partial Observation](https://mlsanity.com/paper/1809.11169/) (2018)
- [PyG 2.0: Scalable Learning on Real World Graphs](https://mlsanity.com/paper/2507.16991/) (2025)
- [Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo](https://mlsanity.com/paper/2212.00541/) (2022)
- [Learning rigid dynamics with face interaction graph networks](https://mlsanity.com/paper/2212.03574/) (2022)
- [MBRL-Lib: A Modular Library for Model-based Reinforcement Learning](https://mlsanity.com/paper/2104.10159/) (2021)
- [Integrating Physics and Topology in Neural Networks for Learning Rigid Body Dynamics](https://mlsanity.com/paper/2411.11467/) (2024)
- [Efficient Online Learning of Contact Force Models for Connector Insertion](https://mlsanity.com/paper/2312.09190/) (2023)
- [Tactile Probabilistic Contact Dynamics Estimation of Unknown Objects](https://mlsanity.com/paper/2409.17470/) (2024)
- [Learning the Contact Manifold for Accurate Pose Estimation During Peg-in-Hole Insertion of Complex Geometries](https://mlsanity.com/paper/2505.19215/) (2025)

## Abstract (arXiv)

We present a learnable physics-based model that predicts motion of the robot end effector and reaction force-torque in contact-rich manipulation. The model represents the end effector and the environment as interacting meshes in a graph structure, and conditions its prediction explicitly on the applied control input. It predicts object-level pose update directly, while the reaction torque emerges from a per-vertex force field. Training is self-supervised using only joint encoder and force-torque data while the robot is randomly touching the environment without task context. In simulation, our model transfers to peg insertion with unseen concave geometry, where an MPC agent using it reaches up to 98% success rate, and after fine-tuning on self-collected data matches an agent planning with the ground truth dynamics at the tightest 1 mm clearance. In the real world, it outperforms the system-identified MuJoCo model by 45% in position and by 74% and 63% in force and torque error.