# Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimization

arXiv: https://arxiv.org/abs/2502.20382 · HTML: https://mlsanity.com/paper/2502.20382/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 93 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 30 of 74)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [RoboNet: Large-Scale Multi-Robot Learning](https://mlsanity.com/paper/1910.11215/) (2019)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning](https://mlsanity.com/paper/2406.08858/) (2024)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)

## Built on by (most cited first, 21 of 21)

- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [DreamGen: Unlocking Generalization in Robot Learning through Video World Models](https://mlsanity.com/paper/2505.12705/) (2025)
- [OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction](https://mlsanity.com/paper/2509.26633/) (2025)
- [Rethinking Video Generation Model for the Embodied World](https://mlsanity.com/paper/2601.15282/) (2026)
- [SPIDER: Scalable Physics-Informed Dexterous Retargeting](https://mlsanity.com/paper/2511.09484/) (2025)
- [A Survey on Imitation Learning for Contact-Rich Tasks in Robotics](https://mlsanity.com/paper/2506.13498/) (2025)
- [Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixels](https://mlsanity.com/paper/2503.22634/) (2025)
- [Do as I Do: Dexterous Manipulation Data from Everyday Human Videos](https://mlsanity.com/paper/2606.19333/) (2026)
- [DemoHLM: From One Demonstration to Generalizable Humanoid Loco-Manipulation](https://mlsanity.com/paper/2510.11258/) (2025)
- [PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation](https://mlsanity.com/paper/2606.28128/) (2026)
- [TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents](https://mlsanity.com/paper/2602.21230/) (2026)
- [MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulation](https://mlsanity.com/paper/2510.18316/) (2025)
- [HumanoidMimicGen: Data Generation for Loco-Manipulation via Whole-Body Planning](https://mlsanity.com/paper/2605.27724/) (2026)
- [Flow with the Force Field: Learning 3D Compliant Flow Matching Policies from Force and Demonstration-Guided Simulation Data](https://mlsanity.com/paper/2510.02738/) (2025)
- [Human Centric General Physical Intelligence for Agile Manufacturing Automation](https://mlsanity.com/paper/2508.11960/) (2025)
- [Diffusing Trajectory Optimization Problems for Recovery During Multi-Finger Manipulation](https://mlsanity.com/paper/2510.07030/) (2025)
- [Accelerating trajectory optimization with Sobolev-trained diffusion policies](https://mlsanity.com/paper/2604.19011/) (2026)
- [DATAFARM: Distribution-Aligned Task and Motion Planning for Fine-Tuning Vision-Language-Action Models](https://mlsanity.com/paper/2609.12316/) (2026)
- [MinInter: Minimizing Trajectory Interpolation During Data Augmentation for Imitation Learning](https://mlsanity.com/paper/2606.24078/) (2026)
- [Continuous Manifold-Decomposed Impedance Retargeting for Contact-Rich Imitation Learning](https://mlsanity.com/paper/2609.15716/) (2026)
- [DexTaG: Tactile-as-Guidance in Reinforcement Learning for Dexterous Manipulation](https://mlsanity.com/paper/2609.33882/) (2026)

## Abstract (arXiv)

We present a low-cost data generation pipeline that integrates physics-based simulation, human demonstrations, and model-based planning to efficiently generate large-scale, high-quality datasets for contact-rich robotic manipulation tasks. Starting with a small number of embodiment-flexible human demonstrations collected in a virtual reality simulation environment, the pipeline refines these demonstrations using optimization-based kinematic retargeting and trajectory optimization to adapt them across various robot embodiments and physical parameters. This process yields a diverse, physically consistent dataset that enables cross-embodiment data transfer, and offers the potential to reuse legacy datasets collected under different hardware configurations or physical parameters. We validate the pipeline's effectiveness by training diffusion policies from the generated datasets for challenging contact-rich manipulation tasks across multiple robot embodiments, including a floating Allegro hand and bimanual robot arms. The trained policies are deployed zero-shot on hardware for bimanual iiwa arms, achieving high success rates with minimal human input. Project website: https://lujieyang.github.io/physicsgen/.