mlsanityData to 14 Oct 2026

PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos

arXiv 2503.17973 ↗First posted 23 Mar 2025cs.CV cs.AI cs.ROcited by 87

Not on the map. Why?

Abstract (arXiv)

Creating a physical digital twin of a real-world object has immense potential in robotics, content creation, and XR. In this paper, we present PhysTwin, a novel framework that uses sparse videos of dynamic objects under interaction to produce a photo- and physically realistic, real-time interactive virtual replica. Our approach centers on two key components: (1) a physics-informed representation that combines spring-mass models for realistic physical simulation, generative shape models for geometry, and Gaussian splats for rendering; and (2) a novel multi-stage, optimization-based inverse modeling framework that reconstructs complete geometry, infers dense physical properties, and replicates realistic appearance from videos. Our method integrates an inverse physics framework with visual perception cues, enabling high-fidelity reconstruction even from partial, occluded, and limited viewpoints. PhysTwin supports modeling various deformable objects, including ropes, stuffed animals, cloth, and delivery packages. Experiments show that PhysTwin outperforms competing methods in reconstruction, rendering, future prediction, and simulation under novel interactions. We further demonstrate its applications in interactive real-time simulation and model-based robotic motion planning.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Builds on 57 of its 73 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

57 of 73 references.

References · 57

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  4. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  5. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  6. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  7. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  8. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  9. 2020Learning Mesh-Based Simulation with Graph Networkscited by 868
  10. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  11. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 759
  12. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
38 more
  1. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 746
  2. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  3. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 587
  4. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 476
  5. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 471
  6. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  7. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 377
  8. 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 347
  9. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 287
  10. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  11. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 220
  12. 2023Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particlecited by 211
  13. 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 200
  14. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 191
  15. 20244D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenescited by 189
  16. 2024VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Realitycited by 179
  17. 2023HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Samplingcited by 174
  18. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 154
  19. 2023DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splattingcited by 124
  20. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
  21. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  22. 2024Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussianscited by 97
  23. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 77
  24. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  25. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 72
  26. 2023Forward Flow for Novel View Synthesis of Dynamic Scenescited by 68
  27. 2021DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cuttingcited by 64
  28. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 63
  29. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  30. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 63
  31. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 62
  32. 2023Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamicscited by 57
  33. 2022Physics Informed Neural Fields for Smoke Reconstruction with Sparse Datacited by 52
  34. 2023PIE-NeRF: Physics-based Interactive Elastodynamics with NeRFcited by 51
  35. 2022NeuPhysics: Editable Neural Geometry and Physics from Monocular Videoscited by 50
  36. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 43
  37. 2023Flow supervision for Deformable NeRFcited by 42
  38. 2022Differentiable Simulation of Soft Multi-body Systemscited by 39

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 63.

87 papers.

Sort:

By area:

  1. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  2. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 62
  3. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 40
  4. 2025PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Imagecited by 37
  5. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 30
  6. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 30
  7. 2025Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixelscited by 24
  8. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 22
  9. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 19
  10. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  11. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 14
  12. 2025Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twincited by 12
38 more
  1. 2026PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objectscited by 11
  2. 2025PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesiscited by 10
  3. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  4. 2025SIMPACT: Simulation-Enabled Action Planning using Vision-Language Modelscited by 8
  5. 2026Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agentscited by 8
  6. 2025SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Trainingcited by 7
  7. 2025Robot Learning from Any Imagescited by 7
  8. 2025TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulationcited by 7
  9. 2026Evolution of Video Generative Foundationscited by 7
  10. 2025Physics-Informed Deformable Gaussian Splatting: Towards Unified Constitutive Laws for Time-Evolving Material Fieldcited by 6
  11. 2026Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fieldscited by 6
  12. 2025NeuSpring: Neural Spring Fields for Reconstruction and Simulation of Deformable Objects from Videoscited by 5
  13. 2026ReconPhys: Reconstruct Appearance and Physical Attributes from Single Videocited by 5
  14. 2025PhysGaia: A Physics-Aware Benchmark with Multi-Body Interactions for Dynamic Novel View Synthesiscited by 4
  15. 2025ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimationcited by 4
  16. 2026SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulationcited by 4
  17. 2026Picasso: Holistic Scene Reconstruction with Physics-Constrained Samplingcited by 4
  18. 2026From World Models to World Action Models: A Concise Tutorial for Roboticscited by 4
  19. 2026D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Graspingcited by 3
  20. 2026WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformercited by 3
  21. 2026MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videoscited by 3
  22. 2026NeuROK: Generative 4D Neural Object Kinematicscited by 3
  23. 2026Structured 4D Latent Predictive Model for Robot Planningcited by 3
  24. 2026MotionPhysics: Learnable Motion Distillation for Text-Guided Simulationcited by 2
  25. 2026DeformMaster: An Interactive Physics-Neural World Model for Deformable Objects from Videoscited by 2
  26. 2026PhysiFormer: Learning to Simulate Mechanics in World Spacecited by 2
  27. 2026Learning Physics-Guided Residual Dynamics for Deformable Object Simulationcited by 2
  28. 2026SiPhy: Single-Image Physical Property Reasoningcited by 2
  29. 2025FoldNet: Learning Generalizable Closed-Loop Policy for Garment Folding via Keypoint-Driven Asset and Demonstration Synthesiscited by 1
  30. 2025TRELLIS-Enhanced Surface Features for Comprehensive Intracranial Aneurysm Analysiscited by 1
  31. 2025Object Reconstruction under Occlusion with Generative Priors and Contact-induced Constraintscited by 1
  32. 2025M-PhyGs: Multi-Material Object Dynamics from Videocited by 1
  33. 2026SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latentscited by 1
  34. 2026SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worldscited by 1
  35. 2026EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templatescited by 1
  36. 20263D Generation for Embodied AI and Robotic Simulation: A Surveycited by 1
  37. 2026Learning Equivariant Neural-Augmented Object Dynamics From Few Interactionscited by 1
  38. 2026MatPhys: Learning Material-Aware Physics Parameters for Deformable Object Simulation from Videoscited by 1

50 most cited of 87 All 87 (JSON)