mlsanityData to 7 Oct 2026

Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions

arXiv 2511.04665 ↗First posted 6 Nov 2025cs.RO cs.CV cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Robotic manipulation policies are advancing rapidly, but their direct evaluation in the real world remains costly, time-consuming, and difficult to reproduce, particularly for tasks involving deformable objects. Simulation provides a scalable and systematic alternative, yet existing simulators often fail to capture the coupled visual and physical complexity of soft-body interactions. We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos and renders robots, objects, and environments with photorealistic fidelity using 3D Gaussian Splatting. We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing, demonstrating that simulated rollouts correlate strongly with real-world execution performance and reveal key behavioral patterns of learned policies. Our results suggest that combining physics-informed reconstruction with high-quality rendering enables reproducible, scalable, and accurate evaluation of robotic manipulation policies. Website: https://real2sim-eval.github.io/

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 63 of its 83 references that are in the map. First posted on arXiv 6 Nov 2025.

Builds on

63 of 83 references.

References · 63

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  8. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  9. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  10. 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
  11. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  12. 2019Solving Rubik's Cube with a Robot Handcited by 699
38 more
  1. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  2. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  3. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  4. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  5. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  6. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  7. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  8. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  9. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
  10. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  11. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  12. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  13. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  14. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 185
  15. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulatorscited by 180
  16. 2024VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Realitycited by 175
  17. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  18. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  19. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  20. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 145
  21. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
  22. 2025A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulationcited by 125
  23. 2024Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussianscited by 92
  24. 2025PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videoscited by 78
  25. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68
  26. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  27. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 63
  28. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  29. 2024AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulationcited by 52
  30. 2024RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulatorcited by 50
  31. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 46
  32. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  33. 2025Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulationcited by 45
  34. 2023IndustReal: Transferring Contact-Rich Assembly Tasks from Simulation to Realitycited by 44
  35. 2024Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editingcited by 44
  36. 2016Analysis and Observations from the First Amazon Picking Challengecited by 41
  37. 2025Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardwarecited by 41
  38. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 40

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 26.

27 papers.

Sort:

By area:

  1. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  2. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 29
  3. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 27
  4. 2026RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policiescited by 19
  5. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 17
  6. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 8
  7. 2026Masked Visual Actions for Unified World Modelingcited by 7
  8. 2026Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agentscited by 6
  9. 2026Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observationscited by 5
  10. 2026RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluationcited by 4
  11. 2026iMaC: Translating Actions into Motion and Contact Images for Embodied World Modelscited by 2
  12. 2026Structured 4D Latent Predictive Model for Robot Planningcited by 2
15 more
  1. 2026EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templatescited by 1
  2. 2026DeformMaster: An Interactive Physics-Neural World Model for Deformable Objects from Videoscited by 1
  3. 2026MatPhys: Learning Material-Aware Physics Parameters for Deformable Object Simulation from Videoscited by 1
  4. 2026Active Real-World Factor-Based Evaluation for Generalist Robot Policiescited by 1
  5. 2026ElasticFlow: One-Step Physics-Consistent Policy with Elastic Time Horizons for Language-Guided Manipulationcited by 0
  6. 2026EgoPhys: Learning Generalizable Physics Models of Deformable Objects from Egocentric Videocited by 0
  7. 2026BendTwin: Robust Dense-to-Sparse Physical Reconstruction with Bending-Aware Differentiable Spring-Mass Modelscited by 0
  8. 2026SCAPE: Scenario-Conditioned Simulation-Augmented Policy Evaluationcited by 0
  9. 2026LightBridge: Feed-Forward Generative Relighting for 3D Gaussian Splattingcited by 0
  10. 2026R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Modelscited by 0
  11. 2026ReVeal: A Reconstruction-Aware Real-to-Sim Framework for VLA Policy Evaluationcited by 0
  12. 2026Intelligence Across Embodimentscited by 0
  13. 2026PointCast: One World Model for Rigid, Articulated, and Deformable Object Manipulationcited by 0
  14. 2026Robot-GST: geometry-aware spatial-temporal robot policy representation and evaluationcited by 0
  15. 2026CoDimRecon: Agentic Reconstruction of Sim-Ready 3D Scenes with Deformable Curves, Surfaces, and Volumescited by 0