# Scalable Real2Sim: Physics-Aware Asset Generation Via Robotic Pick-and-Place Setups

arXiv: https://arxiv.org/abs/2503.00370 · HTML: https://mlsanity.com/paper/2503.00370/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2025.

## Built on by (most cited first, 30 of 30)

- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [Robot Learning from a Physical World Model](https://mlsanity.com/paper/2511.07416/) (2025)
- [GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation](https://mlsanity.com/paper/2510.20813/) (2025)
- [RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation](https://mlsanity.com/paper/2510.23571/) (2025)
- [Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators](https://mlsanity.com/paper/2510.04354/) (2025)
- [Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixels](https://mlsanity.com/paper/2503.22634/) (2025)
- [Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin](https://mlsanity.com/paper/2504.03597/) (2025)
- [OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning](https://mlsanity.com/paper/2512.13100/) (2025)
- [Steerable Scene Generation with Post Training and Inference-Time Search](https://mlsanity.com/paper/2505.04831/) (2025)
- [Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents](https://mlsanity.com/paper/2607.19190/) (2026)
- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [Robot Learning from Any Images](https://mlsanity.com/paper/2509.22970/) (2025)
- [TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulation](https://mlsanity.com/paper/2512.19390/) (2025)
- [Is Single-View Mesh Reconstruction Ready for Robotics?](https://mlsanity.com/paper/2505.17966/) (2025)
- [Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulation](https://mlsanity.com/paper/2510.11689/) (2025)
- [GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models](https://mlsanity.com/paper/2608.05948/) (2026)
- [LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations](https://mlsanity.com/paper/2508.17547/) (2025)
- [D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping](https://mlsanity.com/paper/2603.01151/) (2026)
- [Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Merging](https://mlsanity.com/paper/2508.00354/) (2025)
- [ObjSplat: Geometry-Aware Gaussian Surfels for Active Object Reconstruction](https://mlsanity.com/paper/2601.06997/) (2026)
- [From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation](https://mlsanity.com/paper/2604.15805/) (2026)
- [3D Generation for Embodied AI and Robotic Simulation: A Survey](https://mlsanity.com/paper/2604.26509/) (2026)
- [EMPIRIC: Experiment-Driven Learning of Residual World Models for Robot Planning](https://mlsanity.com/paper/2609.35047/) (2026)
- [AntiGrounding: Executable Robot Trajectories as Visual Prompts for VLM-Guided Manipulation](https://mlsanity.com/paper/2506.12374/) (2025)
- [Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2603.08588/) (2026)
- [WorldAgent: Verification-Guided Agentic Physical World Construction](https://mlsanity.com/paper/2609.33208/) (2026)
- [F4R: Failure-Driven Recognition, Reconstruction, Refinement, and Redeployment for Continual Robot Self-Improvement](https://mlsanity.com/paper/2609.35575/) (2026)
- [FACT: Fidelity-Aware Construction of Articulated Twins](https://mlsanity.com/paper/2609.37067/) (2026)
- [DexNinja: Learning Robust Dexterous Cutting Policy with a Real-to-Sim-to-Real Data Engine](https://mlsanity.com/paper/2610.17208/) (2026)

## Abstract (arXiv)

Simulating object dynamics from real-world perception shows great promise for digital twins and robotic manipulation but often demands labor-intensive measurements and expertise. We present a fully automated Real2Sim pipeline that generates simulation-ready assets for real-world objects through robotic interaction. Using only a robot's joint torque sensors and an external camera, the pipeline identifies visual geometry, collision geometry, and physical properties such as inertial parameters. Our approach introduces a general method for extracting high-quality, object-centric meshes from photometric reconstruction techniques (e.g., NeRF, Gaussian Splatting) by employing alpha-transparent training while explicitly distinguishing foreground occlusions from background subtraction. We validate the full pipeline through extensive experiments, demonstrating its effectiveness across diverse objects. By eliminating the need for manual intervention or environment modifications, our pipeline can be integrated directly into existing pick-and-place setups, enabling scalable and efficient dataset creation. Project page (with code and data): https://scalable-real2sim.github.io/.