# Simulation-Ready Cluttered Scene Estimation via Physics-aware Joint Shape and Pose Optimization

arXiv: https://arxiv.org/abs/2602.20150 · HTML: https://mlsanity.com/paper/2602.20150/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 23 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation](https://mlsanity.com/paper/2412.03558/) (2024)
- [CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image](https://mlsanity.com/paper/2502.12894/) (2025)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Search](https://mlsanity.com/paper/2205.02961/) (2022)
- [An End-to-End Differentiable Framework for Contact-Aware Robot Design](https://mlsanity.com/paper/2107.07501/) (2021)
- [PhyRecon: Physically Plausible Neural Scene Reconstruction](https://mlsanity.com/paper/2404.16666/) (2024)
- [RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation](https://mlsanity.com/paper/2205.05678/) (2022)
- [A Review of Differentiable Simulators](https://mlsanity.com/paper/2407.05560/) (2024)
- [SceneComplete: Open-World 3D Scene Completion in Cluttered Real World Environments for Robot Manipulation](https://mlsanity.com/paper/2410.23643/) (2024)
- [HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video](https://mlsanity.com/paper/2510.05560/) (2025)
- [Vysics: Object Reconstruction Under Occlusion by Fusing Vision and Contact-Rich Physics](https://mlsanity.com/paper/2504.18719/) (2025)
- [One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering](https://mlsanity.com/paper/2412.00259/) (2024)
- [Inferring 3D Shapes of Unknown Rigid Objects in Clutter through Inverse Physics Reasoning](https://mlsanity.com/paper/1903.05749/) (2019)
- [SDRS: Shape-Differentiable Robot Simulator](https://mlsanity.com/paper/2412.19127/) (2024)
- [PhysPose: Refining 6D Object Poses with Physical Constraints](https://mlsanity.com/paper/2503.23587/) (2025)
- [Simultaneous Trajectory Optimization and Contact Selection for Contact-rich Manipulation with High-Fidelity Geometry](https://mlsanity.com/paper/2407.16976/) (2024)
- [Efficient Differentiable Contact Model with Long-range Influence](https://mlsanity.com/paper/2509.20917/) (2025)

## Built on by (most cited first, 4 of 4)

- [Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations](https://mlsanity.com/paper/2604.27106/) (2026)
- [REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image](https://mlsanity.com/paper/2605.30338/) (2026)
- [Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction](https://mlsanity.com/paper/2609.31715/) (2026)
- [I2CD: Direct Image-to-Convex Decomposition for Simulation-Ready Collision Geometry](https://mlsanity.com/paper/2610.03453/) (2026)

## Abstract (arXiv)

Estimating simulation-ready scenes from real-world observations is crucial for downstream planning and policy learning tasks. Regretfully, existing methods struggle in cluttered environments, often exhibiting prohibitive computational cost, poor robustness, and restricted generality when scaling to multiple interacting objects. We propose a unified optimization-based formulation for real-to-sim scene estimation that jointly recovers the shapes and poses of multiple rigid objects under physical constraints. Our method is built on two key technical innovations. First, we leverage the recently introduced shape-differentiable contact model, whose global differentiability permits joint optimization over object geometry and pose while modeling inter-object contacts. Second, we exploit the structured sparsity of the augmented Lagrangian Hessian to derive an efficient linear system solver whose computational cost scales favorably with scene complexity. Building on this formulation, we develop an end-to-end Simulation-ready Physics-Aware Reconstruction for Cluttered Scenes (SPARCS) pipeline, which integrates learning-based object initialization, physics-constrained joint shape-pose optimization, and differentiable texture refinement. Experiments on cluttered scenes with up to 5 objects and 22 convex hulls demonstrate that our approach robustly reconstructs physically valid, simulation-ready object shapes and poses. Project webpage: https://rory-weicheng.github.io/SPARCS/.