# PUGS: Zero-shot Physical Understanding with Gaussian Splatting

arXiv: https://arxiv.org/abs/2502.12231 · HTML: https://mlsanity.com/paper/2502.12231/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 41 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction](https://mlsanity.com/paper/2406.06521/) (2024)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving](https://mlsanity.com/paper/2307.15058/) (2023)
- [Segment Any 3D Gaussians](https://mlsanity.com/paper/2312.00860/) (2023)
- [Open-World Object Manipulation using Pre-trained Vision-Language Models](https://mlsanity.com/paper/2303.00905/) (2023)
- [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023)
- [RaDe-GS: Rasterizing Depth in Gaussian Splatting](https://mlsanity.com/paper/2406.01467/) (2024)
- [SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis](https://mlsanity.com/paper/2311.17590/) (2023)
- [RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools](https://mlsanity.com/paper/2306.14447/) (2023)
- [PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation](https://mlsanity.com/paper/2309.15596/) (2023)
- [AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation](https://mlsanity.com/paper/2407.07889/) (2024)
- [Physical Property Understanding from Language-Embedded Feature Fields](https://mlsanity.com/paper/2404.04242/) (2024)
- [Octopi: Object Property Reasoning with Large Tactile-Language Models](https://mlsanity.com/paper/2405.02794/) (2024)
- [SA-GS: Scale-Adaptive Gaussian Splatting for Training-Free Anti-Aliasing](https://mlsanity.com/paper/2403.19615/) (2024)

## Built on by (most cited first, 24 of 24)

- [Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splatting](https://mlsanity.com/paper/2506.05280/) (2025)
- [Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation](https://mlsanity.com/paper/2512.23705/) (2025)
- [PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding](https://mlsanity.com/paper/2511.20562/) (2025)
- [PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation](https://mlsanity.com/paper/2511.18570/) (2025)
- [TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulation](https://mlsanity.com/paper/2512.19390/) (2025)
- [EA3D: Online Open-World 3D Object Extraction from Streaming Videos](https://mlsanity.com/paper/2510.25146/) (2025)
- [Physically Guided Visual Mass Estimation from a Single RGB Image](https://mlsanity.com/paper/2601.20303/) (2026)
- [Towards Next-Generation SLAM: A Survey on 3DGS-SLAM Focusing on Performance, Robustness, and Future Directions](https://mlsanity.com/paper/2602.04251/) (2026)
- [FieryGS: In-the-Wild Fire Synthesis with Physics-Integrated Gaussian Splatting](https://mlsanity.com/paper/2605.00177/) (2026)
- [SiPhy: Single-Image Physical Property Reasoning](https://mlsanity.com/paper/2607.22355/) (2026)
- [Learning Equivariant Neural-Augmented Object Dynamics From Few Interactions](https://mlsanity.com/paper/2605.02699/) (2026)
- [UniPixie: Unified and Probabilistic 3D Physics Learning via Flow Matching](https://mlsanity.com/paper/2606.05399/) (2026)
- [PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoning](https://mlsanity.com/paper/2606.08655/) (2026)
- [Gen2Physics: Grounding Generated 3D Meshes in Physics via Multi-View Material Decomposition](https://mlsanity.com/paper/2608.23869/) (2026)
- [XDen-1K: A Density Field Dataset of Real-World Objects](https://mlsanity.com/paper/2512.10668/) (2025)
- [PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models](https://mlsanity.com/paper/2603.16958/) (2026)
- [Vision Meets WiFi: Physics-Grounded Estimation of Volumetric Mechanical Properties](https://mlsanity.com/paper/2608.07726/) (2026)
- [KnockGS:interaction-Grounded Calibrationof Physical Gaussian Representations](https://mlsanity.com/paper/2608.27365/) (2026)
- [Analytic Dynamics: Learning Physics-Grounded Representation for Fast Intrinsic Dynamics Inference from Monocular Videos](https://mlsanity.com/paper/2608.31025/) (2026)
- [PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image](https://mlsanity.com/paper/2609.18920/) (2026)
- [Towards Physically Grounded Articulated 3D Assets: A Multi-Agent Pipeline and Benchmark](https://mlsanity.com/paper/2610.13947/) (2026)
- [Physical-Field-Bridged Material Parameter Inference from Multi-View Deformation Videos](https://mlsanity.com/paper/2610.17591/) (2026)
- [From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction](https://mlsanity.com/paper/2610.17630/) (2026)
- [Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation](https://mlsanity.com/paper/2610.17651/) (2026)

## Abstract (arXiv)

Current robotic systems can understand the categories and poses of objects well. But understanding physical properties like mass, friction, and hardness, in the wild, remains challenging. We propose a new method that reconstructs 3D objects using the Gaussian splatting representation and predicts various physical properties in a zero-shot manner. We propose two techniques during the reconstruction phase: a geometry-aware regularization loss function to improve the shape quality and a region-aware feature contrastive loss function to promote region affinity. Two other new techniques are designed during inference: a feature-based property propagation module and a volume integration module tailored for the Gaussian representation. Our framework is named as zero-shot physical understanding with Gaussian splatting, or PUGS. PUGS achieves new state-of-the-art results on the standard benchmark of ABO-500 mass prediction. We provide extensive quantitative ablations and qualitative visualization to demonstrate the mechanism of our designs. We show the proposed methodology can help address challenging real-world grasping tasks. Our codes, data, and models are available at https://github.com/EverNorif/PUGS