# Physical Property Understanding from Language-Embedded Feature Fields

arXiv: https://arxiv.org/abs/2404.04242 · HTML: https://mlsanity.com/paper/2404.04242/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory](https://mlsanity.com/paper/2210.05663/) (2022)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)
- [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023)
- [Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping](https://mlsanity.com/paper/2309.07970/) (2023)
- [Material Recognition in the Wild with the Materials in Context Database](https://mlsanity.com/paper/1412.0623/) (2014)
- [CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets](https://mlsanity.com/paper/2302.02551/) (2023)
- [3D Concept Learning and Reasoning from Multi-View Images](https://mlsanity.com/paper/2303.11327/) (2023)
- [Learning Visual Predictive Models of Physics for Playing Billiards](https://mlsanity.com/paper/1511.07404/) (2015)
- [The Curious Robot: Learning Visual Representations via Physical Interactions](https://mlsanity.com/paper/1604.01360/) (2016)
- [CASA: Category-agnostic Skeletal Animal Reconstruction](https://mlsanity.com/paper/2211.03568/) (2022)

## Built on by (most cited first, 30 of 48)

- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)
- [PhysGen3D: Crafting a Miniature Interactive World from a Single Image](https://mlsanity.com/paper/2503.20746/) (2025)
- [PhysX-3D: Physical-Grounded 3D Asset Generation](https://mlsanity.com/paper/2507.12465/) (2025)
- [Physically Compatible 3D Object Modeling from a Single Image](https://mlsanity.com/paper/2405.20510/) (2024)
- [DRAWER: Digital Reconstruction and Articulation With Environment Realism](https://mlsanity.com/paper/2504.15278/) (2025)
- [AutoVFX: Physically Realistic Video Editing from Natural Language Instructions](https://mlsanity.com/paper/2411.02394/) (2024)
- [DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness](https://mlsanity.com/paper/2503.22677/) (2025)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [PUGS: Zero-shot Physical Understanding with Gaussian Splatting](https://mlsanity.com/paper/2502.12231/) (2025)
- [Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels](https://mlsanity.com/paper/2508.17437/) (2025)
- [GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs](https://mlsanity.com/paper/2412.11258/) (2024)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [Robot Learning from a Physical World Model](https://mlsanity.com/paper/2511.07416/) (2025)
- [HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video](https://mlsanity.com/paper/2510.05560/) (2025)
- [A Survey on Computational Solutions for Reconstructing Complete Objects by Reassembling Their Fractured Parts](https://mlsanity.com/paper/2410.14770/) (2024)
- [PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding](https://mlsanity.com/paper/2511.20562/) (2025)
- [Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents](https://mlsanity.com/paper/2607.19190/) (2026)
- [PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation](https://mlsanity.com/paper/2511.18570/) (2025)
- [Seeing the Wind from a Falling Leaf](https://mlsanity.com/paper/2512.00762/) (2025)
- [TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulation](https://mlsanity.com/paper/2512.19390/) (2025)
- [PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement](https://mlsanity.com/paper/2602.14968/) (2026)
- [MVInverse: Feed-forward Multi-view Inverse Rendering in Seconds](https://mlsanity.com/paper/2512.21003/) (2025)
- [ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimation](https://mlsanity.com/paper/2506.05317/) (2025)
- [NeuROK: Generative 4D Neural Object Kinematics](https://mlsanity.com/paper/2605.30347/) (2026)
- [PAVAS: Physics-Aware Video-to-Audio Synthesis](https://mlsanity.com/paper/2512.08282/) (2025)
- [Physically Guided Visual Mass Estimation from a Single RGB Image](https://mlsanity.com/paper/2601.20303/) (2026)
- [FieryGS: In-the-Wild Fire Synthesis with Physics-Integrated Gaussian Splatting](https://mlsanity.com/paper/2605.00177/) (2026)
- [UniPhysGen: Unified Physical Grounding for Simulation-Ready 3D Assets](https://mlsanity.com/paper/2607.13586/) (2026)
- [SiPhy: Single-Image Physical Property Reasoning](https://mlsanity.com/paper/2607.22355/) (2026)
- [Accelerating Physical Property Reasoning for Augmented Visual Cognition](https://mlsanity.com/paper/2511.03126/) (2025)

## Abstract (arXiv)

Can computers perceive the physical properties of objects solely through vision? Research in cognitive science and vision science has shown that humans excel at identifying materials and estimating their physical properties based purely on visual appearance. In this paper, we present a novel approach for dense prediction of the physical properties of objects using a collection of images. Inspired by how humans reason about physics through vision, we leverage large language models to propose candidate materials for each object. We then construct a language-embedded point cloud and estimate the physical properties of each 3D point using a zero-shot kernel regression approach. Our method is accurate, annotation-free, and applicable to any object in the open world. Experiments demonstrate the effectiveness of the proposed approach in various physical property reasoning tasks, such as estimating the mass of common objects, as well as other properties like friction and hardness.