mlsanityData to 14 Oct 2026

PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models

arXiv 2603.16958 ↗First posted 17 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) are increasingly applied to robotic perception and manipulation, yet their ability to infer physical properties required for manipulation remains limited. In particular, estimating the mass of real-world objects is essential for determining appropriate grasp force and ensuring safe interaction. However, current VLMs lack reliable mass reasoning capabilities, and most existing benchmarks do not explicitly evaluate physical quantity estimation under realistic sensing conditions. In this work, we propose PhysQuantAgent, a framework for real-world object mass estimation using VLMs, together with VisPhysQuant, a new benchmark dataset for evaluation. VisPhysQuant consists of RGB-D videos of real objects captured from multiple viewpoints, annotated with precise mass measurements. To improve estimation accuracy, we introduce three visual prompting methods that enhance the input image with object detection, scale estimation, and cross-sectional image generation to help the model comprehend the size and internal structure of the target object. Experiments show that visual prompting significantly improves mass estimation accuracy on real-world data, suggesting the efficacy of integrating spatial reasoning with VLM knowledge for physical inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

32 of 38 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models22 Sep
  2. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes8 Sep
  3. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding27 Jan
  4. 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
  5. 2024Reflectance Estimation for Proximity Sensing by Vision-Language Models: Utilizing Distributional Semantics for Low-Level Cognition in Robotics11 Apr
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  8. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 24

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2025Qwen3-VL Technical Reportcited by 2,793
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  4. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  5. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  6. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  7. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  8. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  9. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  10. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  11. 2020Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotationscited by 155
  12. 2024RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videoscited by 115
12 more
  1. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 92
  2. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 54
  3. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  4. 2023HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructionscited by 48
  5. 2024Physical Property Understanding from Language-Embedded Feature Fieldscited by 48
  6. 2024Octopi: Object Property Reasoning with Large Tactile-Language Modelscited by 48
  7. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  8. 2025PhysX-3D: Physical-Grounded 3D Asset Generationcited by 42
  9. 2024ContPhy: Continuum Physical Concept Learning and Reasoning from Videoscited by 27
  10. 2025PUGS: Zero-shot Physical Understanding with Gaussian Splattingcited by 24
  11. 20233DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4Vcited by 15
  12. 2019Mass Estimation from Images using Deep Neural Network and Sparse Ground Truthcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Physically Grounded Vision-Language Models for Robotic Manipulation
  2. 2025QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models
  3. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Models
  4. 2026ViTacPhys: Physical Property-Aware Grasping from Human Visual-Tactile Demonstrations ↗
  5. 2025Using Visual Language Models to Control Bionic Hands: Assessment of Object Perception and Grasp Inference
  6. 2025Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling
  7. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation
  8. 2024Task-oriented Robotic Manipulation with Vision Language Models