# NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics

arXiv: https://arxiv.org/abs/2605.08452 · HTML: https://mlsanity.com/paper/2605.08452/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 26 of 26)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [SmolVLM: Redefining small and efficient multimodal models](https://mlsanity.com/paper/2504.05299/) (2025)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding](https://mlsanity.com/paper/2601.10611/) (2026)
- [Cambrian-S: Towards Spatial Supersensing in Video](https://mlsanity.com/paper/2511.04670/) (2025)
- [Stop Measuring Calibration When Humans Disagree](https://mlsanity.com/paper/2210.16133/) (2022)
- [QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models](https://mlsanity.com/paper/2512.19526/) (2025)
- [Reasoning in Space via Grounding in the World](https://mlsanity.com/paper/2510.13800/) (2025)
- [Evaluating Multiview Object Consistency in Humans and Image Models](https://mlsanity.com/paper/2409.05862/) (2024)
- [From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs](https://mlsanity.com/paper/2512.19683/) (2025)
- [Mind the Uncertainty in Human Disagreement: Evaluating Discrepancies between Model Predictions and Human Responses in VQA](https://mlsanity.com/paper/2410.02773/) (2024)
- [Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Models](https://mlsanity.com/paper/2505.23798/) (2025)
- [From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Models](https://mlsanity.com/paper/2601.15533/) (2026)
- [Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answering](https://mlsanity.com/paper/2510.11295/) (2025)

## Abstract (arXiv)

The ability to derive precise spatial and physical insights is a cornerstone of vision-language models (VLMs), yet their poor performances in related spatial intelligence tasks such as physical reasoning remain a fundamental barrier. The community critically lacks a scientific analysis revealing whether VLMs faithfully reach answers or plausibly make guesses. This work aims to provide a fundamental understanding of how VLMs perceive the physical world, and utilize physical laws, while assessing the reliability of model confidence. We propose NICE and FACT, a dual-diagnostic paradigm that explicitly decomposes quantitative reasoning for kinematic physics: FACT diagnoses visual fidelity, physical law comprehension, and temporal grounding. NICE studies our novel neighborhood-informed calibration method and novel metrics to evaluate and calibrate confidence reliability. Evaluated across 6 latest state-of-the-art VLMs, we uncover that models fail to identify visual preconditions or utilize necessary physical laws to reach answers. This work highlights and establishes a standardized diagnostic paradigm to guide the development of faithful, physically-grounded VLMs.