# Teaching Vision-Language Models to Use the Scale They Are Given: Label-Free Equivariance Training for Metric Physical Reasoning

arXiv: https://arxiv.org/abs/2609.00658 · HTML: https://mlsanity.com/paper/2609.00658/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 19 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 17 of 17)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [CoPhy: Counterfactual Learning of Physical Dynamics](https://mlsanity.com/paper/1909.12000/) (2019)
- [LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines](https://mlsanity.com/paper/2411.08027/) (2024)
- [ContPhy: Continuum Physical Concept Learning and Reasoning from Videos](https://mlsanity.com/paper/2402.06119/) (2024)
- [Dimensionless machine learning: Imposing exact units equivariance](https://mlsanity.com/paper/2204.00887/) (2022)
- [Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space](https://mlsanity.com/paper/2202.00368/) (2022)
- [Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning](https://mlsanity.com/paper/2608.27549/) (2026)
- [CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering](https://mlsanity.com/paper/2211.03779/) (2022)
- [SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models](https://mlsanity.com/paper/2512.05955/) (2025)
- [Test-Time Consistency in Vision Language Models](https://mlsanity.com/paper/2506.22395/) (2025)
- [PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning](https://mlsanity.com/paper/2608.04575/) (2026)
- [Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos](https://mlsanity.com/paper/2605.20576/) (2026)
- [Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs](https://mlsanity.com/paper/2606.05966/) (2026)
- [Ill-Posed by Design: Probing Evidence Use in VLMs](https://mlsanity.com/paper/2606.24335/) (2026)

## Abstract (arXiv)

Metric questions about video, such as the speed of a moving object, require a vision-language model to convert visual measurements into physical units using a real-world reference supplied in the prompt. We find that current models use this reference only partially. When every world-space quantity in the prompt is multiplied by a common factor, the prompt still describes the same video and the correct answer changes by exactly that factor, but the predictions of eight models change by less, and their accuracy stays concentrated near the scale that the depicted objects usually have. Asked the same physics in a scale-free form, the two models we test recover the closed-form scaling laws on most items, which indicates that the deficit lies in metric grounding and not in knowledge of the physical mechanism. Because the scaling relation is exact, it can serve as supervision without metric annotations. Equivariance Self-Distillation (EquiSD) projects a model's own prediction onto the functions that satisfy this relation and fine-tunes the model on the resulting targets, with one query per training question and no ground truth. Trained on synthetic video only, EquiSD brings a 3B model close to the exact relation on held-out simulated videos, also at scales not seen in training, and improves its accuracy across scales. Without adaptation, it also improves accuracy across scales on the QuantiPhy benchmark, where its gain reaches 93% of that obtained by supervision with exact simulator answers.