# One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluation

arXiv: https://arxiv.org/abs/2609.06190 · HTML: https://mlsanity.com/paper/2609.06190/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Builds on (most cited first, 30 of 30)

- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Metamorphic Testing: A New Approach for Generating Next Test Cases](https://mlsanity.com/paper/2002.12543/) (2020)
- [Blind Baselines Beat Membership Inference Attacks for Foundation Models](https://mlsanity.com/paper/2406.16201/) (2024)
- [Measurement to Meaning: A Validity-Centered Framework for AI Evaluation](https://mlsanity.com/paper/2505.10573/) (2025)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models](https://mlsanity.com/paper/2412.01784/) (2024)
- [International AI Safety Report 2026](https://mlsanity.com/paper/2602.21012/) (2026)
- [QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models](https://mlsanity.com/paper/2512.19526/) (2025)
- [Blindfold Baselines for Embodied QA](https://mlsanity.com/paper/1811.05013/) (2018)
- [PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models](https://mlsanity.com/paper/2601.16007/) (2026)
- [On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs](https://mlsanity.com/paper/2602.12506/) (2026)
- [Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers](https://mlsanity.com/paper/2410.11672/) (2024)
- [Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations](https://mlsanity.com/paper/2510.11196/) (2025)
- [Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark](https://mlsanity.com/paper/2606.10400/) (2026)
- [Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench](https://mlsanity.com/paper/2510.26865/) (2025)
- [Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?](https://mlsanity.com/paper/2605.22903/) (2026)
- [Metamorphic Testing of Vision-Language Action-Enabled Robots](https://mlsanity.com/paper/2602.22579/) (2026)
- [What AI evaluations for preventing catastrophic risks can and cannot do](https://mlsanity.com/paper/2412.08653/) (2024)
- [IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video](https://mlsanity.com/paper/2603.16432/) (2026)
- [Fundamental Limits of Black-Box Safety Evaluation: Information-Theoretic and Computational Barriers from Latent Context Conditioning](https://mlsanity.com/paper/2602.16984/) (2026)
- [MMGist: A Comprehensive Multimodal Benchmark for 2027](https://mlsanity.com/paper/2606.22437/) (2026)
- [Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands](https://mlsanity.com/paper/2605.15164/) (2026)
- [Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models](https://mlsanity.com/paper/2603.06828/) (2026)
- [Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability](https://mlsanity.com/paper/2605.22168/) (2026)

## Abstract (arXiv)

Behavioral evaluations perturb an input and read the induced change in the output in order to certify that a system uses that input. We show that the number of perturbations such a certificate requires is fixed, and that reporting a single perturbation cannot supply it. Where a response ratio is a property of the policy rather than of the test items, the behavioral record is a linear measurement of an exponent vector recording how much the output depends on each input, so perturbations identify input use exactly when their logarithms span the input space. At least n are needed for n inputs, an incomplete design confuses precisely the policies differing along the kernel of its design matrix, and sharpening a perturbation never substitutes for adding an independent one. We also derive in closed form the score such a test awards a policy that reads nothing, which is far from zero and which none of the probes we survey reports. Instantiating this where the correct response is fixed by dimensional analysis, we run a complete identifying set of three perturbations on three vision--language models reporting a physical quantity from video. All three score far below their own blind bound rather than above it, because each defaults to one of a small set of round calibration values that never matches what the scale asserts; none moves its relabeling response by a single exponent, and none is separable from the same model instructed to ignore the video.