# When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Reads

arXiv: https://arxiv.org/abs/2609.22299 · HTML: https://mlsanity.com/paper/2609.22299/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 33 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 30 of 31)

- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](https://mlsanity.com/paper/2506.18123/) (2025)
- [PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies](https://mlsanity.com/paper/2512.16881/) (2025)
- [RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation](https://mlsanity.com/paper/2507.00435/) (2025)
- [Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processes](https://mlsanity.com/paper/1706.06544/) (2017)
- [ASID: Active Exploration for System Identification in Robotic Manipulation](https://mlsanity.com/paper/2404.12308/) (2024)
- [ODRL: A Benchmark for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.20750/) (2024)
- [RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies](https://mlsanity.com/paper/2604.09860/) (2026)
- [Acquisition Conditioned Oracle for Nongreedy Active Feature Acquisition](https://mlsanity.com/paper/2302.13960/) (2023)
- [A Survey on Active Feature Acquisition Strategies](https://mlsanity.com/paper/2502.11067/) (2025)
- [A Taxonomy for Evaluating Generalist Robot Manipulation Policies](https://mlsanity.com/paper/2503.01238/) (2025)
- [Fast Adaptation via Policy-Dynamics Value Functions](https://mlsanity.com/paper/2007.02879/) (2020)
- [Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap](https://mlsanity.com/paper/2006.01952/) (2020)
- [Evaluation of Active Feature Acquisition Methods for Time-varying Feature Settings](https://mlsanity.com/paper/2312.01530/) (2023)
- [AFABench: A Generic Framework for Benchmarking Active Feature Acquisition](https://mlsanity.com/paper/2508.14734/) (2025)
- [Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics](https://mlsanity.com/paper/2505.13150/) (2025)
- [What Are We Actually Benchmarking in Robot Manipulation?](https://mlsanity.com/paper/2606.04233/) (2026)
- [Value of Information in Feedback Control: Quantification](https://mlsanity.com/paper/1812.07534/) (2018)
- [Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective](https://mlsanity.com/paper/2508.11117/) (2025)
- [Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization](https://mlsanity.com/paper/2508.20294/) (2025)
- [Learning-To-Measure: In-Context Active Feature Acquisition](https://mlsanity.com/paper/2510.12624/) (2025)
- [Adaptive control of a mechatronic system using constrained residual reinforcement learning](https://mlsanity.com/paper/2110.02566/) (2021)
- [Poke and Strike: Learning Task-Informed Exploration Policies](https://mlsanity.com/paper/2509.00178/) (2025)
- [A Statistical Test for the Benefits of Personalizing Interventions](https://mlsanity.com/paper/2607.08951/) (2026)
- [NOCTA: Non-Greedy Objective Cost-Tradeoff Acquisition for Longitudinal Data](https://mlsanity.com/paper/2507.12412/) (2025)
- [Decision-Focused Learning: When and Why Traditional Prediction Models Fail](https://mlsanity.com/paper/2606.21773/) (2026)
- [Active Real-World Factor-Based Evaluation for Generalist Robot Policies](https://mlsanity.com/paper/2607.14439/) (2026)

## Abstract (arXiv)

When a robot faces unfamiliar physical conditions, a common approach is to collect evidence about what changed and adapt. For such diagnosis to improve behavior, six ordered empirical conditions must hold: a meaningful reference, identifiability of the physical condition, use of the acquired evidence, decision value, selection value over a fixed alternative, and safe realization. We test this chain in controlled and public environments. It holds end to end in our controlled environments. After transfer to unseen mechanisms, however, it breaks at evidence use. On decisions requiring the full trace, the frozen decoder does not change its choice. A linear model using only trace increments recovers the correct choice on mechanisms excluded from fitting, showing that the trace is informative but unused. The failure is concentrated at the richest evidence level: those decisions fall to chance, while decisions settled with lower-cost evidence remain correct, a split hidden by aggregate accuracy. The same chain can fail at other links in public environments. Successful physical identification therefore guarantees neither evidence use nor useful adaptation; evaluation should identify where the chain breaks rather than rely on recovery accuracy or aggregate performance alone.