# STAGE: Diagnosing Semantic Transfer at Grounded Execution in Embodied Agents

arXiv: https://arxiv.org/abs/2609.13458 · HTML: https://mlsanity.com/paper/2609.13458/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 11 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Improving Grounded Natural Language Understanding through Human-Robot Dialog](https://mlsanity.com/paper/1903.00122/) (2019)
- [Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making](https://mlsanity.com/paper/2505.19933/) (2025)
- [ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities](https://mlsanity.com/paper/2410.03907/) (2024)
- [Do What? Teaching Vision-Language-Action Models to Reject the Impossible](https://mlsanity.com/paper/2508.16292/) (2025)
- [HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models](https://mlsanity.com/paper/2506.15065/) (2025)

## Abstract (arXiv)

Embodied language grounding requires more than identifying the referent of an instruction: recovered semantics must also control the action an agent exposes. We study this missing link as a semantic-action gap, where instruction semantics are recoverable but weakly expressed in native continuous actions. We introduce SAT-Bench, a fixed-observation counterfactual benchmark that holds the visual scene and agent state fixed while changing only instruction semantics. On LIBERO target-name and pixel-grounded relation swaps, target recovery reaches 100.0% and 95.8%, whereas OpenVLA action sensitivity remains only 6.8% and 7.7%. The gap persists across 1,000 additional compositional and temporal/procedural counterfactuals, with overall action sensitivity of 6.1%. Hidden-state, threshold-free, cross-policy, and rollout diagnostics further support this semantic-action transfer failure. We introduce VISA, a lightweight execution-time interface that converts recovered semantics into ALLOW, DEFER, target-consistency, and verified-selection decisions. VISA reduces invalid-instruction blind execution from 92.7% to 2.8% while preserving 94.0% of normal commands, and verified selection further improves target-consistent action exposure without updating the underlying policy. Overall, embodied language evaluation should measure semantic-action transfer, not semantic parsing alone.