mlsanityData to 7 Oct 2026

STAGE: Diagnosing Semantic Transfer at Grounded Execution in Embodied Agents

arXiv 2609.13458 ↗First posted 11 Sep 2026cs.RO cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Embodied language grounding requires more than identifying the referent of an instruction: recovered semantics must also control the action an agent exposes. We study this missing link as a semantic-action gap, where instruction semantics are recoverable but weakly expressed in native continuous actions. We introduce SAT-Bench, a fixed-observation counterfactual benchmark that holds the visual scene and agent state fixed while changing only instruction semantics. On LIBERO target-name and pixel-grounded relation swaps, target recovery reaches 100.0% and 95.8%, whereas OpenVLA action sensitivity remains only 6.8% and 7.7%. The gap persists across 1,000 additional compositional and temporal/procedural counterfactuals, with overall action sensitivity of 6.1%. Hidden-state, threshold-free, cross-policy, and rollout diagnostics further support this semantic-action transfer failure. We introduce VISA, a lightweight execution-time interface that converts recovered semantics into ALLOW, DEFER, target-consistency, and verified-selection decisions. VISA reduces invalid-instruction blind execution from 92.7% to 2.8% while preserving 94.0% of normal commands, and verified selection further improves target-consistent action exposure without updating the underlying policy. Overall, embodied language evaluation should measure semantic-action transfer, not semantic parsing alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

15 of 21 references.

References · 15

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  4. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  9. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  10. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  11. 2019Improving Grounded Natural Language Understanding through Human-Robot Dialogcited by 25
  12. 2025Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Makingcited by 19
3 more
  1. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activitiescited by 9
  2. 2025Do What? Teaching Vision-Language-Action Models to Reject the Impossiblecited by 7
  3. 2025HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Modelscited by 5

Built on by

Not cited yet.