mlsanityData to 7 Oct 2026

Visual Grounding in Zero-Shot Vision-Language Control

arXiv 2608.06154 ↗First posted 6 Aug 2026cs.RO cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) are increasingly used as zero-shot controllers, but successful trajectories do not necessarily show that decisions are grounded in visual input: simulator dynamics and conservative action priors can produce favourable scores without meaningful perception. We investigate this with an input-ablation battery: blind-image controls, repeated identical inputs, lane-axis reflection, non-visual baselines, and pipeline-integrity checks. Across nine direct-action models, six structured local VLMs, and an exploratory VLM-MPC hierarchy, we analyse 32,874 scored calls over two embodiments and three simulators. The direct-control results are largely negative: a constant-SLOW policy outperforms a scripted geometric controller, several models are image-invariant or nearly constant, and models that recognize longitudinal hazards still fail to transform LEFT and RIGHT under reflection. No local VLM meets the joint longitudinal and lateral grounding criteria. However, an image-only deterministic positive control estimates the lead gap with 0.090 m MAE and exact mirror equivariance, confirming the stimuli carry sufficient visual information; the failures are modular, not universal. A post-hoc, leakage-controlled symmetry-consensus guardian selects two models from 16 calibration frames and freezes a 2-of-4 hazard vote across original and reflected views. On 272 held-out frames it reaches 0.954 balanced accuracy (episode-cluster bootstrap 95% CI [0.895,0.990]); nested leave-one-episode-out recovers the same pair and threshold in all 12 folds. Abstaining on ties raises committed balanced accuracy to 0.973 at 0.824 coverage. With deterministic perception retaining lateral authority, offline modular replay achieves 0.934 action agreement and exact mirror equivariance. These results support current VLMs as bounded, selective hazard assistants, not monolithic zero-shot controllers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 44 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

22 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models30 Mar

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2025Qwen3-VL Technical Reportcited by 2,534
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  9. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  12. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
9 more
  1. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  2. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  3. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  4. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  5. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  6. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  7. 2021Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Controlcited by 68
  8. 2025MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigationcited by 55
  9. 2026Semantic Invariance in Agentic AIcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026World Pilot: Steering Vision-Language-Action Models with World-Action Priors
  2. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models
  3. 2025Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Grounding
  4. 2026Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
  5. 2026SCALE: Self-uncertainty Conditioned Adaptive Looking and Execution for Vision-Language-Action Models
  6. 2026C2Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation
  7. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
  8. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models