mlsanityData to 7 Oct 2026

Show-Harness: Just a VLM Agent Can Play Robots

arXiv 2609.10522 ↗First posted 9 Sep 2026cs.RO cs.AI cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 99 of its 108 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

99 of 108 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026HumanCLAW: Can Vision-Language Models Act Through a Body?29 Jul
  2. 2026Learning Action Priors for Cross-embodiment Robot Manipulation24 Jun
  3. 2026APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies10 Jun
  4. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents8 Apr
  5. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics2 Jun
  6. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete28 Feb
  7. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents13 Feb
  8. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Models27 Jun
  9. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  11. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 88

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  11. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
38 more
  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  7. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  8. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  12. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  13. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  14. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  15. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  16. 2026World Action Models are Zero-shot Policiescited by 411
  17. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  18. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  19. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  20. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  21. 2024Latent Action Pretraining from Videoscited by 302
  22. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  23. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  24. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  25. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
  26. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  27. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  28. 2025Unified Video Action Modelcited by 195
  29. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulatorscited by 180
  30. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  31. 2024Behavior Generation with Latent Actionscited by 159
  32. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  33. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
  34. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  35. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  36. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  37. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  38. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026In-Context Robot Learning with VLM Agentscited by 4
  2. 2026Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokenscited by 2
  3. 2026HarnessPAI: An Evolving Harness for Physical AIcited by 1
  4. 2026Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentationcited by 1
  5. 2026World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsalcited by 1
  6. 2026UAVs Meet Embodied Intelligence: Bridging Human Intents and Flying Dynamics Via Harnessing Physical-Digital AI Agentscited by 0
  7. 2026MaskHarness-WAM: Instance-Grounded Harnessing for Long-Horizon Robot Manipulationcited by 0
  8. 2026RoboICL: Embodied In-Context Learning with GPT-6 Astracited by 0
  9. 2026MaLiang-Harness: A Programmable Path to Image and Video Generationcited by 0
  10. 2026MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulationcited by 0
  11. 2026SimEX: Simulation-Integrated Robotics AutoResearchcited by 0
  12. 2026Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Toolscited by 0
5 more
  1. 2026DynaHarness: A Dynamic Physical Harness for Self-Evolving Robot Agentscited by 0
  2. 2026RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodimentscited by 0
  3. 2026Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policiescited by 0
  4. 2026SuperNav: An Agentic Navigation System for Any Task in Any Scenecited by 0
  5. 2026Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Transferring the Intelligence of VLMs to Robotic Control
  2. 2025Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusion
  3. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
  4. 2025VLMgineer: Vision Language Models as Robotic Toolsmiths
  5. 2026EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents
  6. 2025Mechanistic interpretability for steering vision-language-action models
  7. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control
  8. 2026Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs