# Show-Harness: Just a VLM Agent Can Play Robots

arXiv: https://arxiv.org/abs/2609.10522 · HTML: https://mlsanity.com/paper/2609.10522/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 99 of its 108 references that are in the map.
- First posted on arXiv 9 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 99)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026)

## Built on by (most cited first, 20 of 20)

- [In-Context Robot Learning with VLM Agents](https://mlsanity.com/paper/2609.19138/) (2026)
- [HarnessPAI: An Evolving Harness for Physical AI](https://mlsanity.com/paper/2609.29166/) (2026)
- [Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens](https://mlsanity.com/paper/2610.01939/) (2026)
- [Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation](https://mlsanity.com/paper/2609.29389/) (2026)
- [World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal](https://mlsanity.com/paper/2609.29964/) (2026)
- [SimEX: Simulation-Integrated Robotics AutoResearch](https://mlsanity.com/paper/2609.38982/) (2026)
- [MaLiang-Harness: A Programmable Path to Image and Video Generation](https://mlsanity.com/paper/2609.34309/) (2026)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools](https://mlsanity.com/paper/2609.39018/) (2026)
- [RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments](https://mlsanity.com/paper/2610.10409/) (2026)
- [Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies](https://mlsanity.com/paper/2610.10479/) (2026)
- [Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement](https://mlsanity.com/paper/2610.12369/) (2026)
- [UAVs Meet Embodied Intelligence: Bridging Human Intents and Flying Dynamics Via Harnessing Physical-Digital AI Agents](https://mlsanity.com/paper/2609.18326/) (2026)
- [MaskHarness-WAM: Instance-Grounded Harnessing for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2609.19974/) (2026)
- [RoboICL: Embodied In-Context Learning with GPT-6 Astra](https://mlsanity.com/paper/2609.34261/) (2026)
- [MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation](https://mlsanity.com/paper/2609.38078/) (2026)
- [DynaHarness: A Dynamic Physical Harness for Self-Evolving Robot Agents](https://mlsanity.com/paper/2609.40306/) (2026)
- [WAMJET: A Harness for World Action Model Acceleration](https://mlsanity.com/paper/2610.03797/) (2026)
- [PhysEvo: Astra Can Act, Let It](https://mlsanity.com/paper/2610.08995/) (2026)
- [SuperNav: An Agentic Navigation System for Any Task in Any Scene](https://mlsanity.com/paper/2610.12126/) (2026)

## Abstract (arXiv)

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.