# Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness arXiv: https://arxiv.org/abs/2606.18363 · HTML: https://mlsanity.com/paper/2606.18363/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 33 references that are in the map. - First posted on arXiv 16 Jun 2026. ## Builds on (most cited first, 28 of 28) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024) - [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023) - [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025) - [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022) - [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025) - [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023) - [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024) - [R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization](https://mlsanity.com/paper/2503.10615/) (2025) - [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024) - [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024) - [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024) - [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023) - [LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization](https://mlsanity.com/paper/2510.03827/) (2025) - [Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition](https://mlsanity.com/paper/2307.14535/) (2023) - [HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation](https://mlsanity.com/paper/2502.05485/) (2025) - [GenSim: Generating Robotic Simulation Tasks via Large Language Models](https://mlsanity.com/paper/2310.01361/) (2023) - [CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation](https://mlsanity.com/paper/2603.22435/) (2026) - [AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents](https://mlsanity.com/paper/2401.12963/) (2024) - [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024) - [OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints](https://mlsanity.com/paper/2501.03841/) (2025) - [GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs](https://mlsanity.com/paper/2410.03645/) (2024) - [LLaVA-KD: A Framework of Distilling Multimodal Large Language Models](https://mlsanity.com/paper/2410.16236/) (2024) - [VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation](https://mlsanity.com/paper/2510.23497/) (2025) - [MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation](https://mlsanity.com/paper/2602.11337/) (2026) - [Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robots](https://mlsanity.com/paper/2511.00917/) (2025) - [RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning](https://mlsanity.com/paper/2505.03238/) (2025) - [HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interaction](https://mlsanity.com/paper/2506.20566/) (2025) ## Built on by (most cited first, 10 of 10) - [Show-Harness: Just a VLM Agent Can Play Robots](https://mlsanity.com/paper/2609.10522/) (2026) - [Towards the Harness of Embodied Agents](https://mlsanity.com/paper/2608.11246/) (2026) - [Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation](https://mlsanity.com/paper/2609.29389/) (2026) - [SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them](https://mlsanity.com/paper/2607.27703/) (2026) - [CHOREO: Every Humanoid Skill as a Trajectory](https://mlsanity.com/paper/2609.22274/) (2026) - [ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordination](https://mlsanity.com/paper/2609.24526/) (2026) - [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026) - [From Rollout to Reset: A Graph-Based Harness for Autonomous Long-Horizon Manipulation Evaluation](https://mlsanity.com/paper/2609.19413/) (2026) - [MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation](https://mlsanity.com/paper/2609.38078/) (2026) - [PhysEvo: Astra Can Act, Let It](https://mlsanity.com/paper/2610.08995/) (2026) ## Abstract (arXiv) Language models trained on large-scale vision-language data have demonstrated strong potential for embodied agents. Harnessing models through embodied tools use offers a promising alternative to end-to-end vision-language-action systems by combining high-level reasoning with external modules for perception, planning, and control. However, it remains unclear what makes an effective harness for embodied manipulation, and to what extent such a harness can unlock embodied capabilities in a wide range of reasoning models. In this work, we present Guava, a harness framework for embodied tool use developed through systematic exploration of the design space of agent workflows, action spaces, and observation spaces. Our study identifies three key ingredients for effective embodied agents: iterative perception-reasoning-action loops, semantic action abstractions, and multimodal observations. To understand whether these design principles are universal even to small models, we develop an end-to-end training pipeline that distills embodied manipulation capabilities into a 4B open-source model using fewer than 2K trajectories collected entirely in simulation. Experimental results in both simulation and real-world environments show performance comparable to frontier proprietary models while exhibiting strong generalization to unseen objects, novel instructions, and long-horizon tasks. Results suggest that a well-designed harness can serve as a scalable, model-agnostic interface for embodied manipulation, enabling strong emergent embodied capabilities in compact open-source models with minimal training data.