# Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

arXiv: https://arxiv.org/abs/2607.08448 · HTML: https://mlsanity.com/paper/2607.08448/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 68 of its 77 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 68)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)

## Built on by (most cited first, 30 of 47)

- [Show-Harness: Just a VLM Agent Can Play Robots](https://mlsanity.com/paper/2609.10522/) (2026)
- [JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution](https://mlsanity.com/paper/2608.25593/) (2026)
- [Agent as Policy for Robotic Manipulation](https://mlsanity.com/paper/2609.12541/) (2026)
- [World-Action Models for Robot Learning and Control: A Survey](https://mlsanity.com/paper/2609.16074/) (2026)
- [Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memory](https://mlsanity.com/paper/2608.16889/) (2026)
- [HarnessPAI: An Evolving Harness for Physical AI](https://mlsanity.com/paper/2609.29166/) (2026)
- [Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens](https://mlsanity.com/paper/2610.01939/) (2026)
- [HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness](https://mlsanity.com/paper/2609.15195/) (2026)
- [Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation](https://mlsanity.com/paper/2609.20822/) (2026)
- [Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation](https://mlsanity.com/paper/2609.29389/) (2026)
- [RACaP: Agentic Reasoning, Acting, and Coding as Policies for Evolvable Robot Learning](https://mlsanity.com/paper/2609.29394/) (2026)
- [SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them](https://mlsanity.com/paper/2607.27703/) (2026)
- [CrystalMem: Elastic Memory for Self-Evolving LLM Agents via Knowledge Crystallization](https://mlsanity.com/paper/2608.00303/) (2026)
- [2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation](https://mlsanity.com/paper/2609.11308/) (2026)
- [OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning](https://mlsanity.com/paper/2609.16057/) (2026)
- [World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal](https://mlsanity.com/paper/2609.29964/) (2026)
- [SimEX: Simulation-Integrated Robotics AutoResearch](https://mlsanity.com/paper/2609.38982/) (2026)
- [Scale and Selection: What Makes Automatic Harness Evolution Work for Visual-Interface Robot Agents](https://mlsanity.com/paper/2609.39304/) (2026)
- [EmbodiedRSI: Active Continual Robot Learning Through Hypothesis-Guided Co-Evolution](https://mlsanity.com/paper/2610.10498/) (2026)
- [MobileMem: Learning from a Year of Mobile Experiences](https://mlsanity.com/paper/2608.13606/) (2026)
- [Think Only When Needed: Prompt-Authority Control for Selective Slow-Path Intervention in Vision-Language-Action Manipulation](https://mlsanity.com/paper/2608.23224/) (2026)
- [StageGuard: Learning Stage Transitions for Long-Horizon Robot Tasks via Agentic Distillation](https://mlsanity.com/paper/2609.20791/) (2026)
- [CHOREO: Every Humanoid Skill as a Trajectory](https://mlsanity.com/paper/2609.22274/) (2026)
- [DualManip: Agentic Dynamic Manipulation via Dual-Path Semantic Reasoning and Geometric Adaptation](https://mlsanity.com/paper/2609.31112/) (2026)
- [RoboFoundry: System-as-Policy Evolution for Self-Learning Embodied Agents](https://mlsanity.com/paper/2609.32862/) (2026)
- [Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence](https://mlsanity.com/paper/2609.35432/) (2026)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement](https://mlsanity.com/paper/2610.12369/) (2026)
- [RoboHarness: A Simple Harness Outperforms VLA and World Actions Models](https://mlsanity.com/paper/2610.14018/) (2026)
- [Skills in Weights, Memory in Code: Hybrid Learning for Memory-Dependent Robot Manipulation](https://mlsanity.com/paper/2608.09410/) (2026)

## Abstract (arXiv)

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R. Real-world demonstrations on dual-Franka robots further show target redirection, grasp recovery, and new task compositions with the same frozen VLA. Code is available at https://github.com/RLinf/RPent.