# Skills in Weights, Memory in Code: Hybrid Learning for Memory-Dependent Robot Manipulation

arXiv: https://arxiv.org/abs/2608.09410 · HTML: https://mlsanity.com/paper/2608.09410/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning and chain-of-thought in VLA, one of 13 directions in Vision-language-action models.
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Placed in

- Vision-language-action models / [Reasoning and chain-of-thought in VLA](https://mlsanity.com/vla/reasoning-and-chain-of-thought-in-vla/)

## Builds on (most cited first, 23 of 23)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance](https://mlsanity.com/paper/2410.13816/) (2024)
- [Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents](https://mlsanity.com/paper/2607.08448/) (2026)
- [DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance](https://mlsanity.com/paper/2506.13922/) (2025)
- [Inference-Time Policy Steering through Human Interactions](https://mlsanity.com/paper/2411.16627/) (2024)
- [RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark](https://mlsanity.com/paper/2605.10921/) (2026)
- [EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies](https://mlsanity.com/paper/2606.20092/) (2026)
- [RHO: Your Coding Agent is Secretly a Roboticist](https://mlsanity.com/paper/2606.16458/) (2026)
- [TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance](https://mlsanity.com/paper/2601.20239/) (2026)
- [VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation](https://mlsanity.com/paper/2606.07723/) (2026)
- [Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation](https://mlsanity.com/paper/2603.04466/) (2026)
- [Demonstration-Free Robotic Control via LLM Agents](https://mlsanity.com/paper/2601.20334/) (2026)
- [HiMem-WAM: Hierarchical Memory-Gated World Action Models for Robotic Manipulation](https://mlsanity.com/paper/2606.10363/) (2026)
- [μVLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models](https://mlsanity.com/paper/2606.12497/) (2026)
- [Resolving State Ambiguity in Robot Manipulation via Adaptive Working Memory Recoding](https://mlsanity.com/paper/2512.24638/) (2025)
- [Chronos: A Physics-Informed Full-History Framework for Non-Markovian Long-Horizon Manipulation](https://mlsanity.com/paper/2606.30318/) (2026)
- [Action-Effect Memory Pretraining for Robot Manipulation](https://mlsanity.com/paper/2606.12499/) (2026)

## Abstract (arXiv)

Modern vision-language-action (VLA) policies have acquired broad manipulation skills, but typically generate each action chunk from the current observation or a short fixed-length history. However, real-world manipulation is often non-Markovian, requiring robots to retain and reason over task-relevant information from long-horizon interaction histories to determine the next action. To address this challenge, we propose HyMeS, a hybrid learning framework that leverages the reasoning and memory-management capabilities of coding agents to steer a Markovian VLA for memory-dependent manipulation. Specifically, HyMeS learns low-level motor skills through gradient-based imitation learning, while a coding agent acquires high-level memory-management strategies through heuristic learning by iteratively updating an executable heuristic system from rollout feedback. Furthermore, we close the loop between steering and execution through multimodal stage-completion verification, which updates memory using proprioceptive signals and multi-frame VLM judgments. Compared with end-to-end memory-augmented VLAs, HyMeS requires demonstrations only for reusable motor skills rather than for every history-dependent task configuration, enabling data-efficient compositional generalization. On RoboMemArena, HyMeS improves mean cumulative success from 52.5% to 66.2% and mean task success from 41.3% to 60.1% over pi0.5, while outperforming PrediMem by 4.5 points in cumulative success and 14.5 points in task success.