# 2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation arXiv: https://arxiv.org/abs/2609.11308 · HTML: https://mlsanity.com/paper/2609.11308/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 31 references that are in the map. - First posted on arXiv 10 Sep 2026. ## Builds on (most cited first, 30 of 30) - [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024) - [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024) - [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022) - [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025) - [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022) - [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023) - [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024) - [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025) - [RT-H: Action Hierarchies Using Language](https://mlsanity.com/paper/2403.01823/) (2024) - [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025) - [HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation](https://mlsanity.com/paper/2502.05485/) (2025) - [AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench](https://mlsanity.com/paper/2507.02554/) (2025) - [MemER: Scaling Up Memory for Robot Control via Experience Retrieval](https://mlsanity.com/paper/2510.20328/) (2025) - [RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies](https://mlsanity.com/paper/2603.04639/) (2026) - [Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents](https://mlsanity.com/paper/2607.08448/) (2026) - [RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design](https://mlsanity.com/paper/2603.01229/) (2026) - [RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks](https://mlsanity.com/paper/2603.11558/) (2026) - [Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agents](https://mlsanity.com/paper/2505.23450/) (2025) - [RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark](https://mlsanity.com/paper/2605.10921/) (2026) - [Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective](https://mlsanity.com/paper/2511.11478/) (2025) - [Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control](https://mlsanity.com/paper/2602.13193/) (2026) - [Scaling World Model for Hierarchical Manipulation Policies](https://mlsanity.com/paper/2602.10983/) (2026) - [AIRA_2: Overcoming Bottlenecks in AI Research Agents](https://mlsanity.com/paper/2603.26499/) (2026) - [Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection](https://mlsanity.com/paper/2604.13942/) (2026) - [Long-Horizon Manipulation via Trace-Conditioned VLA Planning](https://mlsanity.com/paper/2604.21924/) (2026) - [ETA: A New Agentic Paradigm for Embodied Tasks](https://mlsanity.com/paper/2608.03924/) (2026) - [FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies](https://mlsanity.com/paper/2605.27284/) (2026) - [CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models](https://mlsanity.com/paper/2604.22238/) (2026) - [Continue or Replan? Bernoulli-Continuation Policy Learning for Adaptive Horizon Execution](https://mlsanity.com/paper/2608.03483/) (2026) - [SparkVLA: Stop-Aware Hierarchical VLA with Adaptive Action Chunking for Long-Horizon Manipulation](https://mlsanity.com/paper/2608.16172/) (2026) ## Built on by (most cited first, 2 of 2) - [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026) - [SmoLSTM: A Compact Vision-Language-Action Model with Recurrent Memory that Persists](https://mlsanity.com/paper/2609.22854/) (2026) ## Abstract (arXiv) Long-horizon robot manipulation requires memory, but not necessarily inside the action policy. To address such tasks, current agentic systems often combine VLAs with planners and geometric tools, sometimes using additional depth or calibrated geometry. These systems confound attribution: gains may come from richer observations or alternative motor tools, while failures may stem from either the policy or an under-specified language interface. We isolate this question through a deliberately constrained design: less tool breadth, but greater interface bandwidth. 2AM makes a multimodal Agent the sole holder of task memory and a single RGB-based, episodically stateless Action Model the sole executor of task-relevant motion. The Agent compiles interaction history into subtask language and optional 2D grasp, place, and move hints that bind its physical intention at different time scales. To teach this steerability to the VLA, we augment demonstrations with structured hint labels and train under condition dropout, spatial noise, and temporal jitter to tolerate imperfect Agent outputs. On LIBERO-Mem, without depth, online geometry, or planner-based object motion, 2AM reaches 76.3% average completion, a 61.5-point improvement over the strongest reported baseline of 14.8%, together with 63.0% relaxed and 11.8% strict success. These results show that task memory can remain Agent-side. They further show that Action Model capability depends not only on what the policy has learned, but on how precisely the Agent can steer it.