# MemER: Scaling Up Memory for Robot Control via Experience Retrieval

arXiv: https://arxiv.org/abs/2510.20328 · HTML: https://mlsanity.com/paper/2510.20328/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies](https://mlsanity.com/paper/2412.10345/) (2024)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023)
- [HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation](https://mlsanity.com/paper/2502.05485/) (2025)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [M-LLM Based Video Frame Selection for Efficient Video Understanding](https://mlsanity.com/paper/2502.19680/) (2025)
- [SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation](https://mlsanity.com/paper/2501.18564/) (2025)
- [VLind-Bench: Measuring Language Priors in Large Vision-Language Models](https://mlsanity.com/paper/2406.08702/) (2024)
- [MLLMs are Deeply Affected by Modality Bias](https://mlsanity.com/paper/2505.18657/) (2025)
- [RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction](https://mlsanity.com/paper/2509.07953/) (2025)
- [AMEGO: Active Memory from long EGOcentric videos](https://mlsanity.com/paper/2409.10917/) (2024)
- [From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control](https://mlsanity.com/paper/2405.04798/) (2024)
- [Learning Long-Context Diffusion Policies via Past-Token Prediction](https://mlsanity.com/paper/2505.09561/) (2025)
- [Understanding Long Videos with Multimodal Language Models](https://mlsanity.com/paper/2403.16998/) (2024)
- [Language-Embedded Gaussian Splats (LEGS): Incrementally Building Room-Scale Representations with a Mobile Robot](https://mlsanity.com/paper/2409.18108/) (2024)
- [Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models](https://mlsanity.com/paper/2407.02666/) (2024)
- [Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory](https://mlsanity.com/paper/2411.16934/) (2024)
- [Semantic Mechanical Search with Large Vision and Language Models](https://mlsanity.com/paper/2302.12915/) (2023)
- [Towards Fast, Memory-based and Data-Efficient Vision-Language Policy](https://mlsanity.com/paper/2503.10322/) (2025)

## Built on by (most cited first, 30 of 67)

- [Causal World Modeling for Robot Control](https://mlsanity.com/paper/2601.21998/) (2026)
- [π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities](https://mlsanity.com/paper/2604.15483/) (2026)
- [MEM: Multi-Scale Embodied Memory for Vision Language Action Models](https://mlsanity.com/paper/2603.03596/) (2026)
- [RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design](https://mlsanity.com/paper/2603.01229/) (2026)
- [Native Video-Action Pretraining for Generalizable Robot Control](https://mlsanity.com/paper/2607.08639/) (2026)
- [ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries](https://mlsanity.com/paper/2603.12942/) (2026)
- [RLDX-1 Technical Report](https://mlsanity.com/paper/2605.03269/) (2026)
- [EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies](https://mlsanity.com/paper/2606.20092/) (2026)
- [RoboTTT: Context Scaling for Robot Policies](https://mlsanity.com/paper/2607.15275/) (2026)
- [BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames](https://mlsanity.com/paper/2602.15010/) (2026)
- [Gated Memory Policy: In-Context Memorization and Adaptation](https://mlsanity.com/paper/2604.18933/) (2026)
- [MemoryWAM: Efficient World Action Modeling with Persistent Memory](https://mlsanity.com/paper/2606.20562/) (2026)
- [Wall-OSS-0.5 Technical Report](https://mlsanity.com/paper/2605.30877/) (2026)
- [Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection](https://mlsanity.com/paper/2604.13942/) (2026)
- [MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models](https://mlsanity.com/paper/2606.09827/) (2026)
- [Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring](https://mlsanity.com/paper/2605.30834/) (2026)
- [Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chaining](https://mlsanity.com/paper/2603.01465/) (2026)
- [HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System](https://mlsanity.com/paper/2604.14125/) (2026)
- [HiMem-WAM: Hierarchical Memory-Gated World Action Models for Robotic Manipulation](https://mlsanity.com/paper/2606.10363/) (2026)
- [μVLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models](https://mlsanity.com/paper/2606.12497/) (2026)
- [VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory](https://mlsanity.com/paper/2603.04910/) (2026)
- [DAM-VLA: Decoupled Asynchronous Multimodal Vision Language Action model](https://mlsanity.com/paper/2606.12105/) (2026)
- [NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2607.06678/) (2026)
- [KEMO: Event-Driven Keyframe Memory for Long-Horizon Robot Manipulation with VLA Policies](https://mlsanity.com/paper/2606.23589/) (2026)
- [Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation](https://mlsanity.com/paper/2607.05377/) (2026)
- [τ0-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation](https://mlsanity.com/paper/2608.16885/) (2026)
- [Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memory](https://mlsanity.com/paper/2608.16889/) (2026)
- [Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action](https://mlsanity.com/paper/2605.22283/) (2026)
- [Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2607.07608/) (2026)
- [RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation](https://mlsanity.com/paper/2603.24060/) (2026)

## Abstract (arXiv)

Humans routinely rely on memory to perform tasks, yet most robot policies lack this capability; our goal is to endow robot policies with the same ability. Naively conditioning on long observation histories is computationally expensive and brittle under covariate shift, while indiscriminate subsampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instructions for a low-level policy to execute. This design is compatible with existing vision-language-action (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, we finetune Qwen2.5-VL-7B-Instruct and π0.5 as the high-level and low-level policies respectively, using demonstrations supplemented with minimal language annotations. Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory. Videos and code can be found at https://jen-pan.github.io/memer/.