# MEM: Multi-Scale Embodied Memory for Vision Language Action Models

arXiv: https://arxiv.org/abs/2603.03596 · HTML: https://mlsanity.com/paper/2603.03596/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning and chain-of-thought in VLA, one of 13 directions in Vision-language-action models.
- Builds on 45 of its 54 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- Vision-language-action models / [Reasoning and chain-of-thought in VLA](https://mlsanity.com/vla/reasoning-and-chain-of-thought-in-vla/)

## Builds on (most cited first, 30 of 45)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [3D-VLA: A 3D Vision-Language-Action Generative World Model](https://mlsanity.com/paper/2403.09631/) (2024)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies](https://mlsanity.com/paper/2412.10345/) (2024)
- [Real-Time Execution of Action Chunking Flow Policies](https://mlsanity.com/paper/2506.07339/) (2025)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control](https://mlsanity.com/paper/2502.05855/) (2025)
- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [Behavior Generation with Latent Actions](https://mlsanity.com/paper/2403.03181/) (2024)
- [Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better](https://mlsanity.com/paper/2505.23705/) (2025)
- [A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation](https://mlsanity.com/paper/2507.05331/) (2025)
- [A Comprehensive Survey on Long Context Language Modeling](https://mlsanity.com/paper/2503.17407/) (2025)
- [Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation](https://mlsanity.com/paper/2408.11812/) (2024)

## Built on by (most cited first, 30 of 61)

- [π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities](https://mlsanity.com/paper/2604.15483/) (2026)
- [EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies](https://mlsanity.com/paper/2606.20092/) (2026)
- [RoboTTT: Context Scaling for Robot Policies](https://mlsanity.com/paper/2607.15275/) (2026)
- [G0.5: One Autoregressive Stream for Robot Reasoning and Action](https://mlsanity.com/paper/2608.11739/) (2026)
- [MemoryWAM: Efficient World Action Modeling with Persistent Memory](https://mlsanity.com/paper/2606.20562/) (2026)
- [Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection](https://mlsanity.com/paper/2604.13942/) (2026)
- [MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models](https://mlsanity.com/paper/2606.09827/) (2026)
- [Long-Horizon Manipulation via Trace-Conditioned VLA Planning](https://mlsanity.com/paper/2604.21924/) (2026)
- [Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring](https://mlsanity.com/paper/2605.30834/) (2026)
- [μVLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models](https://mlsanity.com/paper/2606.12497/) (2026)
- [VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory](https://mlsanity.com/paper/2603.04910/) (2026)
- [Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models](https://mlsanity.com/paper/2606.11324/) (2026)
- [DAM-VLA: Decoupled Asynchronous Multimodal Vision Language Action model](https://mlsanity.com/paper/2606.12105/) (2026)
- [NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2607.06678/) (2026)
- [WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory and In-Context Learning](https://mlsanity.com/paper/2607.18840/) (2026)
- [World-Action Models for Robot Learning and Control: A Survey](https://mlsanity.com/paper/2609.16074/) (2026)
- [DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks](https://mlsanity.com/paper/2604.16484/) (2026)
- [Vesta: A Generalist Embodied Reasoning Model](https://mlsanity.com/paper/2606.20905/) (2026)
- [DIM-WAM: World-Action Modeling with Diverse Historical Event Memory](https://mlsanity.com/paper/2606.27677/) (2026)
- [Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation](https://mlsanity.com/paper/2607.05377/) (2026)
- [τ0-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation](https://mlsanity.com/paper/2608.16885/) (2026)
- [Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memory](https://mlsanity.com/paper/2608.16889/) (2026)
- [OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism](https://mlsanity.com/paper/2603.14371/) (2026)
- [ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine](https://mlsanity.com/paper/2607.28625/) (2026)
- [SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models](https://mlsanity.com/paper/2609.05533/) (2026)
- [ECHO: Continuous Hierarchical Memory for Vision-Language-Action Models](https://mlsanity.com/paper/2605.10993/) (2026)
- [AURA: Action-Gated Memory for Robot Policies at Constant VRAM](https://mlsanity.com/paper/2606.02775/) (2026)
- [MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation](https://mlsanity.com/paper/2606.20679/) (2026)
- [FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptation](https://mlsanity.com/paper/2606.20867/) (2026)
- [FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation](https://mlsanity.com/paper/2607.18231/) (2026)

## Abstract (arXiv)

Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.