# MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation

arXiv: https://arxiv.org/abs/2606.20679 · HTML: https://mlsanity.com/paper/2606.20679/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 13 Jun 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 15 of 15)

- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning](https://mlsanity.com/paper/2601.16163/) (2026)
- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs](https://mlsanity.com/paper/2512.15692/) (2025)
- [Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation](https://mlsanity.com/paper/2508.05635/) (2025)
- [MemER: Scaling Up Memory for Robot Control via Experience Retrieval](https://mlsanity.com/paper/2510.20328/) (2025)
- [MEM: Multi-Scale Embodied Memory for Vision Language Action Models](https://mlsanity.com/paper/2603.03596/) (2026)
- [RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies](https://mlsanity.com/paper/2603.04639/) (2026)
- [World Model for Robot Learning: A Comprehensive Survey](https://mlsanity.com/paper/2605.00080/) (2026)
- [Gated Memory Policy: In-Context Memorization and Adaptation](https://mlsanity.com/paper/2604.18933/) (2026)
- [A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens](https://mlsanity.com/paper/2604.04913/) (2026)

## Built on by (most cited first, 2 of 2)

- [World-Action Models for Robot Learning and Control: A Survey](https://mlsanity.com/paper/2609.16074/) (2026)
- [SmoLSTM: A Compact Vision-Language-Action Model with Recurrent Memory that Persists](https://mlsanity.com/paper/2609.22854/) (2026)

## Abstract (arXiv)

Video-world-model policies learn action-relevant representations by predicting future observations. However, they condition on only a short observation window, which renders long-horizon manipulation non-Markovian when the correct action depends on earlier events that are no longer visible. We present MemoryVAM, an episodic memory mechanism for video-world-model policies. We employ a Recap-Cue (RC) module, in which a Perceiver-based Recap Compressor maps per-frame CLIP embeddings into compact memory tokens, and a lightweight Cue Gate estimates task completion from memory and language. These tokens are injected into both the video backbone and the action decoder, aligning policy imagination with episode progress and conditioning actions on history. Our model trains the memory module with video prediction, a delta-reconstruction auxiliary loss, and episode-boundary supervision, requiring no per-frame progress labels. The same mechanism applies to UNet and Diffusion Transformer (DiT) backbones by changing only the cross-attention injection interface. On LIBERO-Mem, our model improves average success from 5% to 42.5%. On real robots, it achieves 78.3% success on counting tasks, 80.0% on spatial recall, and 75.0% on sequential tracking. Project page: https://MemoryVAM.github.io/