# UniMem: Unifying Multimodal Memory and Control for Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2608.22869 · HTML: https://mlsanity.com/paper/2608.22869/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning and chain-of-thought in VLA, one of 13 directions in Vision-language-action models.
- Builds on 38 of its 41 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Placed in

- Vision-language-action models / [Reasoning and chain-of-thought in VLA](https://mlsanity.com/vla/reasoning-and-chain-of-thought-in-vla/)

## Builds on (most cited first, 30 of 38)

- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [LongVILA: Scaling Long-Context Visual Language Models for Long Videos](https://mlsanity.com/paper/2408.10188/) (2024)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [Real-Time Execution of Action Chunking Flow Policies](https://mlsanity.com/paper/2506.07339/) (2025)
- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [Behavior Generation with Latent Actions](https://mlsanity.com/paper/2403.03181/) (2024)
- [π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities](https://mlsanity.com/paper/2604.15483/) (2026)
- [Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks](https://mlsanity.com/paper/1903.03878/) (2019)
- [OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning](https://mlsanity.com/paper/2505.11917/) (2025)
- [Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration](https://mlsanity.com/paper/1707.02920/) (2017)
- [Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs](https://mlsanity.com/paper/2407.07775/) (2024)
- [Feedback in Imitation Learning: The Three Regimes of Covariate Shift](https://mlsanity.com/paper/2102.02872/) (2021)
- [MemER: Scaling Up Memory for Robot Control via Experience Retrieval](https://mlsanity.com/paper/2510.20328/) (2025)
- [SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation](https://mlsanity.com/paper/2501.18564/) (2025)
- [MEM: Multi-Scale Embodied Memory for Vision Language Action Models](https://mlsanity.com/paper/2603.03596/) (2026)
- [RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design](https://mlsanity.com/paper/2603.01229/) (2026)
- [TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Models](https://mlsanity.com/paper/2509.07962/) (2025)

## Abstract (arXiv)

While Vision-Language-Action (VLA) models have leveraged internet-scale pretraining and task-focused finetuning to achieve strong performance on long-horizon tasks, they often struggle with non-Markovian tasks that require memory. Existing approaches to memory typically involve additional Vision-Language-Models (VLMs) for long-term memory management, introducing a memory bottleneck and a fractured training pipeline. Conditioning on multiple historical frames can provide the VLA with access to more descriptive features of past scenes, but can degrade performance if frames are chosen at arbitrary, fixed intervals. To address these limitations, we present UniMem, a framework that unifies high-level, multimodal memory and low-level control under one backbone. UniMem employs an event classifier for memory updates, a keyframe encoder for dense spatial memory, and a keyframe caching technique to minimize overhead during policy rollouts. We evaluate UniMem across five simulation and four hardware tasks targeting sequential and spatial memory, demonstrating that our unified, single-model system outperforms fixed-interval image sampling baselines (93.4% vs. 68.2%) in simulation and hierarchical baselines (80.0% vs. 43.5%) in hardware, while offering faster inference and a simple training pipeline for easy adoption. Project website: https://losterberg3.github.io/unimem-vla/