# Deep Episodic Memory: Encoding, Recalling, and Predicting Episodic Experiences for Robot Action Execution

arXiv: https://arxiv.org/abs/1801.04134 · HTML: https://mlsanity.com/paper/1801.04134/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jan 2018.

## Built on by (most cited first, 11 of 11)

- [Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos](https://mlsanity.com/paper/2103.16817/) (2021)
- [Humanoid Policy ~ Human Policy](https://mlsanity.com/paper/2503.13441/) (2025)
- [A Memory System of a Robot Cognitive Architecture and its Implementation in ArmarX](https://mlsanity.com/paper/2206.02241/) (2022)
- [Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts](https://mlsanity.com/paper/2407.14872/) (2024)
- [Video-EM: Event-Centric Episodic Memory for Long-Form Video Understanding](https://mlsanity.com/paper/2508.09486/) (2025)
- [Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experience](https://mlsanity.com/paper/2409.17702/) (2024)
- [MemoGuard: An Adaptive Runtime for Guarding Against Memory Traps in Communication-Limited Robot Navigation](https://mlsanity.com/paper/2607.15589/) (2026)
- [Predictive Event Segmentation and Representation with Neural Networks: A Self-Supervised Model Assessed by Psychological Experiments](https://mlsanity.com/paper/2210.05710/) (2022)
- [Cognitive architecture aided by working-memory for self-supervised multi-modal humans recognition](https://mlsanity.com/paper/2103.09072/) (2021)
- [Visual Episodic Memory-based Exploration](https://mlsanity.com/paper/2405.11298/) (2024)
- [Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deployment](https://mlsanity.com/paper/2604.11306/) (2026)

## Abstract (arXiv)

We present a novel deep neural network architecture for representing robot experiences in an episodic-like memory which facilitates encoding, recalling, and predicting action experiences. Our proposed unsupervised deep episodic memory model 1) encodes observed actions in a latent vector space and, based on this latent encoding, 2) infers most similar episodes previously experienced, 3) reconstructs original episodes, and 4) predicts future frames in an end-to-end fashion. Results show that conceptually similar actions are mapped into the same region of the latent vector space. Based on these results, we introduce an action matching and retrieval mechanism, benchmark its performance on two large-scale action datasets, 20BN-something-something and ActivityNet and evaluate its generalization capability in a real-world scenario on a humanoid robot.