# Human-Inspired Context-Selective Multimodal Memory for Social Robots

arXiv: https://arxiv.org/abs/2604.12081 · HTML: https://mlsanity.com/paper/2604.12081/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 59 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory](https://mlsanity.com/paper/2306.03901/) (2023)
- [jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval](https://mlsanity.com/paper/2506.18902/) (2025)
- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents](https://mlsanity.com/paper/2308.01542/) (2023)
- [LLM as A Robotic Brain: Unifying Egocentric Memory and Control](https://mlsanity.com/paper/2304.09349/) (2023)
- [MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2309.04790/) (2023)
- [AMNet: Memorability Estimation with Attention](https://mlsanity.com/paper/1804.03115/) (2018)
- [VITA: A Multi-modal LLM-based System for Longitudinal, Autonomous, and Adaptive Robotic Mental Well-being Coaching](https://mlsanity.com/paper/2312.09740/) (2023)
- [A Memory System of a Robot Cognitive Architecture and its Implementation in ArmarX](https://mlsanity.com/paper/2206.02241/) (2022)
- [Embracing New Techniques in Deep Learning for Estimating Image Memorability](https://mlsanity.com/paper/2105.10598/) (2021)
- [Image Memorability Prediction with Vision Transformers](https://mlsanity.com/paper/2301.08647/) (2023)
- [Memorability: An image-computable measure of information utility](https://mlsanity.com/paper/2104.00805/) (2021)
- [Nadine: An LLM-driven Intelligent Social Robot with Affective Capabilities and Human-like Memory](https://mlsanity.com/paper/2405.20189/) (2024)
- [MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments](https://mlsanity.com/paper/2402.00290/) (2024)
- [OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis](https://mlsanity.com/paper/2506.02891/) (2025)
- [A Machine With Human-Like Memory Systems](https://mlsanity.com/paper/2204.01611/) (2022)
- [Towards Multimodal Social Conversations with Robots: Using Vision-Language Models](https://mlsanity.com/paper/2507.19196/) (2025)

## Abstract (arXiv)

Memory is fundamental to social interaction, enabling humans to recall meaningful past experiences and adapt their behavior accordingly based on the context. However, most current social robots and embodied agents rely on non-selective, text-based memory, limiting their ability to support personalized, context-aware interactions. Drawing inspiration from cognitive neuroscience, we propose a context-selective, multimodal memory architecture for social robots that captures and retrieves both textual and visual episodic traces, prioritizing moments characterized by high emotional salience or scene novelty. By associating these memories with individual users, our system enables socially personalized recall and more natural, grounded dialogue. We evaluate the selective storage mechanism using a curated dataset of social scenarios, achieving a Spearman correlation of 0.506, surpassing human consistency (ρ=0.415) and outperforming existing image memorability models. In multimodal retrieval experiments, our fusion approach improves Recall@1 by up to 13% over unimodal text or image retrieval. Runtime evaluations confirm that the system maintains real-time performance. Qualitative analyses further demonstrate that the proposed framework produces richer and more socially relevant responses than baseline models. This work advances memory design for social robots by bridging human-inspired selectivity and multimodal retrieval to enhance long-term, personalized human-robot interaction.