# Encode-Store-Retrieve: Augmenting Human Memory through Language-Encoded Egocentric Perception

arXiv: https://arxiv.org/abs/2308.05822 · HTML: https://mlsanity.com/paper/2308.05822/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 60 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 15 of 15)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video](https://mlsanity.com/paper/2302.00402/) (2023)
- [Just Ask:An Interactive Learning Framework for Vision and Language Navigation](https://mlsanity.com/paper/1912.00915/) (2019)
- [Self-Attentive Associative Memory](https://mlsanity.com/paper/2002.03519/) (2020)
- [Learning to Rehearse in Long Sequence Memorization](https://mlsanity.com/paper/2106.01096/) (2021)
- [Inclusive AR/VR: Accessibility Barriers for Immersive Technologies](https://mlsanity.com/paper/2304.13465/) (2023)

## Built on by (most cited first, 9 of 9)

- [ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices](https://mlsanity.com/paper/2507.21378/) (2025)
- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understanding](https://mlsanity.com/paper/2508.12687/) (2025)
- [SpeechLess: Micro-utterance with Personalized Spatial Memory-aware Assistant in Everyday Augmented Reality](https://mlsanity.com/paper/2602.00793/) (2026)
- [Personal Visual Context Learning in Large Multimodal Models](https://mlsanity.com/paper/2605.10936/) (2026)
- [Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistant](https://mlsanity.com/paper/2601.17622/) (2026)
- [Enhancing Conversational Agents via Task-Oriented Adversarial Memory Adaptation](https://mlsanity.com/paper/2601.21797/) (2026)
- [From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality](https://mlsanity.com/paper/2602.03059/) (2026)
- [Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge](https://mlsanity.com/paper/2602.22455/) (2026)

## Abstract (arXiv)

We depend on our own memory to encode, store, and retrieve our experiences. However, memory lapses can occur. One promising avenue for achieving memory augmentation is through the use of augmented reality head-mounted displays to capture and preserve egocentric videos, a practice commonly referred to as lifelogging. However, a significant challenge arises from the sheer volume of video data generated through lifelogging, as the current technology lacks the capability to encode and store such large amounts of data efficiently. Further, retrieving specific information from extensive video archives requires substantial computational power, further complicating the task of quickly accessing desired content. To address these challenges, we propose a memory augmentation agent that involves leveraging natural language encoding for video data and storing them in a vector database. This approach harnesses the power of large vision language models to perform the language encoding process. Additionally, we propose using large language models to facilitate natural language querying. Our agent underwent extensive evaluation using the QA-Ego4D dataset and achieved state-of-the-art results with a BLEU score of 8.3, outperforming conventional machine learning models that scored between 3.4 and 5.8. Additionally, we conducted a user study in which participants interacted with the human memory augmentation agent through episodic memory and open-ended questions. The results of this study show that the agent results in significantly better recall performance on episodic memory tasks compared to human participants. The results also highlight the agent's practical applicability and user acceptance.