# CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video

arXiv: https://arxiv.org/abs/2609.17688 · HTML: https://mlsanity.com/paper/2609.17688/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 15 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 20 of 20)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [World Model on Million-Length Video And Language With Blockwise RingAttention](https://mlsanity.com/paper/2402.08268/) (2024)
- [HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World](https://mlsanity.com/paper/2309.17024/) (2023)
- [VideoVista: A Versatile Benchmark for Video Understanding and Reasoning](https://mlsanity.com/paper/2406.11303/) (2024)
- [In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting](https://mlsanity.com/paper/2509.07447/) (2025)
- [The CASTLE 2024 Dataset: Advancing the Art of Multimodal Understanding](https://mlsanity.com/paper/2503.17116/) (2025)
- [CaptionQA: Is Your Caption as Useful as the Image Itself?](https://mlsanity.com/paper/2511.21025/) (2025)
- [SiLVR: A Simple Language-based Video Reasoning Framework](https://mlsanity.com/paper/2505.24869/) (2025)
- [EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding](https://mlsanity.com/paper/2605.09874/) (2026)
- [B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens](https://mlsanity.com/paper/2412.09919/) (2024)
- [Watch Before You Answer: Learning from Visually Grounded Post-Training](https://mlsanity.com/paper/2604.05117/) (2026)
- [EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos](https://mlsanity.com/paper/2605.18734/) (2026)

## Abstract (arXiv)

Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token costs, and long-context retrieval failures. Under these practical constraints, we study whether textual captions can serve as reusable episodic memory. We define the Episodic Memory Video Caption QA task and introduce CapMem, a human-annotated benchmark with 75 videos totaling 33.7 hours, and 1,000 multiple-choice questions across 16 scenarios. On long videos (>20 min), full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA for 10/12 and 8/12 models, respectively. On the same video subset, a matched-frame control across six Qwen models retains mean accuracy gains of 3.22 and 2.55 points, respectively. Our caption-guided retrieve-and-verify harness further improves accuracy by up to 5.3 points. These results support the effectiveness of caption memory for episodic reasoning over long egocentric video.