# Episodic Memory Question Answering

arXiv: https://arxiv.org/abs/2205.01652 · HTML: https://mlsanity.com/paper/2205.01652/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [IQA: Visual Question Answering in Interactive Environments](https://mlsanity.com/paper/1712.03316/) (2017)
- [Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks](https://mlsanity.com/paper/1903.03878/) (2019)
- [Embodied Question Answering in Photorealistic Environments with Point Cloud Perception](https://mlsanity.com/paper/1904.03461/) (2019)
- [Multi-Target Embodied Question Answering](https://mlsanity.com/paper/1904.04686/) (2019)
- [RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/1806.01054/) (2018)
- [Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views](https://mlsanity.com/paper/2010.01191/) (2020)
- [Bayesian Relational Memory for Semantic Visual Navigation](https://mlsanity.com/paper/1909.04306/) (2019)
- [Talk the Walk: Navigating New York City through Grounded Dialogue](https://mlsanity.com/paper/1807.03367/) (2018)
- [Chasing Ghosts: Instruction Following as Bayesian State Tracking](https://mlsanity.com/paper/1907.02022/) (2019)
- [PyRobot: An Open-source Robotics Framework for Research and Benchmarking](https://mlsanity.com/paper/1906.08236/) (2019)
- [Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction](https://mlsanity.com/paper/1811.04179/) (2018)
- [Learning Spatial Common Sense with Geometry-Aware Recurrent Networks](https://mlsanity.com/paper/1901.00003/) (2018)
- [Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agents](https://mlsanity.com/paper/2009.03231/) (2020)
- [The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation](https://mlsanity.com/paper/2108.11550/) (2021)
- [Where Are You? Localization from Embodied Dialog](https://mlsanity.com/paper/2011.08277/) (2020)
- [Geometry-Aware Recurrent Neural Networks for Active Visual Recognition](https://mlsanity.com/paper/1811.01292/) (2018)
- [VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering](https://mlsanity.com/paper/1908.04950/) (2019)

## Built on by (most cited first, 30 of 32)

- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory](https://mlsanity.com/paper/2210.05663/) (2022)
- [GridMM: Grid Memory Map for Vision-and-Language Navigation](https://mlsanity.com/paper/2307.12907/) (2023)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [See and Think: Embodied Agent in Virtual Environment](https://mlsanity.com/paper/2311.15209/) (2023)
- [AMEGO: Active Memory from long EGOcentric videos](https://mlsanity.com/paper/2409.10917/) (2024)
- [Situational Awareness Matters in 3D Vision Language Reasoning](https://mlsanity.com/paper/2406.07544/) (2024)
- [EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding](https://mlsanity.com/paper/2301.02217/) (2023)
- [VideoQA in the Era of LLMs: An Empirical Study](https://mlsanity.com/paper/2408.04223/) (2024)
- [EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queries](https://mlsanity.com/paper/2212.06969/) (2022)
- [Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding](https://mlsanity.com/paper/2503.09143/) (2025)
- [Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation](https://mlsanity.com/paper/2403.08282/) (2024)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4D](https://mlsanity.com/paper/2308.12035/) (2023)
- [Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model](https://mlsanity.com/paper/2404.04619/) (2024)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [FashionNTM: Multi-turn Fashion Image Retrieval via Cascaded Memory](https://mlsanity.com/paper/2308.10170/) (2023)
- [ESceme: Vision-and-Language Navigation with Episodic Scene Memory](https://mlsanity.com/paper/2303.01032/) (2023)
- [Video Question Answering for People with Visual Impairments Using an Egocentric 360-Degree Camera](https://mlsanity.com/paper/2405.19794/) (2024)
- [Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations](https://mlsanity.com/paper/2301.02184/) (2023)
- [HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Models](https://mlsanity.com/paper/2404.19065/) (2024)
- [Learning to Summarize and Answer Questions about a Virtual Robot's Past Actions](https://mlsanity.com/paper/2306.09922/) (2023)
- [EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception](https://mlsanity.com/paper/2506.21080/) (2025)
- [Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance](https://mlsanity.com/paper/2512.13238/) (2025)
- [Estimating more camera poses for ego-centric videos is essential for VQ3D](https://mlsanity.com/paper/2211.10284/) (2022)
- [EgoGraph: Temporal Knowledge Graph for Egocentric Video Understanding](https://mlsanity.com/paper/2602.23709/) (2026)
- [ActiveFly-Bench: Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception](https://mlsanity.com/paper/2607.10180/) (2026)
- [Grounding Large Language Models In Embodied Environment With Imperfect World Models](https://mlsanity.com/paper/2410.02742/) (2024)
- [Binary Tracking for Spatial QA and Navigation with Open Vision-Language Models](https://mlsanity.com/paper/2606.16902/) (2026)
- [DarkQA: Benchmarking Vision-Language Models on Visual-Primitive Question Answering in Low-Light Indoor Scenes](https://mlsanity.com/paper/2512.24985/) (2025)

## Abstract (arXiv)

Egocentric augmented reality devices such as wearable glasses passively capture visual data as a human wearer tours a home environment. We envision a scenario wherein the human communicates with an AI agent powering such a device by asking questions (e.g., where did you last see my keys?). In order to succeed at this task, the egocentric AI assistant must (1) construct semantically rich and efficient scene memories that encode spatio-temporal information about objects seen during the tour and (2) possess the ability to understand the question and ground its answer into the semantic memory representation. Towards that end, we introduce (1) a new task - Episodic Memory Question Answering (EMQA) wherein an egocentric AI assistant is provided with a video sequence (the tour) and a question as an input and is asked to localize its answer to the question within the tour, (2) a dataset of grounded questions designed to probe the agent's spatio-temporal understanding of the tour, and (3) a model for the task that encodes the scene as an allocentric, top-down semantic feature map and grounds the question into the map to localize the answer. We show that our choice of episodic scene memory outperforms naive, off-the-shelf solutions for the task as well as a host of very competitive baselines and is robust to noise in depth, pose as well as camera jitter. The project page can be found at: https://samyak-268.github.io/emqa .