# Episodic Memory Reader: Learning What to Remember for Question Answering from Streaming Data

arXiv: https://arxiv.org/abs/1903.06164 · HTML: https://mlsanity.com/paper/1903.06164/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2019.

## Built on by (most cited first, 14 of 14)

- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin](https://mlsanity.com/paper/2312.09979/) (2023)
- [Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigation](https://mlsanity.com/paper/2503.19622/) (2025)
- [HFT: Half Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2404.18466/) (2024)
- [CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Models](https://mlsanity.com/paper/2405.13684/) (2024)
- [NeoBERT: A Next-Generation BERT](https://mlsanity.com/paper/2502.19587/) (2025)
- [A Machine with Short-Term, Episodic, and Semantic Memory Systems](https://mlsanity.com/paper/2212.02098/) (2022)
- [Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experience](https://mlsanity.com/paper/2409.17702/) (2024)
- [A Machine With Human-Like Memory Systems](https://mlsanity.com/paper/2204.01611/) (2022)
- [A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information](https://mlsanity.com/paper/2305.07565/) (2023)
- [Graceful Forgetting II. Data as a Process](https://mlsanity.com/paper/2211.15441/) (2022)
- [Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents](https://mlsanity.com/paper/2609.39982/) (2026)
- [A Neural Conversation Generation Model via Equivalent Shared Memory Investigation](https://mlsanity.com/paper/2108.09164/) (2021)
- [Learning What to Remember: Long-horizon Counterfactual Memory Optimization](https://mlsanity.com/paper/2609.37930/) (2026)

## Abstract (arXiv)

We consider a novel question answering (QA) task where the machine needs to read from large streaming data (long documents or videos) without knowing when the questions will be given, which is difficult to solve with existing QA methods due to their lack of scalability. To tackle this problem, we propose a novel end-to-end deep network model for reading comprehension, which we refer to as Episodic Memory Reader (EMR) that sequentially reads the input contexts into an external memory, while replacing memories that are less important for answering unseen questions. Specifically, we train an RL agent to replace a memory entry when the memory is full, in order to maximize its QA accuracy at a future timepoint, while encoding the external memory using either the GRU or the Transformer architecture to learn representations that considers relative importance between the memory entries. We validate our model on a synthetic dataset (bAbI) as well as real-world large-scale textual QA (TriviaQA) and video QA (TVQA) datasets, on which it achieves significant improvements over rule-based memory scheduling policies or an RL-based baseline that independently learns the query-specific importance of each memory.