# Learning to Remember Rare Events

arXiv: https://arxiv.org/abs/1703.03129 · HTML: https://mlsanity.com/paper/1703.03129/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2017.

## Built on by (most cited first, 30 of 63)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models](https://mlsanity.com/paper/2308.16137/) (2023)
- [MeGA-CDA: Memory Guided Attention for Category-Aware Unsupervised Domain Adaptive Object Detection](https://mlsanity.com/paper/2103.04224/) (2021)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory](https://mlsanity.com/paper/2211.00924/) (2022)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning](https://mlsanity.com/paper/2206.03996/) (2022)
- [HVPR: Hybrid Voxel-Point Representation for Single-stage 3D Object Detection](https://mlsanity.com/paper/2104.00902/) (2021)
- [kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference](https://mlsanity.com/paper/2303.13824/) (2023)
- [Learning Fast and Slow for Online Time Series Forecasting](https://mlsanity.com/paper/2202.11672/) (2022)
- [von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learning](https://mlsanity.com/paper/2103.15718/) (2021)
- [Learning Dual Memory Dictionaries for Blind Face Restoration](https://mlsanity.com/paper/2210.08160/) (2022)
- [Unbiased Scene Graph Generation in Videos](https://mlsanity.com/paper/2304.00733/) (2023)
- [STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction](https://mlsanity.com/paper/2312.17346/) (2023)
- [SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2111.13495/) (2021)
- [The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey](https://mlsanity.com/paper/2401.07872/) (2024)
- [MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2312.02530/) (2023)
- [Token Turing Machines](https://mlsanity.com/paper/2211.09119/) (2022)
- [Regular Polytope Networks](https://mlsanity.com/paper/2103.15632/) (2021)
- [How to Train Your MAML to Excel in Few-Shot Classification](https://mlsanity.com/paper/2106.16245/) (2021)
- [Context-Aware Meta-Learning](https://mlsanity.com/paper/2310.10971/) (2023)
- [Learning to Remember Patterns: Pattern Matching Memory Networks for Traffic Forecasting](https://mlsanity.com/paper/2110.10380/) (2021)
- [A Closer Look at Few-Shot Video Classification: A New Baseline and Benchmark](https://mlsanity.com/paper/2110.12358/) (2021)
- [SEDM: Selective Self-Evolving Distributed Memory for Agents](https://mlsanity.com/paper/2509.09498/) (2025)
- [Hierarchical Variational Memory for Few-shot Learning Across Domains](https://mlsanity.com/paper/2112.08181/) (2021)
- [Exploiting Structural Consistency of Chest Anatomy for Unsupervised Anomaly Detection in Radiography Images](https://mlsanity.com/paper/2403.08689/) (2024)
- [Few-shot Single-view 3D Reconstruction with Memory Prior Contrastive Network](https://mlsanity.com/paper/2208.00183/) (2022)
- [Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation](https://mlsanity.com/paper/2203.11670/) (2022)

## Abstract (arXiv)

Despite recent advances, memory-augmented deep neural networks are still limited when it comes to life-long and one-shot learning, especially in remembering rare events. We present a large-scale life-long memory module for use in deep learning. The module exploits fast nearest-neighbor algorithms for efficiency and thus scales to large memory sizes. Except for the nearest-neighbor query, the module is fully differentiable and trained end-to-end with no extra supervision. It operates in a life-long manner, i.e., without the need to reset it during training. Our memory module can be easily added to any part of a supervised neural network. To show its versatility we add it to a number of networks, from simple convolutional ones tested on image classification to deep sequence-to-sequence and recurrent-convolutional models. In all cases, the enhanced network gains the ability to remember and do life-long one-shot learning. Our module remembers training examples shown many thousands of steps in the past and it can successfully generalize from them. We set new state-of-the-art for one-shot learning on the Omniglot dataset and demonstrate, for the first time, life-long one-shot learning in recurrent neural networks on a large-scale machine translation task.