# Learning to Remember More with Less Memorization

arXiv: https://arxiv.org/abs/1901.01347 · HTML: https://mlsanity.com/paper/1901.01347/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2019.

## Built on by (most cited first, 14 of 14)

- [Token Turing Machines](https://mlsanity.com/paper/2211.09119/) (2022)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [Unveiling the Hidden: Online Vectorized HD Map Construction with Clip-Level Token Interaction and Propagation](https://mlsanity.com/paper/2411.11002/) (2024)
- [Learning to Rehearse in Long Sequence Memorization](https://mlsanity.com/paper/2106.01096/) (2021)
- [On the Memory Mechanism of Tensor-Power Recurrent Models](https://mlsanity.com/paper/2103.01521/) (2021)
- [Episodic Policy Gradient Training](https://mlsanity.com/paper/2112.01853/) (2021)
- [Beyond Surprise: Improving Exploration Through Surprise Novelty](https://mlsanity.com/paper/2308.04836/) (2023)
- [Recurrent Neural Networks for Learning Long-term Temporal Dependencies with Reanalysis of Time Scale Representation](https://mlsanity.com/paper/2111.03282/) (2021)
- [Learning to Constrain Policy Optimization with Virtual Trust Region](https://mlsanity.com/paper/2204.09315/) (2022)
- [Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations](https://mlsanity.com/paper/2404.05415/) (2024)
- [A Brain-inspired Memory Transformation based Differentiable Neural Computer for Reasoning-based Question Answering](https://mlsanity.com/paper/2301.02809/) (2023)
- [A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information](https://mlsanity.com/paper/2305.07565/) (2023)
- [CNN-Trans-Enc: A CNN-Enhanced Transformer-Encoder On Top Of Static BERT representations for Document Classification](https://mlsanity.com/paper/2209.06344/) (2022)
- [AHMF: Adaptive Hybrid-Memory-Fusion Model for Driver Attention Prediction](https://mlsanity.com/paper/2407.17442/) (2024)

## Abstract (arXiv)

Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.