Transformer with Memory Replay
Not on the map. Why?
Abstract (arXiv)
Transformers achieve state-of-the-art performance for natural language processing tasks by pre-training on large-scale text corpora. They are extremely compute-intensive and have very high sample complexity. Memory replay is a mechanism that remembers and reuses past examples by saving to and replaying from a memory buffer. It has been successfully used in reinforcement learning and GANs due to better sample efficiency. In this paper, we propose Transformer with Memory Replay (TMR), which integrates memory replay with transformer, making transformer more sample-efficient. Experiments on GLUE and SQuAD benchmark datasets show that Transformer with Memory Replay achieves at least 1% point increase compared to the baseline transformer model when pretrained with the same number of examples. Further, by adopting a careful design that reduces the wall-clock time overhead of memory replay, we also empirically achieve a better runtime efficiency.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 19 May 2022.
Builds on
38 of 50 references.
References · 38
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2020Reformer: The Efficient Transformercited by 1,658
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
- 2017Hindsight Experience Replaycited by 1,096
- 2015Prioritized Experience Replaycited by 1,070
26 more
- 2020Efficient Transformers: A Surveycited by 784
- 2018Image Transformercited by 770
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
- 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
- 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
- 2019Green AIcited by 620
- 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
- 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
- 2018Video Action Transformer Networkcited by 316
- 2018Distributed Prioritized Experience Replaycited by 260
- 2020Sparse Sinkhorn Attentioncited by 250
- 2018Music Transformercited by 207
- 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
- 2020Revisiting Fundamentals of Experience Replaycited by 171
- 2019When to use parametric models in reinforcement learning?cited by 118
- 2014Stochastic Optimization with Importance Samplingcited by 91
- 2018Memory Replay GANs: learning to generate images from new categories without forgettingcited by 76
- 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
- 2015Efficient Per-Example Gradient Computationscited by 42
- 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28
- 2019Competitive Experience Replaycited by 23
- 2017The Effects of Memory Replay in Reinforcement Learningcited by 22
- 2020Adam with Bandit Sampling for Deep Learningcited by 10
- 2020Dynamic Experience Replaycited by 7
- 2017Coordinate Descent with Bandit Samplingcited by 5
- 2018A Bandit Approach to Maximum Inner Product Searchcited by 5
Built on by
Citing papers per year: 2022: 2.
2 papers.