Cached Transformers: Improving Transformers with Differentiable Memory Cache
Not on the map. Why?
Abstract (arXiv)
This work introduces a new Transformer model called Cached Transformer, which uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens. GRC attention enables attending to both past and current tokens, increasing the receptive field of attention and allowing for exploring long-range dependencies. By utilizing a recurrent gating unit to continuously update the cache, our model achieves significant advancements in six language and vision tasks, including language modeling, machine translation, ListOPs, image classification, object detection, and instance segmentation. Furthermore, our approach surpasses previous memory-based techniques in tasks such as language modeling and displays the ability to be applied to a broader range of situations.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 20 Dec 2023.
Builds on
43 of 53 references.
References · 43
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2017Mask R-CNNcited by 7,277
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
31 more
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2020Big Bird: Transformers for Longer Sequencescited by 1,684
- 2020Reformer: The Efficient Transformercited by 1,668
- 2020Rethinking Attention with Performerscited by 1,601
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
- 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
- 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
- 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
- 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
- 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
- 2019Stand-Alone Self-Attention in Vision Modelscited by 542
- 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
- 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2019Invariance Matters: Exemplar Memory for Domain Adaptive Person Re-identificationcited by 133
- 2019Cross-Batch Memory for Embedding Learningcited by 132
- 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
- 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
- 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
- 2016Improving Neural Language Models with a Continuous Cachecited by 82
- 2020Memory Transformercited by 76
- 2017Learning to Remember Translation History with a Continuous Cachecited by 39
- 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
- 2021OmniNet: Omnidirectional Representations from Transformerscited by 23
- 2021Dynamic Token Normalization Improves Vision Transformerscited by 10
- 2020Fine-grained Classification via Categorical Memory Networkscited by 3
- 2018Improved Language Modeling by Decoding the Pastcited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.