# Fine-tuning Image Transformers using Learnable Memory

arXiv: https://arxiv.org/abs/2203.15243 · HTML: https://mlsanity.com/paper/2203.15243/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://mlsanity.com/paper/2004.07202/) (2020)
- [Toward Transformer-Based Object Detection](https://mlsanity.com/paper/2012.09958/) (2020)
- [Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets](https://mlsanity.com/paper/1503.01007/) (2015)
- [Dense Associative Memory is Robust to Adversarial Inputs](https://mlsanity.com/paper/1701.00939/) (2017)

## Built on by (most cited first, 30 of 34)

- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Understanding Zero-Shot Adversarial Robustness for Large-Scale Models](https://mlsanity.com/paper/2212.07016/) (2022)
- [Explicit Visual Prompting for Low-Level Structure Segmentations](https://mlsanity.com/paper/2303.10883/) (2023)
- [Bayesian Prompt Learning for Image-Language Model Generalization](https://mlsanity.com/paper/2210.02390/) (2022)
- [Doubly Right Object Recognition: A Why Prompt for Visual Rationales](https://mlsanity.com/paper/2212.06202/) (2022)
- [Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction](https://mlsanity.com/paper/2303.09792/) (2023)
- [NeuMap: Neural Coordinate Mapping by Auto-Transdecoder for Camera Localization](https://mlsanity.com/paper/2211.11177/) (2022)
- [Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis](https://mlsanity.com/paper/2402.01730/) (2024)
- [Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem](https://mlsanity.com/paper/2402.02868/) (2024)
- [Learning Expressive Prompting With Residuals for Vision Transformers](https://mlsanity.com/paper/2303.15591/) (2023)
- [`A-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Prompting](https://mlsanity.com/paper/2302.07994/) (2023)
- [Explicit Visual Prompting for Universal Foreground Segmentations](https://mlsanity.com/paper/2305.18476/) (2023)
- [Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuning](https://mlsanity.com/paper/2309.06922/) (2023)
- [TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understanding](https://mlsanity.com/paper/2311.03427/) (2023)
- [FashionNTM: Multi-turn Fashion Image Retrieval via Cascaded Memory](https://mlsanity.com/paper/2308.10170/) (2023)
- [Interpreting and Controlling Vision Foundation Models via Text Explanations](https://mlsanity.com/paper/2310.10591/) (2023)
- [Spectral Prompt Tuning:Unveiling Unseen Classes for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2312.12754/) (2023)
- [Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection](https://mlsanity.com/paper/2403.01968/) (2024)
- [FlexAvatar: Learning Complete 3D Head Avatars with Partial Supervision](https://mlsanity.com/paper/2512.15599/) (2025)
- [CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformers](https://mlsanity.com/paper/2305.07840/) (2023)
- [Adaptive Human Trajectory Prediction via Latent Corridors](https://mlsanity.com/paper/2312.06653/) (2023)
- [DSeq-JEPA: Discriminative Sequential Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2511.17354/) (2025)
- [Your representations are in the network: composable and parallel adaptation for large scale models](https://mlsanity.com/paper/2303.04105/) (2023)
- [LT-ViT: A Vision Transformer for multi-label Chest X-ray classification](https://mlsanity.com/paper/2311.07263/) (2023)
- [Models Designed to Forget: Machine Unlearning via Key Deletion](https://mlsanity.com/paper/2603.15033/) (2026)
- [PM-DETR: Domain Adaptive Prompt Memory for Object Detection with Transformers](https://mlsanity.com/paper/2307.00313/) (2023)
- [Token Turing Machines are Efficient Vision Models](https://mlsanity.com/paper/2409.07613/) (2024)
- [A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models](https://mlsanity.com/paper/2410.13841/) (2024)
- [EDIT: Enhancing Vision Transformers by Mitigating Attention Sink through an Encoder-Decoder Architecture](https://mlsanity.com/paper/2504.06738/) (2025)
- [DiffPrompter: Differentiable Implicit Visual Prompts for Semantic-Segmentation in Adverse Conditions](https://mlsanity.com/paper/2310.04181/) (2023)

## Abstract (arXiv)

In this paper we propose augmenting Vision Transformer models with learnable memory tokens. Our approach allows the model to adapt to new tasks, using few parameters, while optionally preserving its capabilities on previously learned tasks. At each layer we introduce a set of learnable embedding vectors that provide contextual information useful for specific datasets. We call these "memory tokens". We show that augmenting a model with just a handful of such tokens per layer significantly improves accuracy when compared to conventional head-only fine-tuning, and performs only slightly below the significantly more expensive full fine-tuning. We then propose an attention-masking approach that enables extension to new downstream tasks, with a computation reuse. In this setup in addition to being parameters efficient, models can execute both old and new tasks as a part of single inference at a small incremental cost.