# TOAST: Transfer Learning via Attention Steering

arXiv: https://arxiv.org/abs/2305.15542 · HTML: https://mlsanity.com/paper/2305.15542/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models](https://mlsanity.com/paper/2203.06904/) (2022)
- [Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering](https://mlsanity.com/paper/1511.05234/) (2015)
- [Top-Down Visual Attention from Analysis by Synthesis](https://mlsanity.com/paper/2303.13043/) (2023)

## Built on by (most cited first, 15 of 15)

- [Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs](https://mlsanity.com/paper/2311.02262/) (2023)
- [Vision Transformers Don't Need Trained Registers](https://mlsanity.com/paper/2506.08010/) (2025)
- [Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?](https://mlsanity.com/paper/2401.12902/) (2024)
- [Time-, Memory- and Parameter-Efficient Visual Adaptation](https://mlsanity.com/paper/2402.02887/) (2024)
- [LLoCO: Learning Long Contexts Offline](https://mlsanity.com/paper/2404.07979/) (2024)
- [Low-Resource Vision Challenges for Foundation Models](https://mlsanity.com/paper/2401.04716/) (2024)
- [SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity](https://mlsanity.com/paper/2506.16500/) (2025)
- [Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance](https://mlsanity.com/paper/2410.13136/) (2024)
- [EDIT: Enhancing Vision Transformers by Mitigating Attention Sink through an Encoder-Decoder Architecture](https://mlsanity.com/paper/2504.06738/) (2025)
- [Simulate, Refocus and Ensemble: An Attention-Refocusing Scheme for Domain Generalization](https://mlsanity.com/paper/2507.12851/) (2025)
- [Attention-Aligned Reasoning for Large Language Models](https://mlsanity.com/paper/2510.03223/) (2025)
- [FCL-ViT: Task-Aware Attention Tuning for Continual Learning](https://mlsanity.com/paper/2412.02509/) (2024)
- [DP-SelFT: Differentially Private Selective Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2605.17432/) (2026)
- [Enhancing Multi-Agent Communication through Attention Steering with Context Relevance](https://mlsanity.com/paper/2605.30136/) (2026)
- [Data-Efficient Adaptation of LLMs via Attention Head Reweighting](https://mlsanity.com/paper/2607.13425/) (2026)

## Abstract (arXiv)

Transfer learning involves adapting a pre-trained model to novel downstream tasks. However, we observe that current transfer learning methods often fail to focus on task-relevant features. In this work, we explore refocusing model attention for transfer learning. We introduce Top-Down Attention Steering (TOAST), a novel transfer learning algorithm that keeps the pre-trained backbone frozen, selects task-relevant features in the output, and feeds those features back to the model to steer the attention to the task-specific features. By refocusing the attention only, TOAST achieves state-of-the-art results on a number of transfer learning benchmarks, while having a small number of tunable parameters. Compared to fully fine-tuning, LoRA, and prompt tuning, TOAST substantially improves performance across a range of fine-grained visual classification datasets (e.g., 81.1% -> 86.2% on FGVC). TOAST also outperforms the fully fine-tuned Alpaca and Vicuna models on instruction-following language generation. Code is available at https://github.com/bfshi/TOAST.