mlsanityData to 7 Oct 2026

On the Role of Attention in Prompt-tuning

arXiv 2306.03435 ↗First posted 6 Jun 2023cs.LG cs.CL stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

Prompt-tuning is an emerging strategy to adapt large language models (LLM) to downstream tasks by learning a (soft-)prompt parameter from data. Despite its success in LLMs, there is limited theoretical understanding of the power of prompt-tuning and the role of the attention mechanism in prompting. In this work, we explore prompt-tuning for one-layer attention architectures and study contextual mixture-models where each input token belongs to a context-relevant or -irrelevant set. We isolate the role of prompt-tuning through a self-contained prompt-attention model. Our contributions are as follows: (1) We show that softmax-prompt-attention is provably more expressive than softmax-self-attention and linear-prompt-attention under our contextual data model. (2) We analyze the initial trajectory of gradient descent and show that it learns the prompt and prediction head with near-optimal sample complexity and demonstrate how prompt can provably attend to sparse context-relevant tokens. (3) Assuming a known prompt but an unknown prediction head, we characterize the exact finite sample performance of prompt-attention which reveals the fundamental performance limits and the precise benefit of the context information. We also provide experiments that verify our theoretical insights on real datasets and demonstrate how prompt-tuning enables the model to attend to context-relevant information.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

26 of 31 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  12. 2022A Generalist Agentcited by 802
14 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  3. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  4. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  5. 2022Vision Transformers provably learn spatial structurecited by 92
  6. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  7. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  8. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  9. 2019Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problemcited by 35
  10. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  11. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  12. 2018Stochastic Gradient Descent Learns State Equations with Nonlinear Activationscited by 16
  13. 2022Convexifying Transformers: Improving optimization and understanding of transformer networkscited by 12
  14. 2022The Quarks of Attentioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 22; 2025: 13; 2026: 6.

49 papers.

Sort:

By area:

  1. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  2. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  3. 2023Transformers as Support Vector Machinescited by 67
  4. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  5. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  6. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  7. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  8. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  9. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  10. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  11. 2024From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformerscited by 34
  12. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
37 more
  1. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  2. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  3. 2024Transformers on Markov Data: Constant Depth Sufficescited by 26
  4. 2024Provably learning a multi-head attention layercited by 21
  5. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  6. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 16
  7. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  8. 2024What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encodingcited by 14
  9. 2024Provable Differentially Private Computation of the Cross-Attention Mechanismcited by 14
  10. 2024Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysiscited by 14
  11. 2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
  12. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  13. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  14. 2024Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysiscited by 8
  15. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  16. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Samplingcited by 6
  17. 2025From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformerscited by 6
  18. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  19. 2024On the Power of Convolution Augmented Transformercited by 5
  20. 2024Learning Linear Attention in Polynomial Timecited by 5
  21. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
  22. 2025Memory Limitations of Prompt Tuning in Transformerscited by 4
  23. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  24. 2023Visual Attention Prompted Prediction and Learningcited by 2
  25. 2024Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Modelscited by 2
  26. 2026Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learningcited by 2
  27. 2025Efficient and Privacy-Preserving Soft Prompt Transfer for LLMscited by 1
  28. 2025TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removalcited by 1
  29. 2025VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansioncited by 1
  30. 2026Visual prompting reimagined: The power of the Activation Promptscited by 1
  31. 2026Grokking as Structural Inference: Transformers Need Bayesian Lottery Ticketscited by 1
  32. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  33. 2024Memorization Capacity for Additive Fine-Tuning with Small ReLU Networkscited by 0
  34. 2025Attention with Trained Embeddings Provably Selects Important Tokenscited by 0
  35. 2025Minimax Rates for Learning Pairwise Interactions in Attention-Style Modelscited by 0
  36. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformerscited by 0
  37. 2026Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weightingcited by 0