mlsanityData to 14 Oct 2026

Universality and Limitations of Prompt Tuning

arXiv 2305.18787 ↗First posted 30 May 2023cs.LG cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Despite the demonstrated empirical efficacy of prompt tuning to adapt a pretrained language model for a new task, the theoretical underpinnings of the difference between "tuning parameters before the input" against "the tuning of model weights" are limited. We thus take one of the first steps to understand the role of soft-prompt tuning for transformer-based architectures. By considering a general purpose architecture, we analyze prompt tuning from the lens of both: universal approximation and limitations with finite-depth fixed-weight pretrained transformers for continuous-valued functions. Our universality result guarantees the existence of a strong transformer with a prompt to approximate any sequence-to-sequence function in the set of Lipschitz functions. The limitations of prompt tuning for limited-depth transformers are first proved by constructing a set of datasets, that cannot be memorized by a prompt of any length for a given single encoder layer. We also provide a lower bound on the required number of tunable prompt parameters and compare the result with the number of parameters required for a low-rank update (based on LoRA) for a single-layer setting. We finally extend our analysis to multi-layer settings by providing sufficient conditions under which the transformer can at best learn datasets from invertible functions only. Our theoretical claims are also corroborated by empirical results.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

26 of 39 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  11. 2022Transformers learn in-context by gradient descentcited by 732
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
14 more
  1. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  2. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  3. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  4. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  5. 2018Invertible Residual Networkscited by 365
  6. 2020The Lipschitz Constant of Self-Attentioncited by 174
  7. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  8. 2016Identity Matters in Deep Learningcited by 109
  9. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  10. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  11. 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 78
  12. 2020A Mathematical Theory of Attentioncited by 46
  13. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 39
  14. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 8; 2026: 9.

30 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
  3. 2025Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gatecited by 16
  4. 2023Automatic Engineering of Long Promptscited by 14
  5. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  6. 2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
  7. 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8
  8. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  9. 2024Embedded Visual Prompt Tuningcited by 7
  10. 2023POND: Multi-Source Time Series Domain Adaptation with Information-Aware Prompt Tuningcited by 6
  11. 2024MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusioncited by 5
  12. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
18 more
  1. 2025Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representationcited by 4
  2. 2025Memory Limitations of Prompt Tuning in Transformerscited by 4
  3. 2024Differentially Private Kernel Density Estimationcited by 3
  4. 2024AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discoverycited by 2
  5. 2025Bridging Modalities via Progressive Re-alignment for Multimodal Test-Time Adaptationcited by 2
  6. 2025VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansioncited by 1
  7. 2026Neurosymbolic LoRA: Why and When to Tune Weights vs. Rewrite Promptscited by 1
  8. 2026Orthogonal Knowledge Refreshing for Domain-Incremental Object Detectioncited by 1
  9. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  10. 2024Experimental Design for Active Transductive Inference in Large Language Modelscited by 0
  11. 2024Universal In-Context Approximation By Prompting Fully Recurrent Modelscited by 0
  12. 2025PVeRA: Probabilistic Vector-Based Random Matrix Adaptationcited by 0
  13. 2026Optimizing Soft Prompt Tuning via Structural Evolutioncited by 0
  14. 2026Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Modelscited by 0
  15. 2026Neutral-Reference Prompting for Vision-Language Modelscited by 0
  16. 2026How Many Different Outputs Can a Transformer Generate?cited by 0
  17. 2026On the Capability and Limitation of Hard Promptcited by 0
  18. 2026When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attentioncited by 0