mlsanityData to 7 Oct 2026

Prompting a Pretrained Transformer Can Be a Universal Approximator

arXiv 2402.14753 ↗First posted 22 Feb 2024cs.LG cs.AI math.FAcited by 25

Not on the map. Why?

Abstract (arXiv)

Despite the widespread adoption of prompting, prompt tuning and prefix-tuning of transformer models, our theoretical understanding of these fine-tuning methods remains limited. A key question is whether one can arbitrarily modify the behavior of pretrained model by prompting or prefix-tuning it. Formally, whether prompting and prefix-tuning a pretrained model can universally approximate sequence-to-sequence functions. This paper answers in the affirmative and demonstrates that much smaller pretrained models than previously thought can be universal approximators when prefixed. In fact, the attention mechanism is uniquely suited for universal approximation with prefix-tuning a single attention head being sufficient to approximate any continuous function. Moreover, any sequence-to-sequence function can be approximated by prefixing a transformer with depth linear in the sequence length. Beyond these density-type results, we also offer Jackson-type bounds on the length of the prefix needed to approximate a function to a desired precision.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 30 of its 56 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

30 of 56 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  12. 2022Transformers learn in-context by gradient descentcited by 714
18 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2017Learning multiple visual domains with residual adapterscited by 553
  3. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  4. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  5. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  6. 2023Representational Strengths and Limitations of Transformerscited by 132
  7. 2020The Kolmogorov-Arnold representation theorem revisitedcited by 89
  8. 2015Representation Benefits of Deep Feedforward Networkscited by 84
  9. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  10. 2021On the Expressive Power of Self-Attention Matricescited by 44
  11. 2022Perfectly Secure Steganography Using Minimum Entropy Couplingcited by 42
  12. 2023Memorization Capacity of Multi-Head Attention in Transformerscited by 39
  13. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  14. 2023On the Expressivity Role of LayerNorm in Transformers' Attentioncited by 32
  15. 2023Universality and Limitations of Prompt Tuningcited by 30
  16. 2023Sumformer: Universal Approximation for Efficient Transformerscited by 30
  17. 2020Stolen Probability: A Structural Weakness of Neural Language Modelscited by 21
  18. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  2. 2024Measure-to-measure interpolation using Transformerscited by 28
  3. 2024Near to Mid-term Risks and Opportunities of Open-Source Generative AIcited by 13
  4. 2025Long Context In-Context Compression by Getting to the Gist of Gistingcited by 12
  5. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  6. 2024Risks and Opportunities of Open-Source Generative AIcited by 11
  7. 2024Towards Infinite-Long Prefix in Transformercited by 10
  8. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  9. 2025Approximation Bounds for Transformer Networks with Application to Regressioncited by 4
  10. 2025Memory Limitations of Prompt Tuning in Transformerscited by 4
  11. 2024Expressivity of Neural Networks with Random Weights and Learned Biasescited by 2
  12. 2025Jailbreaking Large Language Models in Infinitely Many Wayscited by 2
13 more
  1. 2025A unified framework for establishing the universal approximation of transformer-type architecturescited by 2
  2. 2026Measure-to-measure Regression with Transformerscited by 1
  3. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  4. 2024Universal In-Context Approximation By Prompting Fully Recurrent Modelscited by 0
  5. 2025Vocabulary In-Context Learning in Transformers: Benefits of Positional Encodingcited by 0
  6. 2025On the Universality of Transformer Architectures; How Much Attention Is Enough?cited by 0
  7. 2026Approximation Error Upper and Lower Bounds for H"older Class with Transformerscited by 0
  8. 2026Learning to Translate from Soft to Hard LLM Promptscited by 0
  9. 2026Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generationcited by 0
  10. 2026Xeno-Interpretability: Investigating the Alien Minds of LLMscited by 0
  11. 2026On the Capability and Limitation of Hard Promptcited by 0
  12. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0
  13. 2026When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attentioncited by 0