mlsanityData to 14 Oct 2026

Understanding Prompt Tuning and In-Context Learning via Meta-Learning

arXiv 2505.17010 ↗First posted 22 May 2025cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Prompting is one of the main ways to adapt a pretrained model to target tasks. Besides manually constructing prompts, many prompt optimization methods have been proposed in the literature. Method development is mainly empirically driven, with less emphasis on a conceptual understanding of prompting. In this paper we discuss how optimal prompting can be understood through a Bayesian view, which also implies some fundamental limitations of prompting that can only be overcome by tuning weights. The paper explains in detail how meta-trained neural networks behave as Bayesian predictors over the pretraining distribution, whose hallmark feature is rapid in-context adaptation. Optimal prompting can be studied formally as conditioning these Bayesian predictors, yielding criteria for target tasks where optimal prompting is and is not possible. We support the theory with educational experiments on LSTMs and Transformers, where we compare different versions of prefix-tuning and different weight-tuning methods. We also confirm that soft prefixes, which are sequences of real-valued vectors outside the token alphabet, can lead to very effective prompts for trained and even untrained networks by manipulating activations in ways that are not achievable by hard tokens. This adds an important mechanistic aspect beyond the conceptual Bayesian theory.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

42 of 56 references.

References · 42

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  5. 2022Transformers learn in-context by gradient descentcited by 732
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  9. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  10. 2016Learning to reinforcement learncited by 377
  11. 2021Transformers Can Do Bayesian Inferencecited by 360
  12. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
30 more
  1. 2023Language Modeling Is Compressioncited by 225
  2. 2024Many-Shot In-Context Learningcited by 206
  3. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  4. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  5. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  6. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  7. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  8. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  9. 2023In-Context Learning through the Bayesian Prismcited by 83
  10. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  11. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  12. 2019Meta-learning of Sequential Strategiescited by 70
  13. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  14. 2023Memory Augmented Large Language Models are Computationally Universalcited by 47
  15. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 43
  16. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  17. 2020Meta-trained agents implement Bayes-optimal agentscited by 38
  18. 2024The broader spectrum of in-context learningcited by 35
  19. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  20. 2024Learning Universal Predictorscited by 31
  21. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25
  22. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  23. 2024Algorithmic Capabilities of Random Transformerscited by 17
  24. 2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
  25. 2023Memory-Based Meta-Learning on Non-Stationary Distributionscited by 15
  26. 2022Sequential Learning Of Neural Networks for Prequential MDLcited by 13
  27. 2024Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Datacited by 13
  28. 2023Black-box Prompt Tuning with Subspace Learningcited by 5
  29. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  30. 2025Towards Interpretable Soft Promptscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Skill Neologisms: Towards Skill-based Continual Learningcited by 4
  2. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  3. 2025PromptLoop: Plug-and-Play Prompt Refinement via Latent Feedback for Diffusion Model Alignmentcited by 0
  4. 2026AsynDBT: Asynchronous Distributed Bilevel Tuning for efficient In-Context Learning with Large Language Modelscited by 0
  5. 2026When Do We Need LLMs? A Diagnostic for Language-Driven Banditscited by 0
  6. 2026Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posteriorcited by 0