mlsanityData to 14 Oct 2026

Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts

arXiv 2210.11292 ↗First posted 20 Oct 2022cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Prompt tuning is a parameter-efficient tuning (PETuning) method for utilizing pre-trained models (PTMs) that simply prepends a soft prompt to the input and only optimizes the prompt to adapt PTMs to downstream tasks. Although it is parameter- and deployment-efficient, its performance still lags behind other state-of-the-art PETuning methods. Besides, the training cost of prompt tuning is not significantly reduced due to the back-propagation through the entire model. Through empirical analyses, we shed some light on the lagging performance of prompt tuning and recognize a trade-off between the propagation distance from label signals to the inserted prompt and the influence of the prompt on model outputs. Further, we present Late Prompt Tuning (LPT) that inserts a late prompt into an intermediate layer of the PTM instead of the input layer or all layers. The late prompt is obtained by a neural prompt generator conditioned on the hidden states before the prompt insertion layer and therefore is instance-dependent. Through extensive experimental results across various tasks and PTMs, we show that LPT can achieve competitive performance to full model tuning and other PETuning methods under both full-data and few-shot scenarios while possessing faster training speed and lower memory cost.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

34 of 49 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
22 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2021GPT Understands, Toocited by 716
  8. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  9. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  10. 2021A Survey of Transformerscited by 401
  11. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  12. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  13. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  14. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  15. 2021Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameterscited by 66
  16. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  17. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  18. 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
  19. 2021Paradigm Shift in Natural Language Processingcited by 39
  20. 2022Instance-aware Prompt Learning for Language Understanding and Generationcited by 18
  21. 2022Context-Tuning: Learning Contextualized Prompts for Natural Language Generationcited by 12
  22. 2021LiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learnerscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 12; 2025: 4; 2026: 1.

23 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  3. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  4. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  5. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  6. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  7. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 31
  8. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 30
  9. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
  10. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuningcited by 19
  11. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  12. 2025AMAS: Adaptively Determining Communication Topology for LLM-based Multi-Agent Systemcited by 15
11 more
  1. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  2. 2024Diffusion Tuning: Transferring Diffusion Models via Chain of Forgettingcited by 12
  3. 2024Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptationcited by 8
  4. 2024PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdificationcited by 8
  5. 2024Fisher Information-based Efficient Curriculum Federated Learning with Large Language Modelscited by 6
  6. 2024DialCLIP: Empowering CLIP as Multi-Modal Dialog Retrievercited by 2
  7. 2023Improving Prompt Tuning with Learned Prompting Layerscited by 1
  8. 2024IAPT: Instruction-Aware Prompt Tuning for Large Language Modelscited by 1
  9. 2025DynaSwarm: Dynamically Graph Structure Selection for LLM-based Multi-agent Systemcited by 1
  10. 2025Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMscited by 0
  11. 2026Gripper-aware Vision Language Action Modelscited by 0