mlsanityData to 14 Oct 2026

PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation

arXiv 2208.10160 ↗First posted 22 Aug 2022cs.CLcited by 31

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Prompt Transfer (PoT) is a recently-proposed approach to improve prompt-tuning, by initializing the target prompt with the existing prompt trained on similar source tasks. However, such a vanilla PoT approach usually achieves sub-optimal performance, as (i) the PoT is sensitive to the similarity of source-target pair and (ii) directly fine-tuning the prompt initialized with source prompt on target task might lead to forgetting of the useful general knowledge learned from source task. To tackle these issues, we propose a new metric to accurately predict the prompt transferability (regarding (i)), and a novel PoT approach (namely PANDA) that leverages the knowledge distillation technique to alleviate the knowledge forgetting effectively (regarding (ii)). Extensive and systematic experiments on 189 combinations of 21 source and 9 target datasets across 5 scales of PLMs demonstrate that: 1) our proposed metric works well to predict the prompt transferability; 2) our PANDA consistently outperforms the vanilla PoT approach by 2.3% average score (up to 24.1%) among all tasks and model sizes; 3) with our PANDA approach, prompt-tuning can achieve competitive and even better performance than model-tuning in various PLM scales scenarios. We have publicly released our code in https://github.com/WHU-ZQH/PANDA.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 55 of its 88 references that are in the map. First posted on arXiv 22 Aug 2022.

Builds on

55 of 88 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
38 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  13. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  14. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  15. 2021GPT Understands, Toocited by 716
  16. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  17. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  18. 2018Born Again Neural Networkscited by 546
  19. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  20. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  21. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  22. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  23. 2018A Survey on Deep Learning for Named Entity Recognitioncited by 282
  24. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  25. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  26. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  27. 2021Does Knowledge Distillation Really Work?cited by 190
  28. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  29. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  30. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  31. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  32. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  33. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  34. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  35. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  36. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  37. 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
  38. 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 56

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 11; 2024: 9; 2025: 7.

31 papers.

Sort:

By area:

  1. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  2. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  3. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  4. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  5. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  6. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 43
  7. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 42
  8. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  9. 2022Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUEcited by 24
  10. 2023Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modelingcited by 20
  11. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  12. 2024Empowering Large Language Model for Continual Video Question Answering with Collaborative Promptingcited by 14
19 more
  1. 2023MerA: Merging Pretrained Adapters For Few-Shot Learningcited by 13
  2. 2023Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUEcited by 11
  3. 2023Revisiting Token Dropping Strategy in Efficient BERT Pretrainingcited by 11
  4. 2023Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Trackingcited by 10
  5. 2023Self-Evolution Learning for Discriminative Language Model Pretrainingcited by 8
  6. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  7. 2025BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editingcited by 7
  8. 2023Prompt-Learning for Cross-Lingual Relation Extractioncited by 4
  9. 2025Understanding Knowledge Transferability for Transfer Learning: A Surveycited by 4
  10. 2024Bayesian Multi-Task Transfer Learning for Soft Prompt Tuningcited by 3
  11. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Modelscited by 3
  12. 2024Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editingcited by 3
  13. 2025Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Productcited by 3
  14. 2025Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decompositioncited by 3
  15. 2023OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML Systemcited by 2
  16. 2025Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainabilitycited by 2
  17. 2024Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Promptingcited by 1
  18. 2025Efficient and Privacy-Preserving Soft Prompt Transfer for LLMscited by 1
  19. 2025Learning Optimal Prompt Ensemble for Multi-source Visual Prompt Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation
  2. 2026Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models
  3. 2025A Survey on Prompt Tuning
  4. 2022Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt
  5. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  6. 2025FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation
  7. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
  8. 2025PromptFlow: Training Prompts Like Neural Networks