mlsanityData to 14 Oct 2026

Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning

arXiv 2205.12453 ↗First posted 25 May 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer learning by updating only a small set of additional parameters while keeping the parameters of the pretrained language model frozen. While proven to be an effective method, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach should affect the pretraining stage. In this work, we show that taking the ultimate choice of fine-tuning method into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pretrained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 1.7 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the tweaks we introduce in MAML are crucial for the attained gains.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

21 of 28 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
9 more
  1. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  2. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  3. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  4. 2021MetaICL: Learning to Learn In Contextcited by 454
  5. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  6. 2019Meta-Learning Representations for Continual Learningcited by 189
  7. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  8. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  9. 2021MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learningcited by 21

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 3.

4 papers.

  1. 2024RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarizationcited by 11
  2. 2024Provable Meta-Learning with Low-Rank Adaptationscited by 2
  3. 2022Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuningcited by 0
  4. 2024Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Networkcited by 0