mlsanityData to 7 Oct 2026

Y-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning

arXiv 2202.09817 ↗First posted 20 Feb 2022cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

With the success of large-scale pre-trained models (PTMs), how efficiently adapting PTMs to downstream tasks has attracted tremendous attention, especially for PTMs with billions of parameters. Although some parameter-efficient tuning paradigms have been proposed to address this problem, they still require large resources to compute the gradients in the training phase. In this paper, we propose Y-Tuning, an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks. Y-tuning learns dense representations for labels Y defined in a given task and aligns them to fixed feature representation. Without tuning the features of input text and model parameters, Y-tuning is both parameter-efficient and training-efficient. For DeBERTaXXL with 1.6 billion parameters, Y-tuning achieves performance more than 96% of full fine-tuning on GLUE Benchmark with only 2% tunable parameters and much fewer training costs.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 20 Feb 2022.

Builds on

32 of 43 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2019How Can We Know What Language Models Know?cited by 953
20 more
  1. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  2. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  3. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  4. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  7. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  8. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  9. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  10. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  11. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  12. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  13. 2021How Many Data Points is a Prompt Worth?cited by 197
  14. 2019Dice Loss for Data-imbalanced NLP Taskscited by 180
  15. 2021Entailment as Few-Shot Learnercited by 122
  16. 2019Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentencecited by 121
  17. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  18. 2020Description Based Text Classification with Reinforcement Learningcited by 27
  19. 2018Explicit Interaction Model towards Text Classificationcited by 16
  20. 2017Label Embedding Network: Learning Label Representation for Soft Training of Deep Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 8; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  2. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  3. 2024A Survey on LoRA of Large Language Modelscited by 67
  4. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  5. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  6. 2023Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuningcited by 26
  7. 2023UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memorycited by 26
  8. 2023Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictionscited by 24
  9. 2024Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Modelscited by 13
  10. 2024SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learningcited by 11
  11. 2024Gradient Weight-normalized Low-rank Projection for Efficient LLM Trainingcited by 11
  12. 2024Structure-aware Fine-tuning for Code Pre-trained Modelscited by 3
3 more
  1. 2024Learning Structured Representations by Embedding Class Hierarchy with Fast Optimal Transportcited by 3
  2. 2024Resource-Efficient Personal Large Language Models Fine-Tuning with Collaborative Edge Computingcited by 1
  3. 2026MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learningcited by 0