mlsanityData to 7 Oct 2026

Parameter-Efficient Tuning with Special Token Adaptation

arXiv 2210.04382 ↗First posted 10 Oct 2022cs.CL cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient tuning aims at updating only a small subset of parameters when adapting a pretrained model to downstream tasks. In this work, we introduce PASTA, in which we only modify the special token representations (e.g., [SEP] and [CLS] in BERT) before the self-attention module at each layer in Transformer-based models. PASTA achieves comparable performance to full finetuning in natural language understanding tasks including text classification and NER with up to only 0.029% of total parameters trained. Our work not only provides a simple yet effective way of parameter-efficient tuning, which has a wide range of practical applications when deploying finetuned models for multiple tasks, but also demonstrates the pivotal role of special tokens in pretrained language models

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

22 of 32 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
10 more
  1. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  2. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  3. 2019Revealing the Dark Secrets of BERTcited by 300
  4. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  5. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  6. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  7. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  8. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  9. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  10. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 2; 2026: 1.

7 papers.

  1. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  2. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  3. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  4. 2024Explaining Relationships Among Research Paperscited by 3
  5. 2024EUREKHA: Enhancing User Representation for Key Hackers Identification in Underground Forumscited by 2
  6. 2025Error-Aware Knowledge Distillation via Targeted Revision for Customer-Service Summarizationcited by 0
  7. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0