mlsanityData to 7 Oct 2026

AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks

arXiv 2205.00305 ↗First posted 30 Apr 2022cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed. AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias. The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pre-trained models. We further find that AdapterBias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 30 Apr 2022.

Builds on

28 of 38 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
16 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  2. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  3. 2018Neural Network Acceptability Judgmentscited by 952
  4. 2017A Survey on Multi-Task Learningcited by 838
  5. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  7. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  8. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  9. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  11. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  12. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  13. 2019Revealing the Dark Secrets of BERTcited by 300
  14. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  15. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  16. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 12; 2024: 8; 2025: 5; 2026: 1.

29 papers.

Sort:

By area:

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  2. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  3. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  4. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  5. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  6. 2023Visual Tuningcited by 34
  7. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  8. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  9. 2023Debiasing Algorithm through Model Adaptationcited by 24
  10. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  11. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 18
  12. 2022CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Modelscited by 13
17 more
  1. 2023When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence Systemcited by 13
  2. 2024Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inferencecited by 12
  3. 2025Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Modelscited by 12
  4. 2023A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Modelcited by 8
  5. 2023Team Triple-Check at Factify 2: Parameter-Efficient Large Foundation Models with Feature Representations for Multi-Modal Fact Verificationcited by 7
  6. 2023HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptationcited by 7
  7. 2025SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splittingcited by 7
  8. 2024MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuningcited by 6
  9. 2024How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenariocited by 5
  10. 2024DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learningcited by 4
  11. 2025On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentationcited by 4
  12. 2023RGAT: A Deeper Look into Syntactic Dependency Information for Coreference Resolutioncited by 3
  13. 2023Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Modelcited by 3
  14. 2024PEFT for Speech: Unveiling Optimal Placement, Merging Strategies, and Ensemble Techniquescited by 3
  15. 2025Dual Debiasing: Remove Stereotypes and Keep Factual Gender for Fair Language Modeling and Translationcited by 2
  16. 2022Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuningcited by 0
  17. 2026Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weightingcited by 0