mlsanityData to 7 Oct 2026

Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models

arXiv 2407.11033 ↗First posted 4 Jul 2024cs.LG cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Recent years, Pre-trained Language models (PLMs) have swept into various fields of artificial intelligence and achieved great success. However, most PLMs, such as T5 and GPT3, have a huge amount of parameters, fine-tuning them is often expensive and time consuming, and storing them takes up a lot of space. Therefore, it is necessary to adopt a parameter-efficient approach to reduce parameters of PLMs in fine-tuning without compromising their performance in downstream tasks. In this paper, we design a novel adapter which only acts on self-attention outputs in PLMs. This adapter adopts element-wise linear transformation using Hadamard product, hence named as Hadamard adapter, requires the fewest parameters compared to previous parameter-efficient adapters. In addition, we also summarize some tuning patterns for Hadamard adapter shared by various downstream tasks, expecting to provide some guidance for further parameter reduction with shared adapters in future studies. The experiments conducted on the widely-used GLUE benchmark with several SOTA PLMs prove that the Hadamard adapter achieves competitive performance with only 0.033% parameters compared with full fine-tuning, and it has the fewest parameters compared with other adapters. Moreover, we further find that there is also some redundant layers in the Hadamard adapter which can be removed to achieve more parameter efficiency with only 0.022% parameters.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

36 of 51 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
24 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  5. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  6. 2013Do Deep Nets Really Need to be Deep?cited by 466
  7. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  8. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  9. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  10. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  11. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  12. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  13. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  14. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  15. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  16. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  17. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  18. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  19. 2015Convolutional neural networks with low-rank regularizationcited by 102
  20. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  21. 2022A Study of the Attention Abnormality in Trojaned BERTscited by 47
  22. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 28
  23. 2022Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer Explanationcited by 16
  24. 2022Y-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 3; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  2. 2024Harnessing Earnings Reports for Stock Predictions: A QLoRA-Enhanced LLM Approachcited by 24
  3. 2024A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentationcited by 15
  4. 2024XMeCap: Meme Caption Generation with Sub-Image Adaptabilitycited by 14
  5. 2025Journey into Automation: Image-Derived Pavement Texture Extraction and Evaluationcited by 7
  6. 2024Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Modelscited by 2
  7. 2024Enhancing Exchange Rate Forecasting with Explainable Deep Learning Modelscited by 2
  8. 2024Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)cited by 0
  9. 2024Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)cited by 0
  10. 2025GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Modelscited by 0
  11. 2025PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapterscited by 0
  12. 2026Towards Privacy-Preserving Mental Health Support with Large Language Modelscited by 0
1 more
  1. 2026PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMscited by 0