mlsanityData to 7 Oct 2026

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

arXiv 2106.04647 ↗First posted 8 Jun 2021cs.CLcited by 458

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-efficient fine-tuning methods, but these approaches either still require a relatively large number of parameters or underperform standard fine-tuning. In this work, we propose Compacter, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work. Compacter accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers. Specifically, Compacter inserts task-specific weight matrices into a pretrained model's weights, which are computed efficiently as a sum of Kronecker products between shared "slow" weights and "fast" rank-one matrices defined per Compacter layer. By only training 0.047% of a pretrained model's parameters, Compacter performs on par with standard fine-tuning on GLUE and outperforms standard fine-tuning on SuperGLUE and low-resource settings. Our code is publicly available at~urlhttps://github.com/rabeehk/compacter.

Key facts. Cited by 458 papers in the map (data through 7 Oct 2026). Builds on 48 of its 72 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

48 of 72 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
36 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  7. 2020What is the State of Neural Network Pruning?cited by 578
  8. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  9. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  10. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  11. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  12. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  13. 2021True Few-Shot Learning with Language Modelscited by 343
  14. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  15. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  16. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  17. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  18. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  19. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  20. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  21. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  22. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  23. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  24. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  25. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  26. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 72
  27. 2021Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameterscited by 66
  28. 2020Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Datacited by 50
  29. 2017Deep Quaternion Networkscited by 48
  30. 2019Quaternion Convolutional Neural Networkscited by 48
  31. 2018Quaternion Recurrent Neural Networkscited by 41
  32. 2018Contextual Parameter Generation for Universal Neural Machine Translationcited by 40
  33. 2019Lightweight and Efficient Neural Natural Language Processing with Quaternion Networkscited by 26
  34. 2019Evaluating Lottery Tickets Under Distributional Shiftscited by 23
  35. 2018Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognitioncited by 21
  36. 2021Parameterized Hypercomplex Graph Neural Networks for Graph Classificationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 58; 2023: 137; 2024: 116; 2025: 86; 2026: 47.

458 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  10. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  12. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
38 more
  1. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  2. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  4. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  5. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  6. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  7. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  8. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  9. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  10. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  11. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  12. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  13. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  14. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  15. 2023Efficient Large Language Models: A Surveycited by 183
  16. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  17. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  18. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  19. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  20. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  21. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  22. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  23. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  24. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  25. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  26. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
  27. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  28. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  29. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  30. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  31. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  32. 2021The Efficiency Misnomercited by 98
  33. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  34. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  35. 2023Modular Deep Learningcited by 82
  36. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  37. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  38. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76

50 most cited of 458 All 458 (JSON)