mlsanityData to 7 Oct 2026

Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters

arXiv 2211.01979 ↗First posted 18 Oct 2022cs.CL cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks by adding and tuning a small number of new parameters. Previously proposed adapter architectures are all feed-forward neural networks. In this paper, we investigate the effectiveness of using tiny-attention -- i.e., attention with extremely small per-head dimensionality -- as adapters. Our tiny-attention adapter learns to modify the hidden states at each position directly conditioned on the hidden states at all the other positions, which is missed by the previously proposed adapters. Moreover, we view its multiple attention heads as a mixture of experts and propose to average their weights during deployment, which further reduces its inference computation cost. On the GLUE benchmark, our tiny-attention adapter outperforms the other parameter-efficient transfer learning methods as well as full fine-tuning while only updating 0.05% of the parameters. On the FewGLUE benchmark, its performance is comparable to that of GPT-3 and PET.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

34 of 52 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
22 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  10. 2021GPT Understands, Toocited by 715
  11. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  12. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  13. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2020What is being transferred in transfer learning?cited by 365
  16. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  17. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  18. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  19. 2022Memorizing Transformerscited by 219
  20. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  21. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  22. 2017ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projectionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 1; 2026: 2.

12 papers.

  1. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  2. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  3. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  4. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  5. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  6. 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8
  7. 2023KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Modelscited by 7
  8. 2023PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniquescited by 4
  9. 2024The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraintscited by 4
  10. 2026ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuningcited by 1
  11. 2025GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Modelscited by 0
  12. 2026TaRA: Training-Aware Low-Rank Adaptation Initializationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Prompt Tuning based Adapter for Vision-Language Model Adaption
  2. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
  3. 2026BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
  4. 2023p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models
  5. 2025DiffLoRA: Differential Low-Rank Adapters for Large Language Models
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
  7. 2026DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles
  8. 2024Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves