mlsanityData to 7 Oct 2026

KronA: Parameter Efficient Tuning with Kronecker Adapter

arXiv 2212.10650 ↗First posted 20 Dec 2022cs.CLcited by 130

Not on the map. Why?

Abstract (arXiv)

Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a well-known paradigm in Natural Language Processing. However, with the ever-growing size of PLMs, training the entire model on several downstream tasks becomes very expensive and resource-hungry. Recently, different Parameter Efficient Tuning (PET) techniques are proposed to improve the efficiency of fine-tuning PLMs. One popular category of PET methods is the low-rank adaptation methods which insert learnable truncated SVD modules into the original model either sequentially or in parallel. However, low-rank decomposition suffers from limited representation power. In this work, we address this problem using the Kronecker product instead of the low-rank representation. We introduce KronA, a Kronecker product-based adapter module for efficient fine-tuning of Transformer-based PLMs. We apply the proposed methods for fine-tuning T5 on the GLUE benchmark to show that incorporating the Kronecker-based modules can outperform state-of-the-art PET methods.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

21 of 25 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
9 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  2. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  3. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  4. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  5. 2021Kronecker Decomposition for GPT Compressioncited by 27
  6. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 19
  7. 2021Convolutional Neural Network Compression through Generalized Kronecker Product Decompositioncited by 14
  8. 2022HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Taskscited by 12
  9. 2019Pushing the limits of RNN Compressioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 60; 2025: 40; 2026: 16.

130 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  3. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  4. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  5. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  6. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  7. 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 164
  8. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  9. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  10. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  11. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 112
  12. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
38 more
  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  3. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  4. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  5. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  6. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  7. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  8. 2023Visual Tuningcited by 34
  9. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  10. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  11. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  12. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 27
  13. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26
  14. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  15. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  16. 2024Hyperbolic Fine-Tuning for Large Language Modelscited by 24
  17. 2025MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoningcited by 24
  18. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  19. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  20. 2024Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotationcited by 21
  21. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  22. 2023Adversarial Score Distillation: When score distillation meets GANcited by 15
  23. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  24. 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
  25. 2024Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compressioncited by 12
  26. 2024DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Modelscited by 11
  27. 2024Large Convolutional Model Tuning via Filter Subspacecited by 11
  28. 2024Controllable Prompt Tuning For Balancing Group Distributional Robustnesscited by 10
  29. 2024SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modulescited by 10
  30. 2024Group and Shuffle: Efficient Structured Orthogonal Parametrizationcited by 10
  31. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  32. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Productcited by 10
  33. 2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9
  34. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  35. 2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
  36. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  37. 2024PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakerscited by 8
  38. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8

50 most cited of 130 All 130 (JSON)