mlsanityData to 7 Oct 2026

LoTR: Low Tensor Rank Weight Adaptation

arXiv 2402.01376 ↗First posted 2 Feb 2024cs.CL cs.AI cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fine-tuning LLMs are based on matrix factorization of gradient update. We introduce LoTR, a novel approach for parameter-efficient fine-tuning of LLMs which represents a gradient update to parameters in a form of tensor decomposition. Low-rank adapter for each layer is constructed as a product of three matrices, and tensor structure arises from sharing left and right multipliers of this product among layers. Simultaneous compression of a sequence of layers with low-rank tensor representation allows LoTR to archive even better parameter efficiency then LoRA especially for deep models. Moreover, the core tensor does not depend on original weight dimension and can be made arbitrary small, which allows for extremely cheap and fast downstream fine-tuning.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

19 of 30 references.

References · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2024Mixtral of Expertscited by 1,785
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
7 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  2. 2021GPT Understands, Toocited by 715
  3. 2017Learning multiple visual domains with residual adapterscited by 553
  4. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  5. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  6. 2017Incremental Learning Through Deep Adaptationcited by 108
  7. 2023Run LoRA Run: Faster and Lighter LoRA Implementationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 11; 2026: 7.

27 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Ultra-Sparse Memory Networkcited by 19
  3. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  4. 2024The Sociolinguistic Foundations of Language Modelingcited by 13
  5. 2024SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modulescited by 10
  6. 2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
  7. 2026KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matricescited by 10
  8. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  9. 2024LoRTA: Low Rank Tensor Adaptation of Large Language Modelscited by 5
  10. 2024Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuningcited by 4
  11. 2025Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformationscited by 4
  12. 2025MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuningcited by 4
15 more
  1. 2024DoTA: Weight-Decomposed Tensor Adaptation for Large Language Modelscited by 3
  2. 2025μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Expertscited by 3
  3. 2025TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Modelscited by 2
  4. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  5. 2024LoRA-PT: Low-Rank Adapting UNETR for Hippocampus Segmentation Using Principal Tensor Singular Values and Vectorscited by 1
  6. 2025tCURLoRA: Tensor CUR Decomposition Based Low-Rank Parameter Adaptation and Its Application in Medical Image Segmentationcited by 1
  7. 2026LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weightscited by 1
  8. 2026All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptationcited by 1
  9. 2025Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximizationcited by 0
  10. 2025Parameter-Efficient Fine-Tuning of 3D DDPM for MRI Image Generation Using Tensor Networkscited by 0
  11. 2025TensLoRA: Tensor Alternatives for Low-Rank Adaptationcited by 0
  12. 2025D-com: Accelerating Iterative Processing to Enable Low-rank Decomposition of Activationscited by 0
  13. 2026Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuningcited by 0
  14. 2026On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuningcited by 0
  15. 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0