mlsanityData to 7 Oct 2026

Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying

arXiv 2311.09578 ↗First posted 16 Nov 2023cs.CL cs.AI cs.LGcited by 65

Not on the map. Why?

Abstract (arXiv)

We introduce Tied-LoRA, a novel paradigm leveraging weight tying and selective training to enhance the parameter efficiency of Low-rank Adaptation (LoRA). Our exploration encompasses different plausible combinations of parameter training and freezing, coupled with weight tying, aimed at identifying the optimal trade-off between performance and the count of trainable parameters. Across 5 diverse tasks and two foundational language models with different parameter counts, our experiments provide comprehensive insights into the inherent trade-offs between efficiency and performance. Our findings reveal a specific Tied-LoRA configuration that distinguishes itself by showcasing comparable performance to LoRA across multiple tasks while utilizing only a fraction of the parameters employed by the standard LoRA method, particularly at elevated ranks. This underscores the efficacy of Tied-LoRA in achieving impressive results with significantly reduced model complexity.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

20 of 30 references.

References · 20

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  12. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
8 more
  1. 2016Using the Output Embedding to Improve Language Modelscited by 266
  2. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  3. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  4. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  5. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  6. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  7. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  8. 2023Efficient Adaptation of Large Vision Transformer via Adapter Re-Composingcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 21; 2026: 13.

65 papers.

Sort:

By area:

  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  3. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  4. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  5. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  6. 2024A Survey on LoRA of Large Language Modelscited by 67
  7. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  8. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  9. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
  10. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  11. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  12. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
38 more
  1. 2024Closed-form merging of parameter-efficient modules for Federated Continual Learningcited by 22
  2. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  3. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 18
  4. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 14
  5. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  6. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  7. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  8. 2024Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixturescited by 12
  9. 2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptationcited by 11
  10. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptationcited by 11
  11. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  12. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  13. 2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
  14. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  15. 2024MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shardscited by 7
  16. 2025SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformationcited by 7
  17. 2025Uni-LoRA: One Vector is All You Needcited by 7
  18. 2025Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptationcited by 6
  19. 2025RaSA: Rank-Sharing Low-Rank Adaptationcited by 6
  20. 2025HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Mergingcited by 6
  21. 2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
  22. 2025PiCa: Parameter-Efficient Fine-Tuning with Column Space Projectioncited by 5
  23. 2024Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuningcited by 4
  24. 2024Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuningcited by 4
  25. 2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuningcited by 4
  26. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  27. 2024Low-rank finetuning for LLMs: A fairness perspectivecited by 3
  28. 2025FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Expertscited by 3
  29. 2025WeightLoRA: Keep Only Necessary Adapterscited by 3
  30. 2025K-Merge: Online Continual Merging of Adapters for On-device Large Language Modelscited by 3
  31. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 3
  32. 2026Investigating Single-Block Recurrence in Vision Transformers for Image Recognitioncited by 3
  33. 2024SBoRA: Low-Rank Adaptation with Regional Weight Updatescited by 2
  34. 2024SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesiscited by 2
  35. 2024ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flawscited by 2
  36. 2025LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Modelcited by 2
  37. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  38. 2026Low-Rank Adaptation Redux for Large Modelscited by 2

50 most cited of 65 All 65 (JSON)