mlsanityData to 14 Oct 2026

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

arXiv 2604.13368 ↗First posted 15 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 18 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

16 of 18 references.

References · 16

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  5. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  6. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  7. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  8. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  9. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  10. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  11. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  12. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
4 more
  1. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 74
  2. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  3. 2025Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptationcited by 2
  4. 2025TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Modelscited by 1

Built on by

Not cited yet.