# Uni-LoRA: One Vector is All You Need

arXiv: https://arxiv.org/abs/2506.00799 · HTML: https://mlsanity.com/paper/2506.00799/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters](https://mlsanity.com/paper/2405.17604/) (2024)
- [Fastfood: Approximate Kernel Expansions in Loglinear Time](https://mlsanity.com/paper/1408.3060/) (2014)
- [Parameter-Efficient Fine-Tuning with Discrete Fourier Transform](https://mlsanity.com/paper/2405.03003/) (2024)
- [Tied-Lora: Enhancing parameter efficiency of LoRA with weight tying](https://mlsanity.com/paper/2311.09578/) (2023)
- [VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks](https://mlsanity.com/paper/2405.15179/) (2024)
- [LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation](https://mlsanity.com/paper/2402.07721/) (2024)
- [Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models](https://mlsanity.com/paper/2305.17446/) (2023)
- [DIMAL: Deep Isometric Manifold Learning Using Sparse Geodesic Sampling](https://mlsanity.com/paper/1711.06011/) (2017)

## Built on by (most cited first, 7 of 7)

- [Learning to Reason in 13 Parameters](https://mlsanity.com/paper/2602.04118/) (2026)
- [FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning](https://mlsanity.com/paper/2511.17979/) (2025)
- [MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2506.09105/) (2025)
- [LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modules](https://mlsanity.com/paper/2602.10993/) (2026)
- [Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2607.14506/) (2026)
- [Compressed Active Subspaces for Scalable Bayesian Inference](https://mlsanity.com/paper/2609.19539/) (2026)
- [FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization](https://mlsanity.com/paper/2610.01537/) (2026)

## Abstract (arXiv)

Low-Rank Adaptation (LoRA) has become the de facto parameter-efficient fine-tuning (PEFT) method for large language models (LLMs) by constraining weight updates to low-rank matrices. Recent works such as Tied-LoRA, VeRA, and VB-LoRA push efficiency further by introducing additional constraints to reduce the trainable parameter space. In this paper, we show that the parameter space reduction strategies employed by these LoRA variants can be formulated within a unified framework, Uni-LoRA, where the LoRA parameter space, flattened as a high-dimensional vector space RD, can be reconstructed through a projection from a subspace Rd, with d ll D. We demonstrate that the fundamental difference among various LoRA methods lies in the choice of the projection matrix, P ∈ RD × d.Most existing LoRA variants rely on layer-wise or structure-specific projections that limit cross-layer parameter sharing, thereby compromising parameter efficiency. In light of this, we introduce an efficient and theoretically grounded projection matrix that is isometric, enabling global parameter sharing and reducing computation overhead. Furthermore, under the unified view of Uni-LoRA, this design requires only a single trainable vector to reconstruct LoRA parameters for the entire LLM - making Uni-LoRA both a unified framework and a "one-vector-only" solution. Extensive experiments on GLUE, mathematical reasoning, and instruction tuning benchmarks demonstrate that Uni-LoRA achieves state-of-the-art parameter efficiency while outperforming or matching prior approaches in predictive performance. Our code is available at https://github.com/KaiyangLi1992/Uni-LoRA.