# EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inference

arXiv: https://arxiv.org/abs/2502.04700 · HTML: https://mlsanity.com/paper/2502.04700/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 61 references that are in the map.
- First posted on arXiv 7 Feb 2025.

## Builds on (most cited first, 30 of 42)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Mixture of LoRA Experts](https://mlsanity.com/paper/2404.13628/) (2024)
- [Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview](https://mlsanity.com/paper/1809.09573/) (2018)
- [A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations](https://mlsanity.com/paper/2302.03025/) (2023)
- [The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction](https://mlsanity.com/paper/2312.13558/) (2023)
- [Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2402.04401/) (2024)
- [Multi-LoRA Composition for Image Generation](https://mlsanity.com/paper/2402.16843/) (2024)
- [Towards Efficient Visual Adaption via Structural Re-parameterization](https://mlsanity.com/paper/2302.08106/) (2023)
- [The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research](https://mlsanity.com/paper/2010.15581/) (2020)
- [Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization](https://mlsanity.com/paper/2305.03937/) (2023)
- [Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping](https://mlsanity.com/paper/2102.11343/) (2021)
- [NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation](https://mlsanity.com/paper/2101.12378/) (2021)
- [Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memories](https://mlsanity.com/paper/2306.05406/) (2023)
- [Unraveling the Hidden Environmental Impacts of AI Solutions for Environment](https://mlsanity.com/paper/2110.11822/) (2021)

## Built on by (most cited first, 9 of 9)

- [The Universal Weight Subspace Hypothesis](https://mlsanity.com/paper/2512.05117/) (2025)
- [Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter](https://mlsanity.com/paper/2606.03723/) (2026)
- [Shared LoRA Subspaces for almost Strict Continual Learning](https://mlsanity.com/paper/2602.06043/) (2026)
- [LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modules](https://mlsanity.com/paper/2602.10993/) (2026)
- [SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization](https://mlsanity.com/paper/2604.08368/) (2026)
- [Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting](https://mlsanity.com/paper/2606.03792/) (2026)
- [Biologically Informed Deep Neural Networks for Multi-Omic Integration, Pathway Activity Inference and Risk Stratification in Cancer](https://mlsanity.com/paper/2607.05306/) (2026)
- [Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving](https://mlsanity.com/paper/2608.03579/) (2026)
- [LoRA-generating hypernetworks for efficient on-device LLM generative personalization](https://mlsanity.com/paper/2609.24979/) (2026)

## Abstract (arXiv)

The rapid growth of large models has raised concerns about their environmental impact and equity in accessibility due to significant computational costs. Low-Rank Adapters (LoRA) offer a lightweight solution for finetuning large models, resulting in an abundance of publicly available adapters tailored to diverse domains. We ask: Can these pretrained adapters be leveraged to further streamline adaptation to new tasks while addressing these challenges? We introduce EigenLoRAx, a parameter-efficient finetuning method that recycles existing adapters to create a principal subspace aligned with their shared domain knowledge which can be further augmented with orthogonal basis vectors in low-resource scenarios. This enables rapid adaptation to new tasks by learning only lightweight coefficients on the principal components of the subspace-eliminating the need to finetune entire adapters. EigenLoRAx requires significantly fewer parameters and memory, improving efficiency for both training and inference. Our method demonstrates strong performance across diverse domains and tasks, offering a scalable for edge-based applications, personalization, and equitable deployment of large models in resource-constrained environments.