# Efficient Modular Learning through Naive LoRA Summation: Leveraging Orthogonality in High-Dimensional Models

arXiv: https://arxiv.org/abs/2508.11985 · HTML: https://mlsanity.com/paper/2508.11985/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 15 references that are in the map.
- First posted on arXiv 16 Aug 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [Mixture of LoRA Experts](https://mlsanity.com/paper/2404.13628/) (2024)
- [Superposition of many models into one](https://mlsanity.com/paper/1902.05522/) (2019)
- [Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering](https://mlsanity.com/paper/2409.16167/) (2024)
- [LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks](https://mlsanity.com/paper/2410.13025/) (2024)
- [On the Complexity of Neural Computation in Superposition](https://mlsanity.com/paper/2409.15318/) (2024)
- [Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging](https://mlsanity.com/paper/2505.15875/) (2025)
- [Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference](https://mlsanity.com/paper/2402.04647/) (2024)
- [Better Prompt Compression Without Multi-Layer Perceptrons](https://mlsanity.com/paper/2501.06730/) (2025)
- [Theory of the superposition principle for randomized connectionist representations in neural networks](https://mlsanity.com/paper/1707.01429/) (2017)

## Abstract (arXiv)

Recent advances in large language models are driven by scale, while parameter-efficient fine-tuning (PEFT) enables updating only a small fraction of parameters. Low-Rank Adaptation (LoRA) stores parameter deltas as the product of two small matrices, which makes them natural building blocks that can be composed. Motivated by the superposition principle, we hypothesize that independently trained LoRA modules on disjoint domains are approximately orthogonal and can be combined by simple addition. Using GPT-2 Small (117M) with LoRA rank 4 and alpha=64, we train adapters for three QA domains (math, medicine, finance). In pairwise tests, adding Math+Medicine adapters improves perplexity by -9.10% relative to merged-data fine-tuning, while Math+Finance and Finance+Medicine change by +4.54% and +27.56%, respectively. Across combinations, the RMS cosine similarity between LoRA deltas correlates positively and approximately linearly with the change in perplexity. Naive summation requires no additional training, can be applied in seconds, and achieves performance comparable to models trained on merged data, while clarifying when interference appears in higher-order compositions.