SEQR: Secure and Efficient QR-based LoRA Routing
Not on the map. Why?
Abstract (arXiv)
Low-Rank Adaptation (LoRA) has become a standard technique for parameter-efficient fine-tuning of large language models, enabling large libraries of LoRAs, each for a specific task or domain. Efficiently selecting the correct LoRA adapter for a given input remains a challenge, particularly in secure environments where supervised training of routers may raise privacy concerns. Motivated by previous approaches, we formalize the goal of unsupervised LoRA routing in terms of activation norm maximization, providing a theoretical framework for analysis. We demonstrate the discriminative power of activation norms and introduce SEQR, an unsupervised LoRA routing algorithm designed to maximize efficiency while providing strict routing guarantees. SEQR provably identifies the norm-maximizing adapter with significantly greater efficiency, making it a highly scalable and effective solution for dynamic LoRA composition. We validate our results through experiments that demonstrate improved multi-task performance and efficiency.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 40 of its 47 references that are in the map. First posted on arXiv 22 Sep 2025.
Builds on
40 of 47 references.
References · 40
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2016Deep Learning with Differential Privacycited by 2,728
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
- 2022Editing Models with Task Arithmeticcited by 1,120
- 2018Neural Network Acceptability Judgmentscited by 958
28 more
- 2021Membership Inference Attacks From First Principlescited by 878
- 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
- 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
- 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
- 2024Massive Activations in Large Language Modelscited by 294
- 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
- 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
- 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
- 2024Model merging with SVD to tie the Knotscited by 122
- 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
- 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
- 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
- 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
- 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 65
- 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 37
- 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
- 2023Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalizationcited by 28
- 2023Understanding the Feature Norm for Out-of-Distribution Detectioncited by 27
- 2024Towards Scalable Exact Machine Unlearning Using Parameter-Efficient Fine-Tuningcited by 27
- 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
- 2024AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guaranteescited by 14
- 2025LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Taskscited by 8
- 2024Risks When Sharing LoRA Fine-Tuned Diffusion Model Weightscited by 6
- 2025SpectR: Dynamically Composing LM Experts with Spectral Routingcited by 6
- 2024Out-of-Distribution Detection using Neural Activation Priorcited by 2
- 2024Representation Norm Amplification for Out-of-Distribution Detection in Long-Tail Learningcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.