mlsanityData to 7 Oct 2026

HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

arXiv 2404.19245 ↗First posted 30 Apr 2024cs.CL cs.AIcited by 118

Not on the map. Why?

Cited by 24 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 17 of its 52 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

17 of 52 references.

References · 17

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021GPT Understands, Toocited by 715
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  11. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  12. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
5 more
  1. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  2. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  3. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  4. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  5. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 61; 2026: 45.

118 papers.

Sort:

By area:

  1. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  2. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  3. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  4. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  5. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  6. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  7. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  8. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
  9. 2025SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Trackingcited by 23
  10. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  11. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  12. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
38 more
  1. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 18
  2. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  3. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  4. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  5. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  6. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  7. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  8. 2025PLoP: Precise LoRA Placement for Efficient Finetuning of Large Modelscited by 13
  9. 2024MFTraj: Map-Free, Behavior-Driven Trajectory Prediction for Autonomous Drivingcited by 12
  10. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
  11. 2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
  12. 2025Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Productcited by 10
  13. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  14. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  15. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  16. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  17. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 9
  18. 2025QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generationcited by 9
  19. 2025Navigating Semantic Drift in Task-Agnostic Class-Incremental Learningcited by 6
  20. 2025MokA: Multimodal Low-Rank Adaptation for MLLMscited by 6
  21. 2026Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptationcited by 6
  22. 2025CoLA: Collaborative Low-Rank Adaptationcited by 5
  23. 2025Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptationcited by 5
  24. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 5
  25. 2025PLAN: Proactive Low-Rank Allocation for Continual Learningcited by 5
  26. 2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 5
  27. 2025LoR2C : Low-Rank Residual Connection Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  28. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  29. 2025Not All LoRA Parameters Are Essential: Insights on Inference Necessitycited by 4
  30. 2025LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Expertscited by 4
  31. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 4
  32. 2025Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stagescited by 4
  33. 2025BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversitycited by 4
  34. 2025Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuningcited by 4
  35. 2025HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalizationcited by 4
  36. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  37. 2025S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuningcited by 3
  38. 2025ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptationcited by 3

50 most cited of 118 All 118 (JSON)