mlsanityData to 14 Oct 2026

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation

arXiv 2506.11638 ↗First posted 13 Jun 2025cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent advances have highlighted the benefits of scaling language models to enhance performance across a wide range of NLP tasks. However, these approaches still face limitations in effectiveness and efficiency when applied to domain-specific tasks, particularly for small edge-side models. We propose the LoRA-Gen framework, which utilizes a large cloud-side model to generate LoRA parameters for edge-side models based on task descriptions. By employing the reparameterization technique, we merge the LoRA parameters into the edge-side model to achieve flexible specialization. Our method facilitates knowledge transfer between models while significantly improving the inference efficiency of the specialized model by reducing the input context length. Without specialized training, LoRA-Gen outperforms conventional LoRA fine-tuning, which achieves competitive accuracy and a 2.1x speedup with TinyLLaMA-1.1B in reasoning tasks. Besides, our method delivers a compression ratio of 10.1x with Gemma-2B on intelligent agent tasks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 41 of its 58 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

41 of 58 references.

References · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
29 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  5. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  7. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  8. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  10. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  11. 2023Adapting Language Models to Compress Contextscited by 356
  12. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  13. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  14. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  15. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  16. 2024Taming Rectified Flow for Inversion and Editingcited by 204
  17. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  18. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  19. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  20. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 111
  21. 2024Higher Layers Need More LoRA Expertscited by 102
  22. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 65
  23. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  24. 2024500xCompressor: Generalized Prompt Compression for Large Language Modelscited by 47
  25. 2024Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domainscited by 30
  26. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  27. 2024UniQA: Unified Vision-Language Pre-training for Image Quality and Aesthetic Assessmentcited by 13
  28. 2025HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generationcited by 13
  29. 2025HaploVL: A Single-Transformer Baseline for Multi-Modal Understandingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 25
  2. 2026Parametric Skillscited by 3
  3. 2025On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMscited by 2
  4. 2026RPMem: Learning Long-Term Recurrent Parametric Memory Across Sessions for LLM Agentscited by 1
  5. 2026TRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Modelcited by 0