mlsanityData to 14 Oct 2026

Text-to-LoRA: Instant Transformer Adaption

arXiv 2506.06105 ↗First posted 6 Jun 2025cs.LG cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

While Foundation Models provide a general tool for rapid content creation, they regularly require task-specific adaptation. Traditionally, this exercise involves careful curation of datasets and repeated fine-tuning of the underlying model. Fine-tuning techniques enable practitioners to adapt foundation models for many new applications but require expensive and lengthy training while being notably sensitive to hyperparameter choices. To overcome these limitations, we introduce Text-to-LoRA (T2L), a model capable of adapting large language models (LLMs) on the fly solely based on a natural language description of the target task. T2L is a hypernetwork trained to construct LoRAs in a single inexpensive forward pass. After training T2L on a suite of 9 pre-trained LoRA adapters (GSM8K, Arc, etc.), we show that the ad-hoc reconstructed LoRA instances match the performance of task-specific adapters across the corresponding test sets. Furthermore, T2L can compress hundreds of LoRA instances and zero-shot generalize to entirely unseen tasks. This approach provides a significant step towards democratizing the specialization of foundation models and enables language-based adaptation with minimal compute requirements. Our code is available at https://github.com/SakanaAI/text-to-lora

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 40 of its 60 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

40 of 60 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2023Mistral 7Bcited by 3,164
28 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  7. 2016HyperNetworkscited by 1,027
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  9. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  10. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  11. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  12. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  13. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  14. 2019Continual learning with hypernetworkscited by 273
  15. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  16. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  17. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  18. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  19. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  20. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 46
  21. 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
  22. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  23. 2022Hyperdecoders: Instance-specific decoders for multi-task NLPcited by 22
  24. 2022Boosting Natural Language Generation from Instructions with Meta-Learningcited by 15
  25. 2024Attention as a Hypernetworkcited by 13
  26. 2023Task-Agnostic Low-Rank Adapters for Unseen English Dialectscited by 9
  27. 2024HyperLoader: Integrating Hypernetwork-Based LoRA and Adapter Layers into Multi-Task Transformers for Sequence Labellingcited by 6
  28. 2024A Single Linear Layer Yields Task-Adapted Low-Rank Matricescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 52.

58 papers.

Sort:

By area:

  1. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 43
  2. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 25
  3. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 18
  4. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 12
  5. 2026LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agentscited by 10
  6. 2026Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agentscited by 10
  7. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuningcited by 9
  8. 2026Learning a Generative Meta-Model of LLM Activationscited by 8
  9. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 6
  10. 2026Context Distillation as Latent Memory Managementcited by 6
  11. 2025TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Controlcited by 5
  12. 2025Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuningcited by 5
38 more
  1. 2026HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Modelscited by 4
  2. 2026KV-Skill: Forging Expertise in the Model's Native Languagecited by 4
  3. 2026Nectar: Neural Estimation of Cached-Token Attention via Regressioncited by 3
  4. 2026How LoRA Remembers? A Parametric Memory Law for LLM Finetuningcited by 3
  5. 2026User as Engram: Internalizing Per-User Memory as Local Parametric Editscited by 3
  6. 2026Parametric Skillscited by 3
  7. 2026MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generationcited by 3
  8. 2026The Override Gap: A Magnitude Account of Knowledge Conflict Failure in Hypernetwork-Based Instant LLM Adaptationcited by 2
  9. 2026Prompt2Fingerprint: Plug-and-Play LLM Fingerprinting via Text-to-Weight Generationcited by 2
  10. 2026Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolutioncited by 2
  11. 2025IA2: Alignment with ICL Activations Improves Supervised Fine-Tuningcited by 1
  12. 2025GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Modelscited by 1
  13. 2026PRISP: Privacy-Safe Few-Shot Personalization via Lightweight Adaptationcited by 1
  14. 2026DynaTokens: Controlling Token Dynamics for Continual Video-Language Understandingcited by 1
  15. 2026Program-as-Weights: A Programming Paradigm for Fuzzy Functionscited by 1
  16. 2026HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoningcited by 1
  17. 2025Hypernetworks for Perspectivist Adaptationcited by 0
  18. 2026TeNet: Text-to-Network for Compact Policy Synthesiscited by 0
  19. 2026Learn-To-Learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLMcited by 0
  20. 2026HyperTransport: Amortized Conditioning of T2I Generative Modelscited by 0
  21. 2026Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atomscited by 0
  22. 2026PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Promptscited by 0
  23. 2026Context Memorization for Efficient Long Context Generationcited by 0
  24. 2026Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Modelscited by 0
  25. 2026PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalizationcited by 0
  26. 2026Prompt2Effect: Training-Free Image-to-Video Model Specialization via LoRA Generationcited by 0
  27. 2026Biologically Informed Deep Neural Networks for Multi-Omic Integration, Pathway Activity Inference and Risk Stratification in Cancercited by 0
  28. 2026Data-Efficient Adaptation of LLMs via Attention Head Reweightingcited by 0
  29. 2026DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptationcited by 0
  30. 2026Parameter-Dynamic Adaptive Fusion and Calibration Network for RGBT Trackingcited by 0
  31. 2026Simulator-Grounded Large Language Models for Industrial Causal Reasoning: Tool-Use, Structured Injection, and Plant-Portable Retrieval for Wastewater Treatment Decision Supportcited by 0
  32. 2026Amortized Low-Rank Adaptation for Model-Based Reinforcement Learningcited by 0
  33. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0
  34. 2026Circuit Hypernetworks for Quantum-Augmented Diffusion Language Modelscited by 0
  35. 2026PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstructioncited by 0
  36. 2026HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworkscited by 0
  37. 2026Doc2LoRA Provides Decodable Representations of Scientific Ideascited by 0
  38. 2026SkillFM: Generating Skills for LLM Agents via Latent Flow Matchingcited by 0

50 most cited of 58 All 58 (JSON)