mlsanityData to 14 Oct 2026

Token-Level Adaptation of LoRA Adapters for Downstream Task Generalization

arXiv 2311.10847 ↗First posted 17 Nov 2023cs.CL cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

This paper introduces a method for adapting LoRA adapters in smaller-sized language models to arbitrary downstream tasks. Unlike standard mixture-of-expert architectures, our method employs a gradient-free routing function to choose a weighted combination of experts without increasing the compute requirements for training or inference. The results show that token-level adaptation of LoRA adapters outperforms the base Llama-2-7b model across mathematical (GSM8K), scientific (ARC-Challenge), reading comprehension (SQuAD), and coding (CodeAlpaca-20k) tasks. Further evaluations also show that the average performance of token-level adaptation outperforms individual models fine-tuned for each of the tasks with the best performance observed in adaptation of every-other token during inference. The code for this study is made available through a public repository.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

17 of 27 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
5 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  3. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  4. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  5. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  4. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  5. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  6. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  7. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 19
  8. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 15
  9. 2024RouterRetriever: Routing over a Mixture of Expert Embedding Modelscited by 12
  10. 2025SpectR: Dynamically Composing LM Experts with Spectral Routingcited by 6
  11. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 5
  12. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
5 more
  1. 2026MoLoRA: Composable Specialization via Per-Token Adapter Routingcited by 3
  2. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Modelscited by 2
  3. 2026R3AG: Retriever Routing for Retrieval-Augmented Generationcited by 1
  4. 2026Rethinking LLM Ensembling from the Perspective of Mixture Modelscited by 1
  5. 2026Compositional Generalization in Autoregressive Models via Logit Compositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
  2. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  3. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
  4. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  5. 2025MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models
  6. 2026Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks
  7. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality
  8. 2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models