mlsanityData to 14 Oct 2026

SpectR: Dynamically Composing LM Experts with Spectral Routing

arXiv 2504.03454 ↗First posted 4 Apr 2025cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Training large, general-purpose language models poses significant challenges. The growing availability of specialized expert models, fine-tuned from pretrained models for specific tasks or domains, offers a promising alternative. Leveraging the potential of these existing expert models in real-world applications requires effective methods to select or merge the models best suited for a given task. This paper introduces SPECTR, an approach for dynamically composing expert models at each time step during inference. Notably, our method requires no additional training and enables flexible, token- and layer-wise model combinations. Our experimental results demonstrate that SPECTR improves routing accuracy over alternative training-free methods, increasing task performance across expert domains.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

42 of 54 references.

References · 42

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2020Knowledge Distillation: A Surveycited by 1,603
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
30 more
  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2018Neural Network Acceptability Judgmentscited by 958
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  6. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  8. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  9. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  10. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  11. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  12. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  13. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  14. 2023DepGraph: Towards Any Structural Pruningcited by 203
  15. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  16. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  17. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  18. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  19. 2024Model merging with SVD to tie the Knotscited by 122
  20. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  21. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 65
  22. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  23. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  24. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  25. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  26. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  27. 2023Token-Level Adaptation of LoRA Adapters for Downstream Task Generalizationcited by 17
  28. 2024AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guaranteescited by 14
  29. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  30. 2024RE-Adapt: Reverse Engineered Adaptation of Large Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 12
  2. 2026Effective LoRA Adapter Routing using Task Representationscited by 6
  3. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  4. 2026Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memorycited by 2
  5. 2025SEQR: Secure and Efficient QR-based LoRA Routingcited by 1
  6. 2026GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioningcited by 1