mlsanityData to 14 Oct 2026

MoFE: Mixture of Frozen Experts Architecture

arXiv 2503.06491 ↗First posted 9 Mar 2025cs.CL cs.LGcited by 1

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

We propose the Mixture of Frozen Experts (MoFE) architecture, which integrates Parameter-efficient Fine-tuning (PEFT) and the Mixture of Experts (MoE) architecture to enhance both training efficiency and model scalability. By freezing the Feed Forward Network (FFN) layers within the MoE framework, MoFE significantly reduces the number of trainable parameters, improving training efficiency while still allowing for effective knowledge transfer from the expert models. This facilitates the creation of models proficient in multiple domains. We conduct experiments to evaluate the trade-offs between performance and efficiency, compare MoFE with other PEFT methodologies, assess the impact of domain expertise in the constituent models, and determine the optimal training strategy. The results show that, although there may be some trade-offs in performance, the efficiency gains are substantial, making MoFE a reasonable solution for real-world, resource-constrained environments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 22 of its 27 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

22 of 27 references.

References · 22

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
10 more
  1. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  3. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  4. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  5. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  6. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  7. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  8. 2021Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speechcited by 31
  9. 2024Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domaincited by 15
  10. 2023Investigating the Role of Feed-Forward Networks in Transformers Using Parallel Attention and Feed-Forward Net Designcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning
  2. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  3. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning
  4. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
  6. 2025Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
  7. 2026UMoE:Unlocking Every Expert in Domain-Specific Training
  8. 2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning