mlsanityData to 7 Oct 2026

Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging

arXiv 2505.14136 ↗First posted 20 May 2025cs.LG cs.AIcited by 9

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture of expert (MoE) models are a promising approach to increasing model capacity without increasing inference cost, and are core components of many state-of-the-art language models. However, current MoE models typically use only few experts due to prohibitive training and inference cost. We propose Test-Time Model Merging (TTMM) which scales the MoE paradigm to an order of magnitude more experts and uses model merging to avoid almost any test-time overhead. We show that TTMM is an approximation of test-time training (TTT), which fine-tunes an expert model for each prediction task, i.e., prompt. TTT has recently been shown to significantly improve language models, but is computationally expensive. We find that performance of TTMM improves with more experts and approaches the performance of TTT. Moreover, we find that with a 1B parameter base model, TTMM is more than 100x faster than TTT at test-time by amortizing the cost of TTT at train-time. Thus, TTMM offers a promising cost-effective approach to scale test-time training.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 64 of its 83 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

64 of 83 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Qwen2.5 Technical Report19 Dec
  2. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts15 Aug
  3. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
38 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  3. 2022Editing Models with Task Arithmeticcited by 1,076
  4. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  8. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  10. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  11. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  12. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  13. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  14. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  15. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  16. 2017"Zero-Shot" Super-Resolution using Deep Internal Learningcited by 285
  17. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  18. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  19. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  20. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  21. 2022Test-Time Training with Masked Autoencoderscited by 208
  22. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  23. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  24. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 109
  25. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  26. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 88
  27. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  28. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
  29. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  30. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  31. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  32. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  33. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  34. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  35. 2024Efficiently Learning at Test-Time: Active Fine-Tuning of LLMscited by 42
  36. 2025Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Mergingcited by 25
  37. 2025LADDER: Self-Improving LLMs Through Recursive Problem Decompositioncited by 24
  38. 2019Dynamic Evaluation of Transformer Language Modelscited by 23

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 11
  2. 2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
  3. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 8
  4. 2025Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Modelscited by 5
  5. 2026ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimationcited by 5
  6. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  7. 2025Maximizing Prefix-Confidence at Test-Time Efficiently Improves Mathematical Reasoningcited by 3
  8. 2026Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columnscited by 2
  9. 2026SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  2. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
  3. 2025Mixture of Experts in Large Language Models
  4. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs
  5. 2024Scaling Laws for Fine-Grained Mixture of Experts
  6. 2025Model Merging in Pre-training of Large Language Models
  7. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  8. 2024A Survey on Mixture of Experts in Large Language Models