mlsanityData to 14 Oct 2026

KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training

arXiv 2603.22755 ↗First posted 24 Mar 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Independently trained domain specialists can be fused post-hoc into a single model that outperforms any individual specialist, and the gain is predictable: gain = 0.82 x divergence - 2.72 (R^2 = 0.856, n=6, 3-26% divergence). This enables practitioners to estimate cooperative value before committing compute. Below ~3.3% divergence, gains approach zero.In the KALAVAI protocol, contributors fine-tune copies of a shared checkpoint independently, then submit for lightweight MoE routing (500 steps). Gains are consistent: +7.72% at 410M (+/-0.02%, 3 seeds), +7.49% at 1B (+/-0.01%, 3 seeds), +6.53% at 6.9B, each over the best specialist. The router matches domain-oracle routing within <10^{-5} nats. Cross-lingual fusion (Tamil/Yoruba/Welsh/Code) achieves +21.76%, with Yoruba perplexity falling 41.9 to 7.7. A 20-contributor federation achieves +16.71% (+/-0.07pp, 3 seeds).Three requirements bound the protocol. Shared initialisation is necessary: checkpoint mismatch degrades routing. Frozen layers are optional below ~10,000 steps and beneficial beyond. Learned routing is essential: uniform averaging degrades by -1.2% vs. best specialist, while any trained router achieves oracle-optimal assignment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 15 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

12 of 15 references.

References · 12

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2024Mixtral of Expertscited by 1,823
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  8. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  9. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  10. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  11. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  12. 2025STAR: STacked AutoRegressive Scheme for Unified Multimodal Learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Token-Level LLM Collaboration via FusionRoute
  2. 2026Divergence Decoding: Training-Free Capability Fusion
  3. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation
  4. 2024Llama 3 Meets MoE: Efficient Upcycling
  5. 2026Geometric Metrics for MoE Specialization: From Fisher Information to Early Failure Detection
  6. 2025Advancing Expert Specialization for Better MoE
  7. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
  8. 2025TRINITY: An Evolved LLM Coordinator