mlsanityData to 14 Oct 2026

Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors

arXiv 2506.14794 ↗First posted 31 May 2025cs.LG cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Requiring 1013-1015 FLOPs to calculate one 8 bit weight in an LLM during pretraining is extremely expensive and seems inefficient. To better leverage the huge investments made into pretrained models, we develop the new "Assembly-of-Experts" (AoE) construction method to create capable child variants of existing Mixture-of-Experts parent models in linear time. Model weight tensors get interpolated individually, allowing to enhance or suppress semantic features of the parents. Varying the proportion of weights taken from the parent models, we observe some properties of the AoE child model changing gradually, while other behavioral traits emerge with a sharp transition. Surprisingly, nearly every generated model is functional and capable, which makes searching the model space straightforward. We construct the DeepSeek R1T "Chimera", a 671B open-weights hybrid model combining DeepSeek's V3-0324 and R1 model variants. The child inherits only the routed expert tensors of R1, but still achieves about R1-level intelligence. At the same time, it uses about 40% fewer output tokens, close to V3 speed. Constructed without any fine-tuning or distillation, the Chimera exhibits surprisingly compact, orderly reasoning compared to its parent models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

29 of 41 references.

References · 29

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  12. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
17 more
  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  3. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 287
  4. 2024Token-Budget-Aware LLM Reasoningcited by 247
  5. 2025Training Language Models to Reason Efficientlycited by 247
  6. 2019Model Fusion via Optimal Transportcited by 219
  7. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  8. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  9. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 119
  10. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  11. 2023Deep Model Fusion: A Surveycited by 91
  12. 2024Can Language Models Learn to Skip Steps?cited by 59
  13. 2025Command A: An Enterprise-Ready Large Language Modelcited by 59
  14. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  15. 2023Transformer Fusion with Optimal Transportcited by 36
  16. 2024Unconstrained Model Merging for Enhanced LLM Reasoningcited by 7
  17. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  2. 2026IsabeLLM: Automated Theorem Proving Applied to Formally Verifying Consensuscited by 1
  3. 2026Benchmarking Large Language Models on Floating-Point Error Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture of Experts for Low-Resource LLMs
  2. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  3. 2025Autonomy-of-Experts Models
  4. 2026Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search
  5. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  6. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  7. 2026ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
  8. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts