mlsanityData to 7 Oct 2026

Rethinking LLM Ensembling from the Perspective of Mixture Models

arXiv 2605.00419 ↗First posted 1 May 2026cs.LG cs.CLcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Model ensembling is a well-established technique for improving the performance of machine learning models. Conventionally, this involves averaging the output distributions of multiple models and selecting the most probable label. This idea has been naturally extended to large language models (LLMs), yielding improved performance but incurring substantial computational cost. This inefficiency stems from directly applying conventional ensemble implementation to LLMs, which require a separate forward pass for each model to explicitly compute the ensemble distribution. In this paper, we propose the Mixture-model-like Ensemble (ME). By reinterpreting the ensemble as a mixture model, ME stochastically selects a single model at each step to generate the next token, thereby avoiding the need to explicitly compute the full ensemble distribution. ME is mathematically equivalent to sampling from the ensemble distribution, but requires invoking only one model, making it 1.78x-2.68x faster than conventional ensembling. Furthermore, this perspective connects LLM ensembling and token-level routing methods, suggesting that LLM ensembling is a special case of routing methods. Our findings open new avenues for efficient LLM ensembling and motivate further exploration of token-level routing strategies for LLMs. Our code is available at https://github.com/Kamichanw/Mixture-model-like-Ensemble.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 29 of its 34 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

29 of 34 references.

References · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2024Mixtral of Expertscited by 1,785
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  12. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
17 more
  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  3. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  4. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  5. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  6. 2024Bridging the Gap between Different Vocabularies for LLM Ensemblecited by 44
  7. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  8. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  9. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  10. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 27
  11. 2024Purifying Large Language Models by Ensembling a Small Language Modelcited by 25
  12. 2023Token-Level Adaptation of LoRA Adapters for Downstream Task Generalizationcited by 17
  13. 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
  14. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  15. 2025Fast Large Language Model Collaborative Decoding via Speculationcited by 12
  16. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 6
  17. 2023On-the-Fly Fusion of Large Language Models and Machine Translationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TokenRouter: Efficient Serving System for Token-Level LLM Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SpecFuse: Ensembling Large Language Models via Next-Segment Prediction
  2. 2026Mixture of Complementary Agents for Robust LLM Ensemble
  3. 2024Efficient Dynamic Ensembling for Multiple LLM Experts
  4. 2024MoD: A Distribution-Based Approach for Merging Large Language Models
  5. 2026Accelerating Dense LLMs via L0-regularized Mixture-of-Experts
  6. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild
  7. 2025Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
  8. 2024A Survey on Mixture of Experts in Large Language Models