mlsanityData to 7 Oct 2026

Bayesian Mixture of Experts For Large Language Models

arXiv 2511.08968 ↗First posted 12 Nov 2025cs.LG cs.CLcited by 4

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We present Bayesian Mixture of Experts (Bayesian-MoE), a post-hoc uncertainty estimation framework for fine-tuned large language models (LLMs) based on Mixture-of-Experts architectures. Our method applies a structured Laplace approximation to the second linear layer of each expert, enabling calibrated uncertainty estimation without modifying the original training procedure or introducing new parameters. Unlike prior approaches, which apply Bayesian inference to added adapter modules, Bayesian-MoE directly targets the expert pathways already present in MoE models, leveraging their modular design for tractable block-wise posterior estimation. We use Kronecker-factored low-rank approximations to model curvature and derive scalable estimates of predictive uncertainty and marginal likelihood. Experiments on common-sense reasoning benchmarks with Qwen1.5-MoE and DeepSeek-MoE demonstrate that Bayesian-MoE improves both expected calibration error (ECE) and negative log-likelihood (NLL) over baselines, confirming its effectiveness for reliable downstream decision-making.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 25 of its 38 references that are in the map. First posted on arXiv 12 Nov 2025.

Builds on

25 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Qwen2.5 Technical Report19 Dec
  2. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  9. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  10. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  11. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  12. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
9 more
  1. 2023LoRA ensembles for large language model fine-tuningcited by 65
  2. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  3. 2022On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliencycited by 31
  4. 2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
  5. 2017Checkpoint Ensembles: Ensemble Methods from a Single Training Processcited by 24
  6. 2023Calibrating Transformers via Sparse Gaussian Processescited by 21
  7. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  8. 2020Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimationcited by 16
  9. 2021Pathologies in priors and inference for Bayesian transformerscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Expertscited by 3
  2. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformerscited by 3
  3. 2026Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillationcited by 1
  4. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know
  2. 2025Mixture of Experts in Large Language Models
  3. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  4. 2024A Survey on Mixture of Experts in Large Language Models
  5. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  6. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  7. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  8. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights