mlsanityData to 14 Oct 2026

DLLG: Dynamic Logit-Level Gating of LLM Experts

arXiv 2606.04378 ↗First posted 3 Jun 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Leveraging multiple specialized LLMs can combine complementary strengths, but existing approaches trade adaptability for stability: routing commits prematurely, heuristic ensembling depends on fragile proxies, and parameter merging introduces interference. We propose DLLG (Dynamic Logit-Level Gating), a dynamic logit-level ensembling framework that learns token-level expert fusion from sparse response-level supervision. A lightweight gating module predicts step-wise fusion weights, linking trajectory-level correctness to generation without token-level labels or expert retraining. Across diverse reasoning and code benchmarks, DLLG consistently outperforms strong routing, heuristic ensembling, and parameter-merging baselines across model scales, highlighting learned logit-level fusion as a robust and scalable paradigm for integrating specialized experts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 43 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

38 of 43 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2024Mixtral of Expertscited by 1,823
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  12. 2024Qwen2.5-Coder Technical Reportcited by 1,203
26 more
  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  7. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  8. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  9. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  10. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  11. 2023Deep Model Fusion: A Surveycited by 91
  12. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  13. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  14. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 67
  15. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  16. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  17. 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 48
  18. 2024Bridging the Gap between Different Vocabularies for LLM Ensemblecited by 45
  19. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 41
  20. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  21. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  22. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 28
  23. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  24. 2024Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Modelscited by 11
  25. 2024Bench-CoE: a Framework for Collaboration of Experts from Benchmarkcited by 7
  26. 2025Token-level Ensembling of Models with Different Vocabulariescited by 3

Built on by

Not cited yet.