mlsanityData to 7 Oct 2026

TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation

arXiv 2608.06396 ↗First posted 31 Jul 2026cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation. Yet current approaches have two limitations: task experts are typically identified from aggregate routing statistics that reflect usage rather than association with successful task completion, and task-expert activations remain underexplored as signals for supervision allocation. We introduce Task-Expert-Aware Supervision (TEXAS), which combines correctness-conditioned task expert discovery with token-level supervision allocation. TEXAS compares expert activations on instances that the base model solves successfully and those it fails to solve, and retains experts more strongly activated on successful instances. During fine-tuning, it upweights answer tokens in failed instances when they activate these experts. TEXAS therefore leverages existing routing behavior without restricting adaptation to a fixed expert subset or imposing an explicit target routing distribution. Across three MoE models and six benchmarks, TEXAS achieves the best or tied-best performance in 17 of 18 settings and improves over the strongest baseline by 1.3--1.5 points on average. Ablations and further analyses validate both the discovered experts and the resulting supervision strategy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 13 of its 19 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

13 of 19 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs10 Nov
  2. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs27 Aug
  3. 2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules4 Aug
  4. 2025Dense Backpropagation Improves Training for Sparse Mixture-of-Experts16 Apr
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 8

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  6. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  7. 2025Advancing Expert Specialization for Better MoEcited by 37
  8. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
  2. 2026Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
  3. 2026It Takes a MAESTRO To Prune Bad Experts
  4. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  5. 2025Rewiring Experts on the Fly:Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models
  6. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts
  7. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
  8. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models