Faster, Smaller, and Smarter: Task-Aware Expert Merging for Online MoE Inference

Sparse Mixture of Experts (SMoE) has become a preferred architecture for scaling Transformer capacity without increasing computational cost, as it activates only a small subset of experts for each input. However, deploying such an approach for textitonline inference remains challenging due to the large size of a full SMoE model and the complexity of expert routing, especially in resource-constrained edge networks. Moreover, during the online inference, task information is often unavailable, making the task-level routing error-prone. In this work, we propose a novel tree-structured adaptive neural bandit router, textttTanbr, to enable efficient and reliable online MoE inference. Instead of relying on explicit task tags, textttTanbr estimates the task distribution over time from historical data and uses it to guide task-aware expert merging within a given pre-trained MoE. To handle the large continuous space of merging weights, textttTanbr employs a binary tree to progressively partition the space and generate finer candidate weights. It then applies a neural bandit to learn the non-linear mapping from merging weight to model performance and decides optimal expert merging. We prove that textttTanbr achieves a sublinear regret bound of small O(√T log(T)) over small T rounds, despite operating over a continuous decision space, matching regret bounds compared to existing methods. Extensive experiments show that textttTanbr reduces inference latency by at least small 45% and memory usage by up to small 25%, while maintaining a high accuracy compared to many state-of-the-art methods.