mlsanityData to 14 Oct 2026

Scalable Heterogeneous Graph Learning via Heterogeneous-aware Orthogonal Prototype Experts

arXiv 2601.05537 ↗First posted 9 Jan 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → MoE for time series and graphs

Abstract (arXiv)

Heterogeneous Graph Neural Networks(HGNNs) have advanced mainly through better encoders, yet their decoding/projection stage still relies on a single shared linear head, assuming it can map rich node embeddings to labels. We call this the Linear Projection Bottleneck: in heterogeneous graphs, contextual diversity and long-tail shifts make a global head miss fine semantics, overfit hub nodes, and underserve tail nodes. While Mixture-of-Experts(MoE) could help, naively applying it clashes with structural imbalance and risks expert collapse. We propose a Heterogeneous-aware Orthogonal Prototype Experts framework named HOPE, a plug-and-play replacement for the standard prediction head. HOPE uses learnable prototype-based routing to assign instances to experts by similarity, letting expert usage follow the natural long-tail distribution, and adds expert orthogonalization to encourage diversity and prevent collapse. Experiments on four real datasets show consistent gains across SOTA HGNN backbones with minimal overhead.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts. Builds on 25 of its 40 references that are in the map. First posted on arXiv 9 Jan 2026.

Builds on

25 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024DA-MoE: Addressing Depth-Sensitivity in Graph-Level Analysis through Mixture of Experts5 Nov
  2. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling6 Apr

Other references · 23

  1. 2017Graph Attention Networkscited by 8,923
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2016Complex Embeddings for Simple Link Predictioncited by 902
  7. 2019Heterogeneous Graph Attention Networkcited by 734
  8. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  9. 2020Heterogeneous Graph Transformercited by 582
  10. 2023From Sparse to Soft Mixtures of Expertscited by 248
  11. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  12. 2024Mixture of LoRA Expertscited by 181
11 more
  1. 2021Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networkscited by 176
  2. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  3. 2022Simple and Efficient Heterogeneous Graph Neural Networkcited by 71
  4. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 54
  5. 2020Scalable Graph Neural Networks for Heterogeneous Graphscited by 31
  6. 2021Heterogeneous Graph Representation Learning with Relation Awarenesscited by 31
  7. 2024GraphMoRE: Mitigating Topological Heterogeneity via Mixture of Riemannian Expertscited by 24
  8. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 19
  9. 2022ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalizationcited by 15
  10. 2021R-GSN: The Relation-based Graph Similar Network for Heterogeneous Graphcited by 6
  11. 2025MoSE: Unveiling Structural Patterns in Graphs via Mixture of Subgraph Expertscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025One Router to Route Them All: Homogeneous Expert Routing for Heterogeneous Graph Transformers
  2. 2025Adaptive Graph Mixture of Residual Experts: Unsupervised Learning on Diverse Graphs with Heterogeneous Specialization
  3. 2025Training Diverse Graph Experts for Ensembles: A Systematic Empirical Study
  4. 2024Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference
  5. 2025GMoPE:A Prompt-Expert Mixture Framework for Graph Foundation Models
  6. 2026Learning How Much to Think: Difficulty-Aware Dynamic MoEs for Graph Node Classification
  7. 2022HIRE: Distilling High-order Relational Knowledge From Heterogeneous Graph Neural Networks
  8. 2026Generalizing GNNs with Tokenized Mixture of Experts