# Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis

arXiv: https://arxiv.org/abs/2410.19225 · HTML: https://mlsanity.com/paper/2410.19225/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 15 of 15)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Handling Distribution Shifts on Graphs: An Invariance Perspective](https://mlsanity.com/paper/2202.02466/) (2022)
- [Sparse Mixture-of-Experts are Domain Generalizable Learners](https://mlsanity.com/paper/2206.04046/) (2022)
- [Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling](https://mlsanity.com/paper/2304.02806/) (2023)
- [Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts](https://mlsanity.com/paper/2210.03885/) (2022)
- [DANE: Domain Adaptive Network Embedding](https://mlsanity.com/paper/1906.00684/) (2019)
- [On the adequacy of untuned warmup for adaptive optimization](https://mlsanity.com/paper/1910.04209/) (2019)
- [IronMan: GNN-assisted Design Space Exploration in High-Level Synthesis via Reinforcement Learning](https://mlsanity.com/paper/2102.08138/) (2021)
- [High-Level Synthesis Performance Prediction using GNNs: Benchmarking, Modeling, and Advancing](https://mlsanity.com/paper/2201.06848/) (2022)
- [AutoDSE: Enabling Software Programmers to Design Efficient FPGA Accelerators](https://mlsanity.com/paper/2009.14381/) (2020)
- [GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Experts](https://mlsanity.com/paper/2312.04693/) (2023)
- [Node-wise Filtering in Graph Neural Networks: A Mixture of Experts Approach](https://mlsanity.com/paper/2406.03464/) (2024)
- [Graph Classification by Mixture of Diverse Experts](https://mlsanity.com/paper/2103.15622/) (2021)

## Built on by (most cited first, 9 of 9)

- [LLM-DSE: Searching Accelerator Parameters with LLM Agents](https://mlsanity.com/paper/2505.12188/) (2025)
- [Multimodal Mixture-of-Experts for ISAC in Low-Altitude Wireless Networks](https://mlsanity.com/paper/2512.01750/) (2025)
- [MoRSE: Task-Oriented Multi-Agent System with Mixture of Role-Subtask Experts](https://mlsanity.com/paper/2608.09251/) (2026)
- [Iceberg: Enhancing HLS Modeling with Synthetic Data](https://mlsanity.com/paper/2507.09948/) (2025)
- [Report for NSF Workshop on AI for Electronic Design Automation](https://mlsanity.com/paper/2601.14541/) (2026)
- [A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts](https://mlsanity.com/paper/2602.01468/) (2026)
- [Conversion rate prediction in online advertising: modeling techniques, performance evaluation and future directions](https://mlsanity.com/paper/2512.01171/) (2025)
- [MalMoE: Mixture-of-Experts Enhanced Encrypted Malicious Traffic Detection Under Graph Drift](https://mlsanity.com/paper/2602.10157/) (2026)
- [DiffHLS: Differential Learning for High-Level Synthesis QoR Prediction with GNNs and LLM Code Embeddings](https://mlsanity.com/paper/2604.09240/) (2026)

## Abstract (arXiv)

High-level synthesis (HLS) is a widely used tool in designing Field Programmable Gate Array (FPGA). HLS enables FPGA design with software programming languages by compiling the source code into an FPGA circuit. The source code includes a program (called "kernel") and several pragmas that instruct hardware synthesis, such as parallelization, pipeline, etc. While it is relatively easy for software developers to design the program, it heavily relies on hardware knowledge to design the pragmas, posing a big challenge for software developers. Recently, different machine learning algorithms, such as GNNs, have been proposed to automate the pragma design via performance prediction. However, when applying the trained model on new kernels, the significant domain shift often leads to unsatisfactory performance. We propose a more domain-generalizable model structure: a two-level hierarchical Mixture of Experts (MoE), that can be flexibly adapted to any GNN model. Different expert networks can learn to deal with different regions in the representation space, and they can utilize similar patterns between the old kernels and new kernels. In the low-level MoE, we apply MoE on three natural granularities of a program: node, basic block, and graph. The high-level MoE learns to aggregate the three granularities for the final decision. To train the hierarchical MoE stably, we further propose a two-stage training method to avoid expert polarization. Extensive experiments verify the effectiveness of the proposed hierarchical MoE. We publicized our codes at https://github.com/weikai-li/HierarchicalMoE.