# HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishou

arXiv: https://arxiv.org/abs/2408.05430 · HTML: https://mlsanity.com/paper/2408.05430/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 10 Aug 2024.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction](https://mlsanity.com/paper/2101.11427/) (2021)
- [PEPNet: Parameter and Embedding Personalized Network for Infusing with Personalized Prior Information](https://mlsanity.com/paper/2302.01115/) (2023)
- [A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks](https://mlsanity.com/paper/1811.06031/) (2018)
- [SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data](https://mlsanity.com/paper/2209.15329/) (2022)
- [TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou](https://mlsanity.com/paper/2302.02352/) (2023)
- [Latent Multi-task Architecture Learning](https://mlsanity.com/paper/1705.08142/) (2017)
- [Multi-task Learning of Hierarchical Vision-Language Representation](https://mlsanity.com/paper/1812.00500/) (2018)
- [Ask the GRU: Multi-Task Learning for Deep Text Recommendations](https://mlsanity.com/paper/1609.02116/) (2016)
- [STEM: Unleashing the Power of Embeddings for Multi-task Recommendation](https://mlsanity.com/paper/2308.13537/) (2023)
- [Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters](https://mlsanity.com/paper/2406.09679/) (2024)
- [Positive, Negative and Neutral: Modeling Implicit Feedback in Session-based News Recommendation](https://mlsanity.com/paper/2205.06058/) (2022)
- [Trinity: Syncretizing Multi-/Long-tail/Long-term Interests All in One](https://mlsanity.com/paper/2402.02842/) (2024)
- [Deep Task-specific Bottom Representation Network for Multi-Task Recommendation](https://mlsanity.com/paper/2308.05996/) (2023)

## Built on by (most cited first, 16 of 16)

- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [See Beyond a Single View: Multi-Attribution Learning Leads to Better Conversion Rate Prediction](https://mlsanity.com/paper/2508.15217/) (2025)
- [DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation](https://mlsanity.com/paper/2503.21135/) (2025)
- [MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation](https://mlsanity.com/paper/2510.15286/) (2025)
- [DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward](https://mlsanity.com/paper/2505.07257/) (2025)
- [RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models](https://mlsanity.com/paper/2509.20883/) (2025)
- [Masked Diffusion Generative Recommendation](https://mlsanity.com/paper/2601.19501/) (2026)
- [CRM: Retrieval Model with Controllable Condition](https://mlsanity.com/paper/2412.13844/) (2024)
- [Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendation](https://mlsanity.com/paper/2510.05125/) (2025)
- [MAC: A Conversion Rate Prediction Benchmark Featuring Labels Under Multiple Attribution Mechanisms](https://mlsanity.com/paper/2603.02184/) (2026)
- [MPFormer: Adaptive Framework for Industrial Multi-Task Personalized Sequential Retriever](https://mlsanity.com/paper/2508.20400/) (2025)
- [AgenticRS-Architecture: System Design for Agentic Recommender Systems](https://mlsanity.com/paper/2603.26085/) (2026)
- [Deep Hierarchical Knowledge Loss for Fault Intensity Diagnosis](https://mlsanity.com/paper/2604.16459/) (2026)
- [Action-Aware Generative Sequence Modeling for Short Video Recommendation](https://mlsanity.com/paper/2604.25834/) (2026)
- [Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence](https://mlsanity.com/paper/2607.20981/) (2026)
- [Multi-Objective Ranking for Live-Streaming: Balancing Fresh and Delayed Signals with Segment-Aware Targeting](https://mlsanity.com/paper/2608.04455/) (2026)

## Abstract (arXiv)

In this paper, we present the practical problems and the lessons learned at short-video services from Kuaishou. In industry, a widely-used multi-task framework is the Mixture-of-Experts (MoE) paradigm, which always introduces some shared and specific experts for each task and then uses gate networks to measure related experts' contributions. Although the MoE achieves remarkable improvements, we still observe three anomalies that seriously affect model performances in our iteration: (1) Expert Collapse: We found that experts' output distributions are significantly different, and some experts have over 90% zero activations with ReLU, making it hard for gate networks to assign fair weights to balance experts. (2) Expert Degradation: Ideally, the shared-expert aims to provide predictive information for all tasks simultaneously. Nevertheless, we find that some shared-experts are occupied by only one task, which indicates that shared-experts lost their ability but degenerated into some specific-experts. (3) Expert Underfitting: In our services, we have dozens of behavior tasks that need to be predicted, but we find that some data-sparse prediction tasks tend to ignore their specific-experts and assign large weights to shared-experts. The reason might be that the shared-experts can perceive more gradient updates and knowledge from dense tasks, while specific-experts easily fall into underfitting due to their sparse behaviors. Motivated by those observations, we propose HoME to achieve a simple, efficient and balanced MoE system for multi-task learning.