# Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint

arXiv: https://arxiv.org/abs/2007.12349 · HTML: https://mlsanity.com/paper/2007.12349/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 24 Jul 2020.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09248/) (2019)
- [Multi-Head Attention with Disagreement Regularization](https://mlsanity.com/paper/1810.10183/) (2018)
- [Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application](https://mlsanity.com/paper/1803.00710/) (2018)
- [Applying Deep Learning To Airbnb Search](https://mlsanity.com/paper/1810.09591/) (2018)
- [Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning](https://mlsanity.com/paper/1811.02765/) (2018)
- [Tree-gated Deep Mixture-of-Experts For Pose-robust Face Alignment](https://mlsanity.com/paper/1910.09450/) (2019)
- [Ensemble Methods for Personalized E-Commerce Search Challenge at CIKM Cup 2016](https://mlsanity.com/paper/1708.04479/) (2017)

## Built on by (most cited first, 6 of 6)

- [From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape](https://mlsanity.com/paper/2312.10868/) (2023)
- [Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction](https://mlsanity.com/paper/2202.06472/) (2022)
- [Automatic Expert Selection for Multi-Scenario and Multi-Task Search](https://mlsanity.com/paper/2205.14321/) (2022)
- [CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval](https://mlsanity.com/paper/2311.02834/) (2023)
- [AdapTraj: A Multi-Source Domain Generalization Framework for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2312.14394/) (2023)
- [Attention Weighted Mixture of Experts with Contrastive Learning for Personalized Ranking in E-commerce](https://mlsanity.com/paper/2306.05011/) (2023)

## Abstract (arXiv)

Product search is the most common way for people to satisfy their shopping needs on e-commerce websites. Products are typically annotated with one of several broad categorical tags, such as "Clothing" or "Electronics", as well as finer-grained categories like "Refrigerator" or "TV", both under "Electronics". These tags are used to construct a hierarchy of query categories. Distributions of features such as price and brand popularity vary wildly across query categories. In addition, feature importance for the purpose of CTR/CVR predictions differs from one category to another. In this work, we leverage the Mixture of Expert (MoE) framework to learn a ranking model that specializes for each query category. In particular, our gate network relies solely on the category ids extracted from the user query. While classical MoE's pick expert towers spontaneously for each input example, we explore two techniques to establish more explicit and transparent connections between the experts and query categories. To help differentiate experts on their domain specialties, we introduce a form of adversarial regularization among the expert outputs, forcing them to disagree with one another. As a result, they tend to approach each prediction problem from different angles, rather than copying one another. This is validated by a much stronger clustering effect of the gate output vectors under different categories. In addition, soft gating constraints based on the categorical hierarchy are imposed to help similar products choose similar gate values. and make them more likely to share similar experts. This allows aggregation of training data among smaller sibling categories to overcome data scarcity.