# MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation

arXiv: https://arxiv.org/abs/2510.15286 · HTML: https://mlsanity.com/paper/2510.15286/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MoE for recommender systems, one of 12 directions in Mixture-of-experts.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Placed in

- Mixture-of-experts / [MoE for recommender systems](https://mlsanity.com/moe/moe-for-recommender-systems/)

## Builds on (most cited first, 23 of 23)

- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [PEPNet: Parameter and Embedding Personalized Network for Infusing with Personalized Prior Information](https://mlsanity.com/paper/2302.01115/) (2023)
- [LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders](https://mlsanity.com/paper/2505.04421/) (2025)
- [RankMixer: Scaling Up Ranking Models in Industrial Recommenders](https://mlsanity.com/paper/2507.15551/) (2025)
- [OnePiece: Bringing Context Engineering and Reasoning to Industrial Cascade Ranking System](https://mlsanity.com/paper/2509.18091/) (2025)
- [HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishou](https://mlsanity.com/paper/2408.05430/) (2024)
- [Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems](https://mlsanity.com/paper/2311.05884/) (2023)
- [SelfGNN: Self-Supervised Graph Neural Networks for Sequential Recommendation](https://mlsanity.com/paper/2405.20878/) (2024)
- [Recommender Systems with Heterogeneous Side Information](https://mlsanity.com/paper/1907.08679/) (2019)
- [Multi-Scenario Ranking with Adaptive Feature Learning](https://mlsanity.com/paper/2306.16732/) (2023)
- [MLoRA: Multi-Domain Low-Rank Adaptive Network for CTR Prediction](https://mlsanity.com/paper/2408.08913/) (2024)
- [A Survey of Real-World Recommender Systems: Challenges, Constraints, and Industrial Perspectives](https://mlsanity.com/paper/2509.06002/) (2025)
- [Detecting Arbitrary Order Beneficial Feature Interactions for Recommender Systems](https://mlsanity.com/paper/2206.13764/) (2022)
- [Scene-wise Adaptive Network for Dynamic Cold-start Scenes Optimization in CTR Prediction](https://mlsanity.com/paper/2408.07278/) (2024)
- [Dynamic Forgetting and Spatio-Temporal Periodic Interest Modeling for Local-Life Service Recommendation](https://mlsanity.com/paper/2508.02451/) (2025)
- [Online Gradient Boosting Decision Tree: In-Place Updates for Efficient Adding/Deleting Data](https://mlsanity.com/paper/2502.01634/) (2025)
- [Next-User Retrieval: Enhancing Cold-Start Recommendations via Generative Next-User Modeling](https://mlsanity.com/paper/2506.15267/) (2025)
- [ACT: Automated Constraint Targeting for Multi-Objective Recommender Systems](https://mlsanity.com/paper/2509.03661/) (2025)
- [Improving Large-Scale Recommender Systems with Auxiliary Learning](https://mlsanity.com/paper/2510.02215/) (2025)

## Built on by (most cited first, 3 of 3)

- [UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems](https://mlsanity.com/paper/2604.00590/) (2026)
- [Compute Only Once: UG-Separation for Efficient Large Recommendation Models](https://mlsanity.com/paper/2602.10455/) (2026)
- [From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs](https://mlsanity.com/paper/2609.01240/) (2026)

## Abstract (arXiv)

Industrial recommender systems critically depend on high-quality ranking models. However, traditional pipelines still rely on manual feature engineering and scenario-specific architectures, which hinder cross-scenario transfer and large-scale deployment. To address these challenges, we propose MTmixAtt, a unified Mixture-of-Experts (MoE) architecture with Multi-Mix Attention, designed for large-scale recommendation tasks. MTmixAtt integrates two key components. The AutoToken module automatically clusters heterogeneous features into semantically coherent tokens, removing the need for human-defined feature groups. The MTmixAttBlock module enables efficient token interaction via a learnable mixing matrix, shared dense experts, and scenario-aware sparse experts, capturing both global patterns and scenario-specific behaviors within a single framework. Extensive experiments on the industrial TRec dataset from Meituan demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines including Transformer-based models, WuKong, HiFormer, MLP-Mixer, and RankMixer. At comparable parameter scales, MTmixAtt achieves superior CTR and CTCVR metrics; scaling to MTmixAtt-1B yields further monotonic gains. Large-scale online A/B tests validate the real-world impact: in the textitHomepage scenario, MTmixAtt increases Payment PV by +3.62% and Actual Payment GTV by +2.54%. Overall, MTmixAtt provides a unified and scalable solution for modeling arbitrary heterogeneous features across scenarios, significantly improving both user experience and commercial outcomes.