# Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems

arXiv: https://arxiv.org/abs/2311.05884 · HTML: https://mlsanity.com/paper/2311.05884/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Detecting Statistical Interactions from Neural Network Weights](https://mlsanity.com/paper/1705.04977/) (2017)
- [Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data](https://mlsanity.com/paper/1807.00311/) (2018)
- [Neural Input Search for Large Scale Recommendation Models](https://mlsanity.com/paper/1907.04471/) (2019)

## Built on by (most cited first, 14 of 14)

- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems](https://mlsanity.com/paper/2604.00590/) (2026)
- [MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation](https://mlsanity.com/paper/2510.15286/) (2025)
- [From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction](https://mlsanity.com/paper/2511.12081/) (2025)
- [Zenith: Scaling up Ranking Models for Billion-scale Livestreaming Recommendation](https://mlsanity.com/paper/2601.21285/) (2026)
- [TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds](https://mlsanity.com/paper/2604.13737/) (2026)
- [Cross-attention Secretly Performs Orthogonal Alignment in Recommendation Models](https://mlsanity.com/paper/2510.09435/) (2025)
- [SOLAR: SVD-Optimized Lifelong Attention for Recommendation](https://mlsanity.com/paper/2603.02561/) (2026)
- [Deferred is Better: A Framework for Multi-Granularity Deferred Interaction of Heterogeneous Features](https://mlsanity.com/paper/2603.12586/) (2026)
- [UniScale: Synergistic Entire Space Data and Model Scaling for Search Ranking](https://mlsanity.com/paper/2603.24226/) (2026)
- [FLUID: From Ephemeral IDs to Multimodal Semantic Codes for Industrial-Scale Livestreaming Recommendation](https://mlsanity.com/paper/2605.21832/) (2026)
- [WHALE: A Scalable Unified Model for Recommendation with Wukong-HSTU Architecture](https://mlsanity.com/paper/2607.17017/) (2026)
- [SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation](https://mlsanity.com/paper/2607.24025/) (2026)
- [From Feature Interaction to Feature Transport - A Unified Block for Scalable Recommendation Models](https://mlsanity.com/paper/2609.01655/) (2026)

## Abstract (arXiv)

Learning feature interaction is the critical backbone to building recommender systems. In web-scale applications, learning feature interaction is extremely challenging due to the sparse and large input feature space; meanwhile, manually crafting effective feature interactions is infeasible because of the exponential solution space. We propose to leverage a Transformer-based architecture with attention layers to automatically capture feature interactions. Transformer architectures have witnessed great success in many domains, such as natural language processing and computer vision. However, there has not been much adoption of Transformer architecture for feature interaction modeling in industry. We aim at closing the gap. We identify two key challenges for applying the vanilla Transformer architecture to web-scale recommender systems: (1) Transformer architecture fails to capture the heterogeneous feature interactions in the self-attention layer; (2) The serving latency of Transformer architecture might be too high to be deployed in web-scale recommender systems. We first propose a heterogeneous self-attention layer, which is a simple yet effective modification to the self-attention layer in Transformer, to take into account the heterogeneity of feature interactions. We then introduce textscHiformer (Heterogeneous Interaction Transformer) to further improve the model expressiveness. With low-rank approximation and model pruning, hiformer enjoys fast inference for online deployment. Extensive offline experiment results corroborates the effectiveness and efficiency of the textscHiformer model. We have successfully deployed the textscHiformer model to a real world large scale App ranking model at Google Play, with significant improvement in key engagement metrics (up to +2.66%).