# HyperFormer: Learning Expressive Sparse Feature Representations via Hypergraph Transformer

arXiv: https://arxiv.org/abs/2305.17386 · HTML: https://mlsanity.com/paper/2305.17386/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 27 May 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Neural Factorization Machines for Sparse Predictive Analytics](https://mlsanity.com/paper/1708.05027/) (2017)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Data Augmentation for Deep Graph Learning: A Survey](https://mlsanity.com/paper/2202.08235/) (2022)
- [Be More with Less: Hypergraph Attention Networks for Inductive Text Classification](https://mlsanity.com/paper/2011.00387/) (2020)
- [Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising](https://mlsanity.com/paper/1806.03514/) (2018)
- [Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction](https://mlsanity.com/paper/1910.05552/) (2019)
- [Higher-Order Factorization Machines](https://mlsanity.com/paper/1607.07195/) (2016)
- [A Model of Two Tales: Dual Transfer Learning Framework for Improved Long-tail Item Recommendation](https://mlsanity.com/paper/2010.15982/) (2020)
- [Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1904.04447/) (2019)
- [Meta Propagation Networks for Graph Few-shot Semi-supervised Learning](https://mlsanity.com/paper/2112.09810/) (2021)
- [Cross-Batch Negative Sampling for Training Two-Tower Recommenders](https://mlsanity.com/paper/2110.15154/) (2021)
- [Eliciting Structural and Semantic Global Knowledge in Unsupervised Graph Contrastive Learning](https://mlsanity.com/paper/2202.08480/) (2022)
- [Dual Graph enhanced Embedding Neural Network for CTR Prediction](https://mlsanity.com/paper/2106.00314/) (2021)
- [Session-based Recommendation with Hypergraph Attention Networks](https://mlsanity.com/paper/2112.14266/) (2021)
- [GraphFM: Graph Factorization Machines for Feature Interaction Modeling](https://mlsanity.com/paper/2105.11866/) (2021)
- [Nothing Stands Alone: Relational Fake News Detection with Hypergraph Neural Networks](https://mlsanity.com/paper/2212.12621/) (2022)

## Built on by (most cited first, 8 of 8)

- [InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding](https://mlsanity.com/paper/2306.04933/) (2023)
- [Navigating Open Set Scenarios for Skeleton-based Action Recognition](https://mlsanity.com/paper/2312.06330/) (2023)
- [GC4NC: A Benchmark Framework for Graph Condensation on Node Classification with New Insights](https://mlsanity.com/paper/2406.16715/) (2024)
- [Muscles in Time: Learning to Understand Human Motion by Simulating Muscle Activations](https://mlsanity.com/paper/2411.00128/) (2024)
- [HGFormer: Topology-Aware Vision Transformer with HyperGraph Learning](https://mlsanity.com/paper/2504.02440/) (2025)
- [GGBall: Graph Generative Model on Poincar'e Ball](https://mlsanity.com/paper/2506.07198/) (2025)
- [Implicit Hypergraph Neural Networks: A Stable Framework for Higher-Order Relational Learning with Provable Guarantees](https://mlsanity.com/paper/2508.09427/) (2025)
- [Effective backdoor attack on graph neural networks in link prediction tasks](https://mlsanity.com/paper/2401.02663/) (2024)

## Abstract (arXiv)

Learning expressive representations for high-dimensional yet sparse features has been a longstanding problem in information retrieval. Though recent deep learning methods can partially solve the problem, they often fail to handle the numerous sparse features, particularly those tail feature values with infrequent occurrences in the training data. Worse still, existing methods cannot explicitly leverage the correlations among different instances to help further improve the representation learning on sparse features since such relational prior knowledge is not provided. To address these challenges, in this paper, we tackle the problem of representation learning on feature-sparse data from a graph learning perspective. Specifically, we propose to model the sparse features of different instances using hypergraphs where each node represents a data instance and each hyperedge denotes a distinct feature value. By passing messages on the constructed hypergraphs based on our Hypergraph Transformer (HyperFormer), the learned feature representations capture not only the correlations among different instances but also the correlations among features. Our experiments demonstrate that the proposed approach can effectively improve feature representation learning on sparse features.