# Addressing Information Loss and Interaction Collapse: A Dual Enhanced Attention Framework for Feature Interaction

arXiv: https://arxiv.org/abs/2503.11233 · HTML: https://mlsanity.com/paper/2503.11233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks](https://mlsanity.com/paper/1708.04617/) (2017)
- [FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09433/) (2019)
- [PEPNet: Parameter and Embedding Personalized Network for Infusing with Personalized Prior Information](https://mlsanity.com/paper/2302.01115/) (2023)
- [Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising](https://mlsanity.com/paper/1806.03514/) (2018)
- [MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask](https://mlsanity.com/paper/2102.07619/) (2021)
- [FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction](https://mlsanity.com/paper/2304.00902/) (2023)
- [On the Embedding Collapse when Scaling up Recommendation Models](https://mlsanity.com/paper/2310.04400/) (2023)
- [FiBiNet++: Reducing Model Size by Low Rank Feature Interaction Layer for CTR Prediction](https://mlsanity.com/paper/2209.05016/) (2022)
- [ContextNet: A Click-Through Rate Prediction Framework Using Contextual information to Refine Feature Embedding](https://mlsanity.com/paper/2107.12025/) (2021)
- [MemoNet: Memorizing All Cross Features' Representations Efficiently via Multi-Hash Codebook Network for CTR Prediction](https://mlsanity.com/paper/2211.01334/) (2022)

## Abstract (arXiv)

The Transformer has proven to be a significant approach in feature interaction for CTR prediction, achieving considerable success in previous works. However, it also presents potential challenges in handling feature interactions. Firstly, Transformers may encounter information loss when capturing feature interactions. By relying on inner products to represent pairwise relationships, they compress raw interaction information, which can result in a degradation of fidelity. Secondly, due to the long-tail features distribution, feature fields with low information-abundance embeddings constrain the information abundance of other fields, leading to collapsed embedding matrices. To tackle these issues, we propose a Dual Attention Framework for Enhanced Feature Interaction, known as Dual Enhanced Attention. This framework integrates two attention mechanisms: the Combo-ID attention mechanism and the collapse-avoiding attention mechanism. The Combo-ID attention mechanism directly retains feature interaction pairs to mitigate information loss, while the collapse-avoiding attention mechanism adaptively filters out low information-abundance interaction pairs to prevent interaction collapse. Extensive experiments conducted on industrial datasets have shown the effectiveness of Dual Enhanced Attention.